<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div dir="auto">Howdy!<br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Thank you for your reply.<br><br><br>Looks like it finished.</div><div dir="auto"><br></div><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div><div>ping: 332000000 elements. Time: 0:02:12:06.877788<br></div><div dir="auto"><div>34291264<br></div><div>332000000 asWords 'three hundred thirty-two million' <br></div></div></div></blockquote><div dir="auto">Those are XMLElements, not documents per-se. Now that I have a rough guestimate of the time involved, I will start tweaking to collect meta-data stuff.<br><br><br>" how much memory do you have on your machine?"<br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div>bash-4.3$ grep MemTotal /proc/meminfo <br></div><div>MemTotal: 32814924 kB<br></div></blockquote>in gigabytes:<br><br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div>32.814924<br></div></blockquote><br>As far as how big the image footprint is....let's seee....<br><br><br><blockquote style="border: 1px solid rgb(204, 204, 204); padding: 7px; background-color: rgb(245, 245, 245);"><div>Information<br></div><div>20,978,056 bytes (internal)<br></div><div>26,689,136 bytes (physical)<br></div><div>26,689,136 bytes (total) <br></div></blockquote><br>An interesting phenomena after the crash happened.<br><br>I killed the process, then saved the image. <br>That save took a long time compared to the original image.<br>I presume that the image grew.<br><br>Another presumption/suspicion is that the Image tried to grow as the application needed it, but then fell behind and caused the process to barf.<br><br>But this is just a guess.<br><br>"<span style="color: rgb(0, 0, 0); font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">I would be trying to parse it in Squeak. Fixing the FileSystem incompatibilities shouldn’t be as difficult as trying to exchange data between images and is very useful)."<br><br></span>I will attempt that. A quick heads up, though.<br><br>I had to change from pharo9 to pharo8 as the jump from 8 to 9 is introducing filesystem incompatablities within pharo (:<br><br>Also, I presume that means modifying <a href="https://github.com/squeak-smalltalk/squeak-filesystem" target="_blank">https://github.com/squeak-smalltalk/squeak-filesystem</a> is this true? <br><br>Or do you have another suggestion?</div><div dir="auto"><br><br>cheers,<br><br>t<br><br>-------------------------------------------------<br><br>Hi,<div> how much memory do you have on your machine? If you parse a smaller, but still substantial, xml file with the same schema, what is the ratio between the document size and its in image version? ie to know if you can process the full 73Gb you need a good estimate of how big the in image footprint is.<br></div><div><br></div><div>I would be trying to parse it in Squeak. Fixing the FileSystem incompatibilities shouldn’t be as difficult as trying to exchange data between images and is very useful).<br><br><div dir="ltr"><span class="highlight" style="background-color:rgba(255, 255, 255, 0)">_,,,^..^,,,_ (phone)</span><br></div><div dir="ltr"><br><blockquote>On Oct 16, 2021, at 3:50 AM, gettimothy via Squeak-dev <<a href="mailto:squeak-dev@lists.squeakfoundation.org" target="_blank">squeak-dev@lists.squeakfoundation.org</a>> wrote:<br><br></blockquote></div><blockquote><div dir="ltr"><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div>Hi Folks,<br></div><div><br></div><div>Because of FileSystem incompats, I am attempting a SAXParser parse of a 73Gb file by running this on pharo:<br></div><div><br></div><blockquote style="border : 1px solid rgb(204, 204, 204); padding : 7px; background-color : rgb(245, 245, 245);"><div><div>|t|<br></div><div>t := [[(DocDemoSaxHandler on:('/bulkstorage/enwiki-20200501-pages-articles-multistream.xml' asFileReference)) optimizeForLargeDocuments;parseDocument] forkAt:Processor lowIOPriority named:'SAX'] timeToRun.<br></div><div>t inspect.<br></div></div></blockquote><div>I am posting here as I am betting that this sort of thing can be common to any platform.<br></div><div><br></div><div>My goal is to see how long this parse will take.<br></div><div>I do not need the data in-image.<br></div><div><br></div><div>During the SAXParse, when I hit a certain Element (or two) , I will be taking those element contents and sending them via network connection to a PEGParser running on Squeak with XTreams.<br></div><div><br></div><div>So, on the SAXParser side, I just need...<br></div><div><br></div><blockquote style="border : 1px solid rgb(204, 204, 204); padding : 7px; background-color : rgb(245, 245, 245);"><div><div>stream from 1 to X<br></div><div>send a portion of 1 to X Squeak on another image.<br></div><div>dispose of 1 to X.<br></div><div><br></div></div></blockquote><div><br></div><div>Any pointers on how to approach this sort of problem are greatly appreciated.<br></div><div><br></div><div>Linux has the concept of routing to /dev/null to make stuff disappear.<br></div><div><br></div><div>I have never seen that concept in Smalltalk.<br></div><div><br></div><div>cheers,<br></div><div><br></div><div>t<br></div></div><br><span></span><br></div></blockquote></div></div><div><br></div></div><br></body></html>