Hi,

    how much memory do you have on your machine?  If you parse a smaller, but still substantial, xml file with the same schema, what is the ratio between the document size and its in image version? ie to know if you can process the full 73Gb you need a good estimate of how big the in image footprint is.

I would be trying to parse it in Squeak.  Fixing the FileSystem incompatibilities shouldn’t be as difficult as trying to exchange data between images and is very useful).

_,,,^..^,,,_ (phone)

> On Oct 16, 2021, at 3:50 AM, gettimothy via Squeak-dev <squeak-dev at lists.squeakfoundation.org> wrote:
> 
> 
> Hi Folks,
> 
> Because of FileSystem incompats, I am attempting a SAXParser parse of a 73Gb file by running this on pharo:
> 
> |t|
> t :=  [[(DocDemoSaxHandler on:('/bulkstorage/enwiki-20200501-pages-articles-multistream.xml' asFileReference))  optimizeForLargeDocuments;parseDocument] forkAt:Processor lowIOPriority named:'SAX'] timeToRun.
> t inspect.
> I am posting here as I am betting that this sort of thing can be common to any platform.
> 
> My goal is to see how long this parse will take.
> I do not need the data in-image.
> 
> During the SAXParse, when I hit a certain Element (or two) , I will be taking those element contents and sending them via network connection to a PEGParser running on Squeak with XTreams.
> 
> So, on the SAXParser side, I just need...
> 
> stream from 1 to X
> send a portion of 1 to X Squeak on another image.
> dispose of 1 to X.
> 
> 
> Any pointers on how to approach this sort of problem are greatly appreciated.
> 
> Linux has the concept of routing to /dev/null to make stuff disappear.
> 
> I have never seen that concept in Smalltalk.
> 
> cheers,
> 
> t
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20211016/8ec9f6e1/attachment.html>