Hi Folks,



Because of FileSystem incompats, I am attempting a SAXParser parse of a 73Gb file by running this on pharo:



|t|

t :=  [[(DocDemoSaxHandler on:('/bulkstorage/enwiki-20200501-pages-articles-multistream.xml' asFileReference))  optimizeForLargeDocuments;parseDocument] forkAt:Processor lowIOPriority named:'SAX'] timeToRun.

t inspect.



I am posting here as I am betting that this sort of thing can be common to any platform.



My goal is to see how long this parse will take.

I do not need the data in-image.



During the SAXParse, when I hit a certain Element (or two) , I will be taking those element contents and sending them via network connection to a PEGParser running on Squeak with XTreams.



So, on the SAXParser side, I just need...



stream from 1 to X

send a portion of 1 to X Squeak on another image.

dispose of 1 to X.







Any pointers on how to approach this sort of problem are greatly appreciated.



Linux has the concept of routing to /dev/null to make stuff disappear.



I have never seen that concept in Smalltalk.


cheers,



t
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20211016/9d644048/attachment.html>