[Challenge] large files smart compare (was: Re: Squeak for I/O and Memory Intensive tasks )

Yoel Jacobsen yoel at emet.co.il
Tue Jan 29 09:41:36 UTC 2002


Bob,

    First, my mistake. It took 12 minutes for 100K entries (thanks for 
finding this out)
    Second, for 10K entries it consumed +3M.

    The real case was with a 450M file (not 1.8M as the 10K demo 
produces) with 18M lines. I _afraid_ to measure it on squeak because it 
will hang my computer for a long long while even if I'll find a machine 
with 750MB (450/1.8 * 3) of free RAM.

        Yoel

Bob Arning wrote:

>On Tue, 29 Jan 2002 09:39:54 +0200 Yoel Jacobsen <yoel at emet.co.il> wrote:
>
>>The attached file is an almost empty LDAPObject class with fromStr: class method only. This method parse a single LDIF entry. Look the the class information for the commands to create a 10K lines LDIF file and to parse it.
>>
>>Parsing 10K lines took me about 12 minutes in which the image was working on this as a single task. Profiling shoes the time is mostly spent on adding to collections.
>>
>
>On Tue, 29 Jan 2002 01:12:08 -0700 Jon Hylands <jon at huv.com> wrote:
>
>>I filed your code into a 3.1 image (3828), and it took 11.8 seconds to
>>parse the 10,000 lines...
>>
>
>Yoel,
>
>If you really mean 12 *minutes*, then something is slowing you down big-time. Your test case is going to consume over 10M of memory while running. Could you be short of real memory available to squeak? If you change your test to do
>
>	MessageTally spyOn: [
>		blocks _ lines findBetweenSubStrs: { crcr  }.
>		objs _ blocks collect: [ :bl | LDAPObject fromStr: bl ].
>	].
>
>What do you get at the end of the report? How does it differ from this:
>
>**Memory**
>	old			+10,229,352 bytes
>	young		-1,790,776 bytes
>	used		+8,438,576 bytes
>	free		-8,438,576 bytes
>
>**GCs**
>	full			0 totalling 0ms (0.0% uptime)
>	incr		467 totalling 1,878ms (13.0% uptime), avg 4.0ms
>	tenures		220 (avg 2 GCs/tenure)
>	root table	0 overflows
>
>Cheers,
>Bob
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20020129/0465589d/attachment.htm


More information about the Squeak-dev mailing list