[Newbies] Linux locks up when handling large data sets

[johnps11 at bigpond.com]

Hi Stan!

Have you tried using the -mmap option when starting Squeak?  I notice that
according to the squeakvm man page:

  squeak uses a dynamic heap by default with the maximum size  set  to
  75%  of the available virtual memory or 1 gigabyte, whichever is smaller.

Perhaps Windows doesn't have this limit.  Is one instance of Object bigger
than 200 bytes? If so you'd expect it to oom.

Assuming you have 2 gig of virtual memory, try starting the VM with

-mmap  1500m

as an option and see if it goes further.

Note you can temporarily increase your virtual memory by running (as root):

dd if=/dev/zero of=/tmp/swapfile bs=1024 count=1M
mkswap /tmp/swapfile
swapon /tmp/swapfile

This will increase your VM by 1 Gigabyte - note that on an intel machine
your VM is usually restricted to a max of about 2gig (unless you start the
kernel with the right options, and you have a large memory machine, and
you have a kernel built to support over 4gig of memory).  I'm also
assuming you have a spare gig of space in /tmp . This lasts till you
reboot or until you run

     swapoff /tmp/swapfile

as root.

You'll also want to rm that file when you've finished!

I guess that the *nix ports have this restriction for the squeak VM
because of the assumption that apps should play nice and not suck the
system into oblivion!

It might also be instructive to run

     vmstat 10 20

in another terminal while you allocate your insanely large array.  You'll
be able to see if the "freeze" is simply massive disk usage as linux goes
crazy seeking inside the swapfile - you are allocating 100 million
references before you even start to allocate Objects! Read the man page on
vmstat on how to interpret it if you don't know the command.

I hope this helps, and doesn't just confuse...

John


> Hi, I've been testing Squeak's ability to handle large amounts of data.
> This
> snippet:
>
> testArrayFilling
> 	| startTime endTime iArray jArray kArray |
> 	iArray := Array ofSize: 100.
> 	1
> 		to: 100
> 		do: [:i |
> 			jArray := Array ofSize: 1000.
> 			1
> 				to: 1000
> 				do: [:j |
> 					startTime := Time millisecondClockValue.
> 					kArray := Array ofSize: 1000.
> 					1
> 						to: 1000
> 						do: [:k | kArray at: k put: Object new].
> 					jArray at: j put: kArray.
> 					endTime := Time millisecondClockValue.
> 					Transcript cr; show: i asString , ',' , j asString , ' ' , (endTime -
> startTime) asFloat asString.
> 					startTime := Time millisecondClockValue].
> 			iArray at: i put: jArray].
> 	Transcript cr; show: 'Finished'
>
> creates about 5 million objects , then the image freezes. When I run on
> the same
> machine under Windows, it happily continues until the short of memory
> warning (
> about 70 Million objects in my case). The VM is 3.7 in both cases, the
> image
> Damien's 3.9 development image.
>
> There seem to be a number of Unix lockup issues. Is a later (or earlier)
> VM
> likely to fix this please?
>
> Thanks,   ...Stan
>
> _______________________________________________
> Beginners mailing list
> Beginners at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/beginners
>