[Newbies] Linux locks up when handling large data sets
squeak414 at free.fr
squeak414 at free.fr
Sat May 3 19:50:42 UTC 2008
Quoting johnps11 at bigpond.com:
> Hi Stan!
>
> Have you tried using the -mmap option when starting Squeak? I notice that
> according to the squeakvm man page:
>
> squeak uses a dynamic heap by default with the maximum size set to
> 75% of the available virtual memory or 1 gigabyte, whichever is smaller.
>
> Perhaps Windows doesn't have this limit. Is one instance of Object bigger
> than 200 bytes? If so you'd expect it to oom.
>
> Assuming you have 2 gig of virtual memory, try starting the VM with
>
> -mmap 1500m
>
> as an option and see if it goes further.
>
> Note you can temporarily increase your virtual memory by running (as root):
>
> dd if=/dev/zero of=/tmp/swapfile bs=1024 count=1M
> mkswap /tmp/swapfile
> swapon /tmp/swapfile
>
> This will increase your VM by 1 Gigabyte - note that on an intel machine
> your VM is usually restricted to a max of about 2gig (unless you start the
> kernel with the right options, and you have a large memory machine, and
> you have a kernel built to support over 4gig of memory). I'm also
> assuming you have a spare gig of space in /tmp . This lasts till you
> reboot or until you run
>
> swapoff /tmp/swapfile
>
> as root.
>
> You'll also want to rm that file when you've finished!
>
> I guess that the *nix ports have this restriction for the squeak VM
> because of the assumption that apps should play nice and not suck the
> system into oblivion!
>
> It might also be instructive to run
>
> vmstat 10 20
>
> in another terminal while you allocate your insanely large array. You'll
> be able to see if the "freeze" is simply massive disk usage as linux goes
> crazy seeking inside the swapfile - you are allocating 100 million
> references before you even start to allocate Objects! Read the man page on
> vmstat on how to interpret it if you don't know the command.
>
> I hope this helps, and doesn't just confuse...
>
> John
>
> http://lists.squeakfoundation.org/mailman/listinfo/beginners
>
Hi John, not confusing- an excellent response, thanks.
With the memory option it also cruises on under Linux, until it freezes at 70
million objects.
While it's still loading vmstat shows:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 32528 38552 22792 758560 0 0 183 520 295 2399 66 9 17 7
3 0 32528 38420 22800 758568 0 0 0 3 254 1779 92 8 0 0
4 0 32528 38420 22804 758568 0 0 0 6 234 2100 93 7 0 0
4 0 32528 38420 22804 758568 0 0 0 0 239 2125 93 7 0 0
2 0 32528 38420 22808 758568 0 0 0 0 233 1379 95 5 0 0
2 0 32528 38420 22816 758568 0 0 0 1 251 2255 92 8 0 0
2 0 32528 38296 22824 758568 0 0 0 3 244 1963 94 6 0 0
4 0 32528 38296 22824 758568 0 0 0 0 230 1863 93 7 0 0
5 0 32528 38296 22832 758568 0 0 0 4 234 2108 91 9 0 0
2 0 32528 38296 22840 758568 0 0 0 12 228 1616 94 6 0 0
4 0 32528 38296 22840 758568 0 0 0 0 234 2142 93 7 0 0
2 0 32528 36608 22680 751824 0 0 0 1 228 1639 94 6 0 0
1 0 32528 36608 22680 751824 0 0 0 7 234 2128 92 8 0 0
3 0 32528 36484 22680 751824 0 0 0 3 248 2170 92 9 0 0
2 0 32528 39080 22560 749472 0 0 2 4 231 1610 93 7 0 0
4 0 32528 35104 22560 749584 0 0 0 0 239 2137 89 11 0 0
3 0 32528 34636 22560 749584 0 0 0 0 236 2084 92 8 0 0
3 0 32528 34960 22568 749584 0 0 0 4 233 1898 93 7 0 0
4 0 32528 38956 22568 749584 0 0 0 1 238 2277 90 10 0 0
2 0 32528 36724 22568 749584 0 0 0 0 227 1719 94 6 0 0
Without the -mmap option, once the image has frozen, vmstat shows:
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 32528 41884 23220 703504 0 0 130 370 272 2402 74 9 12 5
1 0 32528 41624 23228 703512 0 0 0 4 215 1094 97 3 0 0
1 0 32528 41628 23228 703512 0 0 0 0 186 703 98 2 0 0
1 0 32528 41628 23228 703512 0 0 0 0 179 667 97 3 0 0
1 0 32528 41628 23236 703512 0 0 0 6 180 668 97 3 0 0
1 0 32528 41628 23244 703512 0 0 0 4 180 667 97 3 0 0
1 0 32528 41628 23244 703512 0 0 0 0 179 667 98 2 0 0
1 0 32528 41628 23244 703512 0 0 0 0 204 765 96 4 0 0
1 0 32528 41628 23244 703512 0 0 0 2 205 1019 97 3 0 0
1 0 32528 41628 23260 703516 0 0 0 6 180 665 97 3 0 0
1 0 32528 41628 23268 703516 0 0 0 1 179 658 98 2 0 0
1 0 32528 41628 23276 703520 0 0 0 6 180 657 97 3 0 0
1 0 32528 41628 23276 703520 0 0 0 3 180 655 97 3 0 0
1 0 32528 41628 23276 703520 0 0 0 0 180 654 97 3 0 0
1 0 32528 41628 23280 703520 0 0 0 0 179 656 97 3 0 0
1 0 32528 41628 23280 703520 0 0 0 0 179 659 98 3 0 0
1 0 32528 41628 23288 703520 0 0 0 1 179 653 97 3 0 0
1 0 32528 41628 23288 703520 0 0 0 0 179 654 98 2 0 0
1 0 32528 41628 23288 703520 0 0 0 0 179 654 98 2 0 0
1 0 32528 41628 23296 703520 0 0 0 3 180 658 98 2 0 0.
Nothing obviously different to me.
At least I can work around this as long as I keep sizes moderate.
Thanks again, Stan
More information about the Beginners
mailing list