[squeak-dev] Can I create a 75 Gb Image? (about those post-build stats)

gettimothy gettimothy at zoho.com
Tue Oct 19 11:03:40 UTC 2021


Howdy,





I will re-run the parse on squeak after todays Dos/Dolphin study with before and after stats for you.



Levente mentioned a #traceSomething  for the block that I will be adding to that during the run.



I am only going to run it for a million elements in that it gets slightly higher than that during an eleven hour run.





cheers,



t










---- On Mon, 18 Oct 2021 19:56:31 -0400 Eliot Miranda <eliot.miranda at gmail.com> wrote ----



Hi Timothy,




On Sun, Oct 17, 2021 at 11:41 AM gettimothy <mailto:gettimothy at zoho.com> wrote:



heh..





uptime		0m 5s	(runtime 0m 2s, idletime 0m 2s)

memory		635,437,056 bytes

old			96,954,144 bytes (15.3%)

young		528,447,488 bytes (83.2%)

used		-237,651,736 bytes (-37.4%)

free		363,654,048 bytes (57.2%)

GCs			2 (2743.5 ms between GCs 1294 ms runtime between GCs)

full		1 totalling 57 ms (2.2% runtime), avg 57 ms

marking		32 ms (56.1%) avg 32 ms,

compacting	25 ms (43.9%) avg 25 ms

scavenges	1 totalling 0 ms (0% runtime), avg 0 ms

tenures		0

Code compactions

0





this was save and quit then restart.






Right, but I'm curious to see that info after a parse.  i.e. how many GCs were there during a parse and of what duration etc....







---- On Sun, 17 Oct 2021 14:08:14 -0400 gettimothy via Squeak-dev <mailto:squeak-dev at lists.squeakfoundation.org> wrote ----



SmalltalkImage current vmParameterAt:55  -> 0.33333298563957214




I am trying to figure out where that decimal came from. I tried to find the log for base 2,e and 10 and got nothing...so I am going to set it to .75 i guess.











It's the ratio old space has to grow by before the VM will do a full GC after a scavenge.  SO at 1/3, every time old space grows by 1/3 the VM does a GC.  e.g. when a 100mb image grows to about 133Mb the VM will GC.  If you set it to 1 then the image will need to double in size before the VM will GC after a scavenge.  If you know you're in a growth phase (which you are, parsing a humngous XML file) you'll save much time on GCs by setting this high.  e.g. you could set it to 10 for the duration of the parse and then the VM will only GC when a 100Mb image grows to 1Gb, ewtc.  So one could imagine it doing something like grow from 100Mb to 1Gb with no full GCs, GC down to 750mb, grow to 7.5Gb with no GCs, GC down to 5.625Gb, grow to 56Gb with no GCs, etc.







Here is Extras->VMStatistics before the settings:







uptime		3m 48s	(runtime 0m 43s, idletime 3m 5s)

memory		128,974,848 bytes

old			120,022,816 bytes (93.1%)

young		7,190,528 bytes (5.6%)

used		97,823,320 bytes (75.8%)

free		27,227,304 bytes (21.1%)

GCs			126 (1811.3 ms between GCs 342.7 ms runtime between GCs)

full		4 totalling 273 ms (0.63% runtime), avg 68.3 ms

marking		129 ms (47.3%) avg 32.3 ms,

compacting	144 ms (52.7%) avg 36 ms

scavenges	122 totalling 57 ms (0.13% runtime), avg 0.5 ms

tenures		92,778 (avg 760 tenures per scavenge)

Code compactions

12 totalling 11 ms (0.025% runtime), avg 0.9 ms

Since last view	106 (1478 ms between GCs, 307 ms runtime between GCs)

uptime				156.8 s, runtime 32.6 s, idletime 124.2 s

full					3 totalling 215 ms (0.7% runtime), avg 71.7 ms

scavenge				103 totalling 47 ms (0.1% runtime), avg 0.5 ms

tenures				13,351 (avg 129 tenures per scavenge)

code compactions	12 totalling 11 ms (0.034% runtime), avg 0.9 ms




here are the settings


SmalltalkImage current vmParameterAt:25 16777216 

SmalltalkImage current vmParameterAt:25 put: (2 raisedTo:30).





SmalltalkImage current vmParameterAt:45  0

SmalltalkImage current vmParameterAt:45 put:536870912.





SmalltalkImage current vmParameterAt:55   0.33333298563957214

SmalltalkImage current vmParameterAt:55 put: 0.75





here are the VM Statistics post settings:





uptime		1h 0m 20s	(runtime 6m 53s, idletime 53m 27s)

memory		145,752,064 bytes

old			136,800,032 bytes (93.9%)

young		7,190,528 bytes (4.9%)

used		94,899,280 bytes (65.1%)

free		45,887,768 bytes (31.5%)

GCs			1,206 (3001.9 ms between GCs 342.6 ms runtime between GCs)

full		11 totalling 709 ms (0.17% runtime), avg 64.5 ms

marking		354 ms (49.9%) avg 32.2 ms,

compacting	355 ms (50.1%) avg 32.3 ms

scavenges	1,195 totalling 634 ms (0.15% runtime), avg 0.5 ms

tenures		118,269 (avg 98 tenures per scavenge)

Code compactions

35 totalling 33 ms (0.008% runtime), avg 0.9 ms

Since last view	1,080 (3140 ms between GCs, 342 ms runtime between GCs)

uptime				3392.1 s, runtime 369.9 s, idletime 3022.1 s

full					7 totalling 436 ms (0.1% runtime), avg 62.3 ms

scavenge				1,073 totalling 577 ms (0.2% runtime), avg 0.5 ms

tenures				25,491 (avg 23 tenures per scavenge)

code compactions	23 totalling 22 ms (0.006% runtime), avg 1 ms






HOWEVER...your wrote:



I’d be interested to see the vm stats at the end of a build. Those will help you tune and find out how much time is being spent in GC.




Does that mean I must do the whole VMMaker thing? 











No :-).  I just want to see that data for the parse :-)



If so, please point me (again) to the instructions on that and I will get that done this week.





Thanks for your time,



t













---- On Sun, 17 Oct 2021 12:14:41 -0400 Eliot Miranda <mailto:eliot.miranda at gmail.com> wrote ----



Hi Timothy,



On Oct 17, 2021, at 8:41 AM, gettimothy via Squeak-dev <mailto:squeak-dev at lists.squeakfoundation.org> wrote:





Hi Folks,





I am thinking about the result of parsing that 73Gb xml file and storing many of its elements in squeak as live objects.



Assuming an ~1:1 relationship betwen that file and the resultant image, can Squeak handle that?



Can I tell squeak to "SmalltalkImage getLarge:73Gb"  ? 





I think It would be a fun experiment if this possible.



thx in advance.






In theory there should be no problem other than a rather sedate snapshot and start up time.  If your machine has enough ram and disc Spur will save and restore that size of image.  You may find that increasing the size of new space/eden to 128/256/512mb, increasing the old space segment size to 1gb, and especially changing the growth to size GC ratio, decreases build time substantially.



About Squeak is your friend.  Under vm parameters you’ll find info on sizes and rooting around you’ll find the levers for setting these sizes through vmParameterAt:put:.  See parameters 25, 45, & 55. #25 is badly described.  It is actually the minimum old space segment size.  Setting this to eg 1gb means a lot less segments to traverse than the default 16mb.



I’d be interested to see the vm stats at the end of a build. Those will help you tune and find out how much time is being spent in GC.



_,,,^..^,,,_ (phone)



























-- 

_,,,^..^,,,_

best, Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20211019/80cbf7b9/attachment.html>


More information about the Squeak-dev mailing list