[squeak-dev] Can I create a 75 Gb Image? (about those post-build stats)

Eliot Miranda eliot.miranda at gmail.com
Mon Oct 18 23:56:31 UTC 2021


Hi Timothy,

On Sun, Oct 17, 2021 at 11:41 AM gettimothy <gettimothy at zoho.com> wrote:

> heh..
>
>
> uptime 0m 5s (runtime 0m 2s, idletime 0m 2s)
> memory 635,437,056 bytes
> old 96,954,144 bytes (15.3%)
> young 528,447,488 bytes (83.2%)
> used -237,651,736 bytes (-37.4%)
> free 363,654,048 bytes (57.2%)
> GCs 2 (2743.5 ms between GCs 1294 ms runtime between GCs)
> full 1 totalling 57 ms (2.2% runtime), avg 57 ms
> marking 32 ms (56.1%) avg 32 ms,
> compacting 25 ms (43.9%) avg 25 ms
> scavenges 1 totalling 0 ms (0% runtime), avg 0 ms
> tenures 0
> Code compactions
> 0
>
>
> this was save and quit then restart.
>

Right, but I'm curious to see that info after a parse.  i.e. how many GCs
were there during a parse and of what duration etc....

>
>
> ---- On Sun, 17 Oct 2021 14:08:14 -0400 *gettimothy via Squeak-dev
> <squeak-dev at lists.squeakfoundation.org
> <squeak-dev at lists.squeakfoundation.org>>* wrote ----
>
> SmalltalkImage current vmParameterAt:55  -> 0.33333298563957214
>
>
> I am trying to figure out where that decimal came from. I tried to find
> the log for base 2,e and 10 and got nothing...so I am going to set it to
> .75 i guess.
>
>
It's the ratio old space has to grow by before the VM will do a full GC
after a scavenge.  SO at 1/3, every time old space grows by 1/3 the VM does
a GC.  e.g. when a 100mb image grows to about 133Mb the VM will GC.  If you
set it to 1 then the image will need to double in size before the VM will
GC after a scavenge.  If you know you're in a growth phase (which you are,
parsing a humngous XML file) you'll save much time on GCs by setting this
high.  e.g. you could set it to 10 for the duration of the parse and then
the VM will only GC when a 100Mb image grows to 1Gb, ewtc.  So one could
imagine it doing something like grow from 100Mb to 1Gb with no full GCs, GC
down to 750mb, grow to 7.5Gb with no GCs, GC down to 5.625Gb, grow to 56Gb
with no GCs, etc.


>
> Here is Extras->VMStatistics before the settings:
>
>
>
> uptime 3m 48s (runtime 0m 43s, idletime 3m 5s)
> memory 128,974,848 bytes
> old 120,022,816 bytes (93.1%)
> young 7,190,528 bytes (5.6%)
> used 97,823,320 bytes (75.8%)
> free 27,227,304 bytes (21.1%)
> GCs 126 (1811.3 ms between GCs 342.7 ms runtime between GCs)
> full 4 totalling 273 ms (0.63% runtime), avg 68.3 ms
> marking 129 ms (47.3%) avg 32.3 ms,
> compacting 144 ms (52.7%) avg 36 ms
> scavenges 122 totalling 57 ms (0.13% runtime), avg 0.5 ms
> tenures 92,778 (avg 760 tenures per scavenge)
> Code compactions
> 12 totalling 11 ms (0.025% runtime), avg 0.9 ms
> Since last view 106 (1478 ms between GCs, 307 ms runtime between GCs)
> uptime 156.8 s, runtime 32.6 s, idletime 124.2 s
> full 3 totalling 215 ms (0.7% runtime), avg 71.7 ms
> scavenge 103 totalling 47 ms (0.1% runtime), avg 0.5 ms
> tenures 13,351 (avg 129 tenures per scavenge)
> code compactions 12 totalling 11 ms (0.034% runtime), avg 0.9 ms
>
>
> here are the settings
>
> SmalltalkImage current vmParameterAt:25 16777216
> SmalltalkImage current vmParameterAt:25 put: (2 raisedTo:30).
>
>
> SmalltalkImage current vmParameterAt:45  0
> SmalltalkImage current vmParameterAt:45 put:536870912.
>
>
> SmalltalkImage current vmParameterAt:55   0.33333298563957214
> SmalltalkImage current vmParameterAt:55 put: 0.75
>
>
> here are the VM Statistics post settings:
>
>
> uptime 1h 0m 20s (runtime 6m 53s, idletime 53m 27s)
> memory 145,752,064 bytes
> old 136,800,032 bytes (93.9%)
> young 7,190,528 bytes (4.9%)
> used 94,899,280 bytes (65.1%)
> free 45,887,768 bytes (31.5%)
> GCs 1,206 (3001.9 ms between GCs 342.6 ms runtime between GCs)
> full 11 totalling 709 ms (0.17% runtime), avg 64.5 ms
> marking 354 ms (49.9%) avg 32.2 ms,
> compacting 355 ms (50.1%) avg 32.3 ms
> scavenges 1,195 totalling 634 ms (0.15% runtime), avg 0.5 ms
> tenures 118,269 (avg 98 tenures per scavenge)
> Code compactions
> 35 totalling 33 ms (0.008% runtime), avg 0.9 ms
> Since last view 1,080 (3140 ms between GCs, 342 ms runtime between GCs)
> uptime 3392.1 s, runtime 369.9 s, idletime 3022.1 s
> full 7 totalling 436 ms (0.1% runtime), avg 62.3 ms
> scavenge 1,073 totalling 577 ms (0.2% runtime), avg 0.5 ms
> tenures 25,491 (avg 23 tenures per scavenge)
> code compactions 23 totalling 22 ms (0.006% runtime), avg 1 ms
>
>
>
> HOWEVER...your wrote:
>
> I’d be interested to see the vm stats at the end of a build. Those will
> help you tune and find out how much time is being spent in GC.
>
>
> Does that mean I must do the whole VMMaker thing?
>
>
No :-).  I just want to see that data for the parse :-)

> If so, please point me (again) to the instructions on that and I will get
> that done this week.
>
>
> Thanks for your time,
>
> t
>
>
>
>
>
>
> ---- On Sun, 17 Oct 2021 12:14:41 -0400 *Eliot Miranda
> <eliot.miranda at gmail.com <eliot.miranda at gmail.com>>* wrote ----
>
> Hi Timothy,
>
> On Oct 17, 2021, at 8:41 AM, gettimothy via Squeak-dev <
> squeak-dev at lists.squeakfoundation.org> wrote:
>
> 
> Hi Folks,
>
>
> I am thinking about the result of parsing that 73Gb xml file and storing
> many of its elements in squeak as live objects.
>
> Assuming an ~1:1 relationship betwen that file and the resultant image,
> can Squeak handle that?
>
> Can I tell squeak to "SmalltalkImage getLarge:73Gb"  ?
>
>
> I think It would be a fun experiment if this possible.
>
> thx in advance.
>
>
> In theory there should be no problem other than a rather sedate snapshot
> and start up time.  If your machine has enough ram and disc Spur will save
> and restore that size of image.  You may find that increasing the size of
> new space/eden to 128/256/512mb, increasing the old space segment size to
> 1gb, and especially changing the growth to size GC ratio, decreases build
> time substantially.
>
> About Squeak is your friend.  Under vm parameters you’ll find info on
> sizes and rooting around you’ll find the levers for setting these sizes
> through vmParameterAt:put:.  See parameters 25, 45, & 55. #25 is badly
> described.  It is actually the minimum old space segment size.  Setting
> this to eg 1gb means a lot less segments to traverse than the default 16mb.
>
> I’d be interested to see the vm stats at the end of a build. Those will
> help you tune and find out how much time is being spent in GC.
>
> _,,,^..^,,,_ (phone)
>
>
>
>
>
>
>

-- 
_,,,^..^,,,_
best, Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20211018/2a5746a9/attachment-0001.html>


More information about the Squeak-dev mailing list