<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>Howdy,<br></div><div><br></div><div><br></div><div>I will re-run the parse on squeak after todays Dos/Dolphin study with before and after stats for you.<br></div><div><br></div><div>Levente mentioned a #traceSomething for the block that I will be adding to that during the run.<br></div><div><br></div><div>I am only going to run it for a million elements in that it gets slightly higher than that during an eleven hour run.<br></div><div><br></div><div><br></div><div>cheers,<br></div><div><br></div><div>t</div><div><br></div><div><br></div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Mon, 18 Oct 2021 19:56:31 -0400 <b>Eliot Miranda <eliot.miranda@gmail.com></b> wrote ----<br></div><div><br></div><blockquote style="margin: 0px;"><div><div dir="ltr"><div dir="ltr"><div class="x_-1167648781gmail_default" style="font-size : small;">Hi Timothy,<br></div></div><div><br></div><div class="x_-1167648781gmail_quote"><div dir="ltr" class="x_-1167648781gmail_attr">On Sun, Oct 17, 2021 at 11:41 AM gettimothy <<a href="mailto:gettimothy@zoho.com" target="_blank">gettimothy@zoho.com</a>> wrote:<br></div><blockquote class="x_-1167648781gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left-width : 1px; border-left-style : solid; border-left-color : rgb(204,204,204); padding-left : 1ex;"><div><u></u><br></div><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div>heh..<br></div><div><br></div><div><br></div><div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div>uptime 0m 5s (runtime 0m 2s, idletime 0m 2s)<br></div><div>memory 635,437,056 bytes<br></div><div>old 96,954,144 bytes (15.3%)<br></div><div>young 528,447,488 bytes (83.2%)<br></div><div>used -237,651,736 bytes (-37.4%)<br></div><div>free 363,654,048 bytes (57.2%)<br></div><div>GCs 2 (2743.5 ms between GCs 1294 ms runtime between GCs)<br></div><div>full 1 totalling 57 ms (2.2% runtime), avg 57 ms<br></div><div>marking 32 ms (56.1%) avg 32 ms,<br></div><div>compacting 25 ms (43.9%) avg 25 ms<br></div><div>scavenges 1 totalling 0 ms (0% runtime), avg 0 ms<br></div><div>tenures 0<br></div><div>Code compactions<br></div><div>0<br></div></blockquote><div><br></div></div><div>this was save and quit then restart.<br></div></div></div></blockquote><div><br></div><div class="x_-1167648781gmail_default" style="font-size : small;">Right, but I'm curious to see that info after a parse. i.e. how many GCs were there during a parse and of what duration etc....<br></div><div class="x_-1167648781gmail_default" style="font-size : small;"><br></div><blockquote class="x_-1167648781gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left-width : 1px; border-left-style : solid; border-left-color : rgb(204,204,204); padding-left : 1ex;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div style="border-top-width : 1px; border-top-style : solid; border-top-color : rgb(204,204,204); min-height: 0px; margin-top : 10px; margin-bottom : 10px; line-height : 0px;"><br></div><div><div><br></div><div id="x_-1167648781gmail-m_875078966493271573Zm-_Id_-Sgn1">---- On Sun, 17 Oct 2021 14:08:14 -0400 <b>gettimothy via Squeak-dev <<a href="mailto:squeak-dev@lists.squeakfoundation.org" target="_blank">squeak-dev@lists.squeakfoundation.org</a>></b> wrote ----<br></div><div><br></div><blockquote style="margin : 0px;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div>SmalltalkImage current vmParameterAt:55 -> 0.33333298563957214<br></div></blockquote><div><br></div><div>I am trying to figure out where that decimal came from. I tried to find the log for base 2,e and 10 and got nothing...so I am going to set it to .75 i guess.<br></div></div></div></div></blockquote></div></div></div></blockquote><div><br></div><div class="x_-1167648781gmail_default" style="font-size : small;">It's the ratio old space has to grow by before the VM will do a full GC after a scavenge. SO at 1/3, every time old space grows by 1/3 the VM does a GC. e.g. when a 100mb image grows to about 133Mb the VM will GC. If you set it to 1 then the image will need to double in size before the VM will GC after a scavenge. If you know you're in a growth phase (which you are, parsing a humngous XML file) you'll save much time on GCs by setting this high. e.g. you could set it to 10 for the duration of the parse and then the VM will only GC when a 100Mb image grows to 1Gb, ewtc. So one could imagine it doing something like grow from 100Mb to 1Gb with no full GCs, GC down to 750mb, grow to 7.5Gb with no GCs, GC down to 5.625Gb, grow to 56Gb with no GCs, etc.<br></div><div class="x_-1167648781gmail_default" style="font-size : small;"><br></div><blockquote class="x_-1167648781gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left-width : 1px; border-left-style : solid; border-left-color : rgb(204,204,204); padding-left : 1ex;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div><blockquote style="margin : 0px;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div><div><br></div><div><br></div><div>Here is Extras->VMStatistics before the settings:<br></div><div><br></div><div><br></div><div><br></div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div>uptime 3m 48s (runtime 0m 43s, idletime 3m 5s)<br></div><div>memory 128,974,848 bytes<br></div><div>old 120,022,816 bytes (93.1%)<br></div><div>young 7,190,528 bytes (5.6%)<br></div><div>used 97,823,320 bytes (75.8%)<br></div><div>free 27,227,304 bytes (21.1%)<br></div><div>GCs 126 (1811.3 ms between GCs 342.7 ms runtime between GCs)<br></div><div>full 4 totalling 273 ms (0.63% runtime), avg 68.3 ms<br></div><div>marking 129 ms (47.3%) avg 32.3 ms,<br></div><div>compacting 144 ms (52.7%) avg 36 ms<br></div><div>scavenges 122 totalling 57 ms (0.13% runtime), avg 0.5 ms<br></div><div>tenures 92,778 (avg 760 tenures per scavenge)<br></div><div>Code compactions<br></div><div>12 totalling 11 ms (0.025% runtime), avg 0.9 ms<br></div><div>Since last view 106 (1478 ms between GCs, 307 ms runtime between GCs)<br></div><div>uptime 156.8 s, runtime 32.6 s, idletime 124.2 s<br></div><div>full 3 totalling 215 ms (0.7% runtime), avg 71.7 ms<br></div><div>scavenge 103 totalling 47 ms (0.1% runtime), avg 0.5 ms<br></div><div>tenures 13,351 (avg 129 tenures per scavenge)<br></div><div>code compactions 12 totalling 11 ms (0.034% runtime), avg 0.9 ms<br></div></blockquote><div><br></div><div>here are the settings<br></div></div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div><div>SmalltalkImage current vmParameterAt:25 16777216 <br></div><div>SmalltalkImage current vmParameterAt:25 put: (2 raisedTo:30).<br></div><div><br></div><div><br></div><div>SmalltalkImage current vmParameterAt:45 0<br></div><div>SmalltalkImage current vmParameterAt:45 put:536870912.<br></div><div><br></div><div><br></div><div>SmalltalkImage current vmParameterAt:55 0.33333298563957214<br></div><div>SmalltalkImage current vmParameterAt:55 put: 0.75<br></div></div></blockquote><div><div><br></div><div>here are the VM Statistics post settings:<br></div><div><br></div><div><br></div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div>uptime 1h 0m 20s (runtime 6m 53s, idletime 53m 27s)<br></div><div>memory 145,752,064 bytes<br></div><div>old 136,800,032 bytes (93.9%)<br></div><div>young 7,190,528 bytes (4.9%)<br></div><div>used 94,899,280 bytes (65.1%)<br></div><div>free 45,887,768 bytes (31.5%)<br></div><div>GCs 1,206 (3001.9 ms between GCs 342.6 ms runtime between GCs)<br></div><div>full 11 totalling 709 ms (0.17% runtime), avg 64.5 ms<br></div><div>marking 354 ms (49.9%) avg 32.2 ms,<br></div><div>compacting 355 ms (50.1%) avg 32.3 ms<br></div><div>scavenges 1,195 totalling 634 ms (0.15% runtime), avg 0.5 ms<br></div><div>tenures 118,269 (avg 98 tenures per scavenge)<br></div><div>Code compactions<br></div><div>35 totalling 33 ms (0.008% runtime), avg 0.9 ms<br></div><div>Since last view 1,080 (3140 ms between GCs, 342 ms runtime between GCs)<br></div><div>uptime 3392.1 s, runtime 369.9 s, idletime 3022.1 s<br></div><div>full 7 totalling 436 ms (0.1% runtime), avg 62.3 ms<br></div><div>scavenge 1,073 totalling 577 ms (0.2% runtime), avg 0.5 ms<br></div><div>tenures 25,491 (avg 23 tenures per scavenge)<br></div><div>code compactions 23 totalling 22 ms (0.006% runtime), avg 1 ms<br></div></blockquote><div><br></div><div><br></div><div>HOWEVER...your wrote:<br></div><div><br></div><blockquote style="border : 1px solid rgb(204,204,204); padding : 7px; background-color : rgb(245,245,245);"><div><span class="colour" style="color:rgb(0, 0, 0)"><span class="font" style="font-family:Verdana, Arial, Helvetica, sans-serif"><span class="size" style="font-size: 13px; font-style: normal; font-weight: 400; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; display: inline; float: none;">I’d be interested to see the vm stats at the end of a build. Those will help you tune and find out how much time is being spent in GC.</span></span></span><br></div></blockquote><div><br></div><div>Does that mean I must do the whole VMMaker thing? <br></div></div></div></div></blockquote></div></div></div></blockquote><div><br></div><div class="x_-1167648781gmail_default" style="font-size : small;">No :-). I just want to see that data for the parse :-)<br></div><div class="x_-1167648781gmail_default" style="font-size : small;"><br></div><blockquote class="x_-1167648781gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left-width : 1px; border-left-style : solid; border-left-color : rgb(204,204,204); padding-left : 1ex;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div><blockquote style="margin : 0px;"><div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div><div>If so, please point me (again) to the instructions on that and I will get that done this week.<br></div><div><br></div><div><br></div><div>Thanks for your time,<br></div><div><br></div><div>t<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div id="x_-1167648781gmail-m_875078966493271573x_-594066468Zm-_Id_-Sgn1">---- On Sun, 17 Oct 2021 12:14:41 -0400 <b>Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com" target="_blank">eliot.miranda@gmail.com</a>></b> wrote ----<br></div><div><br></div><blockquote style="margin : 0px;"><div dir="auto"><div>Hi Timothy,<br></div><div dir="ltr"><div><br></div><blockquote>On Oct 17, 2021, at 8:41 AM, gettimothy via Squeak-dev <<a href="mailto:squeak-dev@lists.squeakfoundation.org" target="_blank">squeak-dev@lists.squeakfoundation.org</a>> wrote:<br><br></blockquote></div><blockquote><div dir="ltr"><div><br></div><div style="font-family : Verdana, Arial, Helvetica, sans-serif; font-size : 10pt;"><div>Hi Folks,<br></div><div><br></div><div><br></div><div>I am thinking about the result of parsing that 73Gb xml file and storing many of its elements in squeak as live objects.<br></div><div><br></div><div>Assuming an ~1:1 relationship betwen that file and the resultant image, can Squeak handle that?<br></div><div><br></div><div>Can I tell squeak to "SmalltalkImage getLarge:73Gb" ? <br></div><div><br></div><div><br></div><div>I think It would be a fun experiment if this possible.<br></div><div><br></div><div>thx in advance.<br></div></div></div></blockquote><div><br></div><div>In theory there should be no problem other than a rather sedate snapshot and start up time. If your machine has enough ram and disc Spur will save and restore that size of image. You may find that increasing the size of new space/eden to 128/256/512mb, increasing the old space segment size to 1gb, and especially changing the growth to size GC ratio, decreases build time substantially.<br></div><div><br></div><div>About Squeak is your friend. Under vm parameters you’ll find info on sizes and rooting around you’ll find the levers for setting these sizes through vmParameterAt:put:. See parameters 25, 45, & 55. #25 is badly described. It is actually the minimum old space segment size. Setting this to eg 1gb means a lot less segments to traverse than the default 16mb.<br></div><div><br></div><div><div>I’d be interested to see the vm stats at the end of a build. Those will help you tune and find out how much time is being spent in GC.<br></div><div><br></div><div>_,,,^..^,,,_ (phone)<br></div></div></div></blockquote></div><div><br></div></div><div><br></div></div><div><br></div></blockquote></div><div><br></div></div><div><br></div></div></blockquote></div><div><br></div><div><br></div><div>-- <br></div><div dir="ltr" class="x_-1167648781gmail_signature"><div dir="ltr"><div><span class="size" style="font-size: small; border-collapse: separate;"><div>_,,,^..^,,,_<br></div><div>best, Eliot<br></div></span></div></div></div></div></div></blockquote></div><div><br></div></div><br></body></html>