<div id="__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
Hi Eliot --<div><br></div><div>I answered too fast. I discovered one more important thing. Neither is</div><div>LowSpaceSemaphore signaled nor special object 23 set in the VM.</div><div>The entire code path is not reached, for example, after primitive 71</div><div>failed for the third time:</div><div><br></div><div>Behavior >> #basicNew:</div><div>Behavior >> #handleFailingBasicNew:<br></div><div>Behavior >> #handleFailingFailingBasicNew:</div><div><br></div><div>So, "OutOfMemory signal" is left alone with its try to signal the</div><div>semaphore via #defaultAction, not updating special object 23.</div><div><br></div><div>***</div><div><br></div><div>Given these two hooks for a low-space warning:</div><div><span style="font-size: 10pt"> StackInterpreter >> #checkForEventsMayContextSwitch:<br> NewObjectMemory >> #allocateChunkAfterGC:</span><br></div><div>We end up with these checks not being correct in a low-space</div><div>situation:</div><div> SpurMemoryManager >> #sufficientSpaceAfterGC:</div><div> NewObjectMemory >> #sufficientSpaceToAllocate:</div><div><br></div><div>Hmm... we don't use <span style="font-size: 10pt">NewObjectMemory in Spur images, correct?</span></div><div><span style="font-size: 10pt">So, we only look at</span></div><div><span style="font-size: 10pt"> </span><span style="font-size: 10pt">StackInterpreter >> #checkForEventsMayContextSwitch:</span></div><div><span style="font-size: 10pt"> </span><span style="font-size: 10pt">SpurMemoryManager >> #sufficientSpaceAfterGC:</span></div><div><span style="font-size: 10pt"><br></span></div><div><span style="font-size: 10pt">And now I wonder whether that code path is used at all during</span></div><div><span style="font-size: 10pt">primitive 71 ...</span></div><div><span style="font-size: 10pt"> StackInterpreter >> #</span><span style="font-size: 13.3333px">setSignalLowSpaceFlagAndSaveProcess</span></div><div><span style="font-size: 13.3333px"><br></span></div><div>Please find attached a change set that improves the image side</div><div>a bit.</div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">Best,</span></div><div><span style="font-size: 13.3333px">Marcel</span></div><div><br></div><div class="mb_sig"></div>
<blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color: #AAAAAA; margin-top: 10px;">Am 13.07.2022 10:30:55 schrieb Marcel Taeumel <marcel.taeumel@hpi.de>:</p><div style="font-family:Arial,Helvetica,sans-serif"><div id="__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
Hi Eliot --<div><br></div><div>> <span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">'nuff said?</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">:-) This will help me to better understand the issue. Thanks. The fact is,</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">that the current VM will easily crash when you stress an endless recursion.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">Let me sum up the issues I discovered so far:</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">1. "OutOfMemory signal" is incompatible with #lowSpaceWatcher because</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"> its #defaultAction omits to set special object 23.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">2. #lowSpaceWatcher cannot deal with cases where special object 23 was</span></div><div><span style="font-family: Arial, Helvetica, sans-serif"> not set. Using "Processor preemptedProcess" might help.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">3. #lowSpaceWatcher uses the path for user-interrupt debuggers, which</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"> has a lower bound of process priority 40. So, memory issues in background</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"> processes get not caught/interrupted. I will fix this now.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">4. An endless recursion can crash the VM. An upper memory limit via VM</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"> parameter 67 will let this happen sooner. The #lowSpaceWatcher has no</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"> chance to suspend the erroneous Squeak process in time.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div><span style="font-size: 10pt;font-family: Arial, Helvetica, sans-serif">In VMMaker, I found two places where the </span><span style="font-size: 10pt;font-family: Arial, Helvetica, sans-serif">LowSpaceSemaphore is </span><br></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">signalled:</span></div><div> StackInterpreter >> #checkForEventsMayContextSwitch:</div><div> NewObjectMemory >> #allocateChunkAfterGC:</div><div><br></div><div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">I have no clue why StackInterpreter >> #activeProcess does not seem to</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">work properly. The interrupted process cannot be accessed via special</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt">object 23 in the #lowSpaceWatcher.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div><span style="font-family: Arial, Helvetica, sans-serif">There might be an extra limit missing the avoid a VM crash during endless</span></div></div><div><span style="font-family: Arial, Helvetica, sans-serif">recursion. We cannot enforce the upper limit from parameter 67 if we still</span></div><div><span style="font-family: Arial, Helvetica, sans-serif">need some more frames to let the #lowSpaceWatcher do its work.</span></div><div><span style="font-family: Arial, Helvetica, sans-serif"><br></span></div><div><span style="font-family: Arial, Helvetica, sans-serif">Best,</span></div><div><span style="font-family: Arial, Helvetica, sans-serif">Marcel</span></div><div><span style="font-family: Arial, Helvetica, sans-serif;font-size: 10pt"><br></span></div><div class="mb_sig"></div>
<blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color: #AAAAAA; margin-top: 10px;">Am 12.07.2022 18:58:19 schrieb Eliot Miranda <eliot.miranda@gmail.com>:</p><div style="font-family:Arial,Helvetica,sans-serif">
<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size: 10pt">Hi Marcel,</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 11, 2022 at 11:15 PM Marcel Taeumel <<a href="mailto:marcel.taeumel@hpi.de">marcel.taeumel@hpi.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left-width: 1px;border-left-style: solid;border-left-color: rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr">
Aha. Via #handleFailingFailingBasicNew:, OutOfMemory >> #defaultAction signals the LowSpaceSemaphore. And at that point, special object 23 was not yet set. Hmm....</div></div></blockquote><div><br></div><div class="gmail_default" style="font-size: 10pt"> the VM sets Smalltalk specialObjects at: 23 and signals the low space semaphore when it detects that heap space is low. This is complicated, but essentially Smalltalk will have asked for some allocation to be done, and typically only a large allocation will trip the low space limit, but a cumulative series of small allocations *that are held onto and not reclaimable via a scavenge* can also trip the level. A flag in the vm (needGCFlag) will get set, and the VM's "you need to respond to events" flag gets set by the allocation that failed and/or by the allocation that tripped the new space max size threshold. Then, in checking for events, the VM runs the garbage collector (the scavenger). At this point it may see that available space is low and then there is a fork in the road.</div><div class="gmail_default" style="font-size: 10pt"><br></div><div class="gmail_default" style="font-size: 10pt">If there is no old space size limit in effect, the VM will try and grow old space by adding a new segment. This is a path that leads quickly to the VM growing memory to a level where the system swaps itself into unresponsiveness.</div><div class="gmail_default" style="font-size: 10pt"><br></div><div class="gmail_default" style="font-size: 10pt">If there is a size limit in effect then the VM won't be able to grow old space, will find that the total ammount of free space is below the low space threshold, and will signal the low space semaphore and set Smalltalk specialObjects at: 23 to the process that is active.</div><div class="gmail_default" style="font-size: 10pt"><br></div><div class="gmail_default" style="font-size: 10pt">Sorry this is so complicated; it "was always like this". </div><div class="gmail_default" style="font-size: 10pt"><br></div><div class="gmail_default" style="font-size: 10pt">Executive summary, test the system with an old space size limit in place (Smalltalk vmParameterAt: 67, look for "#67 0 the maximum allowed size of old space (if zero there is no limit)" in the About Squeak->VM Parameters tab in the SystermReporter). Then, if that doesn't work, come back and complain. If it does work then we have some documenting to do to explain how this fragile and complex mechanism can work "if it's Thursday, in the winter months, and there are clouds in the west".</div><div class="gmail_default" style="font-size: 10pt"><br></div><div class="gmail_default" style="font-size: 10pt">'nuff said?</div><div class="gmail_default" style="font-size: 10pt"></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left-width: 1px;border-left-style: solid;border-left-color: rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr"><div><br></div><div>Best,</div><div>Marcel</div><div></div><blockquote type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color:rgb(170,170,170);margin-top:10px">Am 12.07.2022 08:13:15 schrieb Marcel Taeumel <<a href="mailto:marcel.taeumel@hpi.de" target="_blank">marcel.taeumel@hpi.de</a>>:</p><div style="font-family:Arial,Helvetica,sans-serif"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr">
Hi Eliot --<div></div>
<div><br></div><div>> <span style="font-size: 10pt">Our #lowSpaceWatcher seems to be broken because "Smalltalk specialObjectsArray at: 23" is always "nil".</span></div><div>>> [...] <span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">The vm sets it to the active process when it signals the low space semaphore.</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">And that's not working at the moment. Even after the LowSpaceSemaphore got signaled:</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px">Smalltalk vmParameterAt: 67 put: 1 * 1024 * 1024 * 1024.</span></span></div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px">Array new: (0.125 * 1 * 1024 * 1024 * 1024) rounded.</span></span></div></div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px"><br></span></span></div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px">One has to push CMD+Dot to then see that the #lowSpaceWatcher complains that it cannot retrieve the preemptedProcess via special object 23.</span></span></div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px"><br></span></span></div><div><span style="font-family:Arial,Helvetica,sans-serif"><span style="font-size: 13px">> </span></span><span style="font-family: Arial,Helvetica,sans-serif;font-size: 10pt">Has #primSignalAtBytesLeft: (primitive 125) any effect these days?</span></div><span style="font-size: 10pt;font-family: Arial,Helvetica,sans-serif">>> Yes.</span><div><span style="font-size: 10pt;font-family: Arial,Helvetica,sans-serif"><br></span></div><div><span style="font-size: 10pt;font-family: Arial,Helvetica,sans-serif">Hmm... I can only find places that set it to 0. :-) ... Wait ... Ah! Nevermind. xD</span></div><div><span style="font-size: 10pt;font-family: Arial,Helvetica,sans-serif"><br></span></div><div><span style="font-size: 10pt;font-family: Arial,Helvetica,sans-serif">>> </span><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">So infinite recursion (as it always did) causes heap growth, and the issue of sluggishness did to paging comes to the fore.</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">Ah! That was my hypothesis. Yet, since the #lowSpaceWatcher seems to be broken, I wondered what was going on.</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">Thanks! :-)</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">Best,</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px">Marcel</span></div><div><span style="font-family: Arial,Helvetica,sans-serif;font-size: 13px"><br></span></div><div><br></div><blockquote type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color:rgb(170,170,170);margin-top:10px">Am 12.07.2022 08:02:28 schrieb Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com" target="_blank">eliot.miranda@gmail.com</a>>:</p><div style="font-family:Arial,Helvetica,sans-serif">Hi Marcel,<div><br>On Jul 11, 2022, at 6:15 AM, Marcel Taeumel <<a href="mailto:marcel.taeumel@hpi.de" target="_blank">marcel.taeumel@hpi.de</a>> wrote:<br><div dir="ltr"><blockquote type="cite"><br></blockquote></div><blockquote type="cite" style="min-width: 500px"><div dir="ltr"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr">Hi all --<div></div><div><br></div><div>What's the current state of our low-space guards in Squeak? That OutOfMemory error only works with larger allocations, right?</div><div><br></div><div>Our #lowSpaceWatcher seems to be broken because "Smalltalk specialObjectsArray at: 23" is always "nil". Should we replace it with "Processor preemptedProcecss"?</div></div></div></blockquote><div><br></div>No, it *should* be nil. The vm sets it to the active process when it signals the low space semaphore. <span>Hence "Smalltalk specialObjectsArray at: 23" reliably informs the image level low space code which process performed the allocation that tripped the low space trigger.</span></div><div><br><blockquote type="cite" style="min-width: 500px"><div dir="ltr"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr"><div>Has #lowSpaceThreshold any effect these days?</div></div></div></blockquote><div><br></div>Yes. But given that Spur attempts to grow the heap dynamically and contemporary OSs are only too happy to comply, the low space trigger may only be seen to work if a limit on old space size has been installed. Otherwise it’ll most likely not trigger before the system starts to page itself into sluggishness.</div><div><br><blockquote type="cite" style="min-width: 500px"><div dir="ltr"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr"><div><br></div><div>Has #primSignalAtBytesLeft: (primitive 125) any effect these days?</div></div></div></blockquote><div><br></div>Yes.</div><div><br><blockquote type="cite" style="min-width: 500px"><div dir="ltr"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr"><div><br></div><div>Can there ever be an image-level protection against stack-page shortage? VM parameter 42 seems to be 50 (num stack pages), which lasts for quite some time in endless recursion ... but CMD+Dot stops working at some point because of some stack-growing issues? And then the VM crashes eventually.</div></div></div></blockquote><div><br></div>Again that’s not how things work. The stack zone is fixed size. When a new stack page is needed the oldest page is evacuated to the heap in the form of Context objects. So infinite recursion (as it always did) causes heap growth, and the issue of sluggishness did to paging comes to the fore.</div><div><br></div><div><br><blockquote type="cite" style="min-width: 500px"><div dir="ltr"><div id="gmail-m_5171203880409086616__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0);text-align: left" dir="ltr"><div><br></div><div>Best,</div><div>Marcel</div></div><span></span><br></div></blockquote></div></div></blockquote></div></div></blockquote>
</div></div><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-size: 10pt;border-collapse: separate"><div>_,,,^..^,,,_<br></div><div>best, Eliot</div></span></div></div></div></div>
</div></blockquote></div></div></blockquote></div>