[squeak-dev] Low-space considerations ... :-)

David T. Lewis lewis at mail.msen.com
Wed Jul 20 17:33:16 UTC 2022


Some historical background on our low space interrupt handling:

The current mechanism was introduced in 2005, beginning with some change
sets posted on the mailing lists here:

  http://lists.squeakfoundation.org/pipermail/vm-dev/2005-April/000130.html

The issue and related discussion was moved to the Mantis bug tracker (Mantis
issue number 1041). Our old Mantis bug tracker is currently off line (hopefully
not permanently), but some of the discussion can still be seen on the mailing
list archives in the thread above.

Andreas, Tim and I worked out the details on the mailing list and on Mantis,
and the final result was added to the VM in this update:

   Name: VMMaker-tpr.20
   Author: tpr
   Time: 25 April 2005, 7:20:15 pm
   UUID: a0f6e9f2-3752-41b9-8b89-f5da12539592
   Ancestors: VMMaker-tpr.19
   
   Merge dtl's changes to support VM reporting of the guilty lowSpace causing process into the 64bit stream.


A small but important reference (quoted in http://lists.squeakfoundation.org/pipermail/vm-dev/2005-April/000158.html)
is the four Andreas tests for interruptability:

  "[true] whileTrue"
  "[[true] whileTrue] forkAt: Processor userSchedulingPriority + 1"
  "Smalltalk createStackOverflow"
  "[Smalltalk createStackOverflow] forkAt: Processor userSchedulingPriority + 1"


These are the four simple cases for which the user interrupt needs to work
in both Morphic and MVC. Andreas had posted this earlier either on the
mailing list or in the Mantis discussion, and it is a set of basic tests
that needs to be preserved for posterity.

Dave


On Tue, Jul 19, 2022 at 01:58:11PM -0400, David T. Lewis wrote:
> Here is a method comment that explains the overall approach:
> 
> setSignalLowSpaceFlagAndSaveProcess
> 	"The low space semaphore is about to be signaled. Set the signalLowSpace flag,
> 	 and force an interrupt check.  Save the currently active process in the special
> 	 objects array so that the low space handler will be able to determine the process
> 	 that first triggered a low space condition. The image's low space handler is expected
> 	 to nil out the special objects array slot when it handles the low space condition."
> 
> Dave
> 
> 
> On Wed, Jul 13, 2022 at 11:01:16AM +0200, Marcel Taeumel wrote:
> > Hi Eliot --
> > 
> > I answered too fast. I discovered one more important thing. Neither is
> > LowSpaceSemaphore signaled nor special object 23 set in the VM.
> > The entire code path is not reached, for example, after primitive 71
> > failed for the third time:
> > 
> > Behavior >> #basicNew:
> > Behavior >> #handleFailingBasicNew:
> > 
> > Behavior >> #handleFailingFailingBasicNew:
> > 
> > So, "OutOfMemory signal" is left alone with its try to signal the
> > semaphore via #defaultAction, not updating special object 23.
> > 
> > ***
> > 
> > Given these two hooks for a low-space warning:
> > ?? ??StackInterpreter >> #checkForEventsMayContextSwitch:
> > ?? ??NewObjectMemory >> #allocateChunkAfterGC:
> > 
> > We end up with these checks not being correct in a low-space
> > situation:
> > ?? ??SpurMemoryManager >> #sufficientSpaceAfterGC:
> > ?? ??NewObjectMemory >> #sufficientSpaceToAllocate:
> > 
> > Hmm... we don't use??NewObjectMemory in Spur images, correct?
> > So, we only look at
> > ?? ??StackInterpreter >> #checkForEventsMayContextSwitch:
> > ?? ??SpurMemoryManager >> #sufficientSpaceAfterGC:
> > 
> > And now I wonder whether that code path is used at all during
> > primitive 71 ...
> > ?? ??StackInterpreter >> #setSignalLowSpaceFlagAndSaveProcess
> > 
> > Please find attached a change set that improves the image side
> > a bit.
> > 
> > Best,
> > Marcel
> > 
> > Am 13.07.2022 10:30:55 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:
> > Hi Eliot --
> > 
> > >??'nuff said?
> > 
> > :-) This will help me to better understand the issue. Thanks. The fact is,
> > that the current VM will easily crash when you stress an endless recursion.
> > 
> > Let me sum up the issues I discovered so far:
> > 
> > 1. "OutOfMemory signal" is incompatible with #lowSpaceWatcher because
> > ?? ??its #defaultAction omits to set special object 23.
> > 2. #lowSpaceWatcher cannot deal with cases where special object 23 was
> > ?? ??not set. Using "Processor preemptedProcess" might help.
> > 3. #lowSpaceWatcher uses the path for user-interrupt debuggers, which
> > ?? ??has a lower bound of process priority 40. So, memory issues in background
> > ?? ??processes get not caught/interrupted. I will fix this now.
> > 4. An endless recursion can crash the VM. An upper memory limit via VM
> > ?? ??parameter 67 will let this happen sooner. The #lowSpaceWatcher has no
> > ?? ??chance to suspend the erroneous Squeak process in time.
> > 
> > In VMMaker, I found two places where the??LowSpaceSemaphore is??
> > 
> > signalled:
> > ?? ??StackInterpreter >> #checkForEventsMayContextSwitch:
> > ?? ??NewObjectMemory >> #allocateChunkAfterGC:
> > 
> > I have no clue why StackInterpreter >> #activeProcess does not seem to
> > work properly. The interrupted process cannot be accessed via special
> > object 23 in the #lowSpaceWatcher.
> > 
> > There might be an extra limit missing the avoid a VM crash during endless
> > recursion. We cannot enforce the upper limit from parameter 67 if we still
> > need some more frames to let the #lowSpaceWatcher do its work.
> > 
> > Best,
> > Marcel
> > 
> > Am 12.07.2022 18:58:19 schrieb Eliot Miranda <eliot.miranda at gmail.com>:
> > Hi Marcel,
> > 
> > On Mon, Jul 11, 2022 at 11:15 PM Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> wrote:
> > 
> > Aha. Via #handleFailingFailingBasicNew:, OutOfMemory >> #defaultAction signals the LowSpaceSemaphore. And at that point, special object 23 was not yet set. Hmm....
> > 
> > ?? the VM sets Smalltalk specialObjects at: 23 and signals the low space semaphore when it detects that heap space is low.?? This is complicated, but essentially Smalltalk will have asked for some allocation to be done, and typically only a large allocation will trip the low space limit, but a cumulative series of small allocations *that are held onto and not reclaimable via a scavenge* can also trip the level.?? A flag in the vm (needGCFlag) will get set, and ??the VM's "you need to respond to events" flag gets set by the allocation that failed and/or by the allocation that tripped the new space max size threshold.?? Then, in checking for events, the VM runs the garbage collector (the scavenger).?? At this point it may see that available space is low and then there is a fork in the road.
> > 
> > If there is no old space size limit in effect, the VM will try and grow old space by adding a new segment.?? This is a path that leads quickly to the VM growing memory to a level where the system swaps itself into unresponsiveness.
> > 
> > If there is a size limit in effect then the VM won't be able to grow old space, will find that the total ammount of free space is below the low space threshold, and will signal the low space semaphore and set Smalltalk specialObjects at: 23 to the process that is active.
> > 
> > Sorry this is so complicated; it "was always like this". ??
> > 
> > Executive summary, test the system with an old space size limit in place (Smalltalk vmParameterAt: 67, look for "#67 0 the maximum allowed size of old space (if zero there is no limit)" in the About Squeak->VM Parameters tab in the SystermReporter).?? Then, if that doesn't work, come back and complain.?? If it does work then we have some documenting to do to explain how this fragile and complex mechanism can work "if it's Thursday, in the winter months, and there are clouds in the west".
> > 
> > 'nuff said?
> > 
> > Best,
> > Marcel
> > Am 12.07.2022 08:13:15 schrieb Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]>:
> > Hi Eliot --
> > 
> > >??Our #lowSpaceWatcher seems to be broken because "Smalltalk specialObjectsArray at: 23" is always "nil".
> > >> [...]??The vm sets it to the active process when it signals the low space semaphore.
> > 
> > And that's not working at the moment. Even after the LowSpaceSemaphore got signaled:
> > 
> > Smalltalk vmParameterAt: 67 put: 1 * 1024 * 1024 * 1024.
> > Array new: (0.125 * 1 * 1024 * 1024 * 1024) rounded.
> > 
> > One has to push CMD+Dot to then see that the #lowSpaceWatcher complains that?? it cannot retrieve the preemptedProcess via special object 23.
> > 
> > >??Has #primSignalAtBytesLeft: (primitive 125) any effect these days?
> > >> Yes.
> > 
> > Hmm... I can only find places that set it to 0. :-) ... Wait ... Ah! Nevermind. xD
> > 
> > >>??So infinite recursion (as it always did) causes heap growth, and the issue of sluggishness did to paging comes to the fore.
> > 
> > Ah! That was my hypothesis. Yet, since the #lowSpaceWatcher seems to be broken, I wondered what was going on.
> > 
> > Thanks! :-)
> > 
> > Best,
> > Marcel
> > 
> > 
> > Am 12.07.2022 08:02:28 schrieb Eliot Miranda <eliot.miranda at gmail.com [mailto:eliot.miranda at gmail.com]>:
> > Hi Marcel,
> > 
> > On Jul 11, 2022, at 6:15 AM, Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> wrote:
> > 
> > 
> > ???
> > Hi all --
> > 
> > What's the current state of our low-space guards in Squeak? That OutOfMemory error only works with larger allocations, right?
> > 
> > Our #lowSpaceWatcher seems to be broken because "Smalltalk specialObjectsArray at: 23" is always "nil". Should we replace it with "Processor preemptedProcecss"?
> > 
> > No, it *should* be nil.?? The vm sets it to the active process when it signals the low space semaphore. ??Hence??"Smalltalk specialObjectsArray at: 23" reliably informs the image level low space code which process performed the allocation that tripped the low space trigger.
> > 
> > Has #lowSpaceThreshold any effect these days?
> > 
> > Yes.?? But given that Spur attempts to grow the heap dynamically and contemporary OSs are only too happy to comply, the low space trigger may only be seen to work if a limit on old space size has been installed.?? Otherwise it???ll most likely not trigger before the system starts to page itself into sluggishness.
> > 
> > 
> > Has #primSignalAtBytesLeft: (primitive 125) any effect these days?
> > 
> > Yes.
> > 
> > 
> > Can there ever be an image-level protection against stack-page shortage? VM parameter 42 seems to be 50 (num stack pages), which lasts for quite some time in endless recursion ... but CMD+Dot stops working at some point because of some stack-growing issues? And then the VM crashes eventually.
> > 
> > Again that???s not how things work.?? The stack zone is fixed size.?? When a new stack page is needed the oldest page is evacuated to the heap in the form of Context objects.?? So infinite recursion (as it always did) causes heap growth, and the issue of sluggishness did to paging comes to the fore.
> > 
> > 
> > 
> > Best,
> > Marcel
> > 
> > 
> > 
> > 
> > --
> > 
> > _,,,^..^,,,_
> > 
> > best,??Eliot
> > ???'From Squeak6.1alpha of 12 July 2022 [latest update: #22122] on 13 July 2022 at 11:00:14 am'!

!SmalltalkImage methodsFor: 'memory space' stamp: 'mt 7/13/2022 10:58'!
lowSpaceWatcher
	"Wait until the low space semaphore is signalled, then take appropriate actions."

	| free preemptedProcess |
	self garbageCollectMost <= self lowSpaceThreshold
		ifTrue: [self garbageCollect <= self lowSpaceThreshold
				ifTrue: ["free space must be above threshold before
					starting low space watcher"
					^ Beeper beep]].

	Smalltalk specialObjectsArray at: 23 put: nil.  "process causing low space will be saved here"
	LowSpaceSemaphore := Semaphore new.
	self primLowSpaceSemaphore: LowSpaceSemaphore.
	self primSignalAtBytesLeft: self lowSpaceThreshold.  "enable low space interrupts"

	LowSpaceSemaphore wait.  "wait for a low space condition..."

	self primSignalAtBytesLeft: 0.  "disable low space interrupts"
	self primLowSpaceSemaphore: nil.
	LowSpaceProcess := nil.

	"The process that was active at the time of the low space interrupt."
	preemptedProcess := Smalltalk specialObjectsArray at: 23.
	Smalltalk specialObjectsArray at: 23 put: nil.
	
	"Note: user now unprotected until the low space watcher is re-installed"

	self memoryHogs isEmpty
		ifFalse: [free := self bytesLeft.
			self memoryHogs
				do: [ :hog | hog freeSomeSpace ].
			self bytesLeft > free
				ifTrue: [ ^ self installLowSpaceWatcher ]].

	Preferences logDebuggerStackToFile ifTrue: [
		self
			logError: 'Space is low'
			inContext: preemptedProcess suspendedContext
			to: 'LowSpaceDebug.log'].
			
	preemptedProcess
		debugWithTitle: 'Space is low'
		full: false
		contents: self lowSpaceChoices.
! !

!SmalltalkImage methodsFor: 'memory space' stamp: 'mt 7/13/2022 10:34'!
signalLowSpace
	"Signal the low-space semaphore to alert the user that space is running low. Treat the active process as the culprit. See OutOfMemory and its uses."

	Smalltalk specialObjectsArray at: 23 put: Processor activeProcess.
	LowSpaceSemaphore signal.! !
> > 
> 
> 


More information about the Squeak-dev mailing list