Alexander

this is part of the absence of ressources. Now how do we fix it?

Stef

One part of this issue was that (at least for me, on a no-grow-OM VM) the recursion would use up all memory in less than the time I could move my hands fomr the cmd-d to the interrupt. The apparent non-responsiveness was _not_ because the interrupt was being ignored but becasue the system had already crashed and was spending a long time trying to write out the low space debug log.

I can't find much trace of emails on the subject but I think the low space signal was being whacked and for whatever reason the scheduler wasn't getting around to dealing with it in a way that actually did anything helpful. IT seems that things went wrong somewhat differently on different OSs as well. Dave's change stopped the problem from killing the system.

My brief testing was pretty much the only review it got and if nobody else with more time and other hardware could find time to look at it, well that is the cost of volunteer development. Quite possibly it shouldn't have been harvested without more input but nobody can come along and twist your arm (for example) to look at things.

tim -- Tim Rowledge, tim@sumeru.stanford.edu, http://sumeru.stanford.edu/tim The computing field is always in need of new cliches. - Alan Perlis

On Thu, Mar 31, 2005 at 06:27:02PM -0800, Andreas Raab wrote:

Hi David -

Thanks for the history. Is there any post with a thorough analysis of the sequence of events that made things go wrong, and consequently, why changing the code the way you did would address this the problem? I'm asking, because I am still not clear about why a change in this particular place would make any difference on the low space watcher behavior whatsoever (neither did the debugging facilities you sent along seem to be related to Project>>interruptName:).

Hi Andreas,

I'm afraid that my memory is not too clear on the details at this point, other that what I pulled out of the BFAV postings. There were a few other messages, but the heart of it was what I sent to you.

If you can wait until this weekend, I will spend some time figuring out where this "fix" went wrong, and hopefully send you a more helpful response (I have been looking at it this evening, but I am out of time and energy until the weekend).

<speculation> I think there is some difference in the way the low space semaphore is being handled, versus the way the user interrupt semaphore is being handled. Obviously I mis-diagnosed it, and I also did not consider the impact on the user interrupt handler. Perhaps the real problem is related to "Debugger openInterrupt: labelString onProcess: preemptedProcess" not being able to open the debugger in time to stop the runaway recursion, but that's just a guess at this point (and obviously my guesswork has been less than satisfactory). </speculation>

I believe it is true that the VM crash will happen only if: - VM is running with fixed memory allocation (hence you will not see it on the Windows VM). - Current project is Morphic (MVC is OK). - Background processes (e.g. event tickler) are running at higher priority than the UI process.

With respect to the debugging facilities that I sent, the output on stdout (console) for Squeak running a Unix VM with fixed memory allocation looks like this (note, I added one additional debugging line here to indicate when the debugger is about to be opened, hence my speculation above):

lewis@dtlewis:~/squeak/squeak3.7> squeak -memory 10m Dev 3048:2445:Project class>>interruptName::about to open debugger with label Space is low 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 0 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 1 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 2 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 3 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 4 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 5 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 6 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 7 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 8 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 9 2561:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 10 2561:3864:SystemDictionary>>createStackOverflow::terminate recursion at depth: 10

More later,

Dave

Andreas Raab andreas.raab@gmx.de wrote:

Yes. Easily so. The "major safeguard" that Tim referred to is actually a

Not me so far as I can tell...

If this is really the problem, then the only true solution I can see is to modify the VM so that we have enough information available to figure out what the preempted process was at the time that particular signal (user interrupt or low space) happened[***]. Note that the information might be indirect - if, for example, we could measure how long a process was running *without* giving up control to a lower priority process (e.g., actively yielding or waiting on a semaphore) this should already be enough to identify the "right" guy to interrupt.

Well we know the active context of course and for a little more work we know the active process, so the info is there. The amusing part is that the semaphore used for low space has the lowspace process on its list and we remove it as part of signalling. Maybe if we stuck the active process on the end as some 'helpful context' it would make life simpler. Or a subclass of Semaphore with a more explicit ivar(s) if we want to reduce confusion potential.

tim -- Tim Rowledge, tim@sumeru.stanford.edu, http://sumeru.stanford.edu/tim The colder the X-ray table, the more of your body is required on it.

Andreas Raab andreas.raab@gmx.de wrote:

Tim Rowledge wrote:

...

Andreas Raab andreas.raab@gmx.de wrote:

...

Yes. Easily so. The "major safeguard" that Tim referred to is actually a

Not me so far as I can tell...

Then this must be a clear case of identity theft ;-))

http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-May/078325.html

Golly. That _was_ a long time ago. But yes, I said it and I think in context I stand by it.

...

The amusing part is that the semaphore used for low space has the lowspace process on its list and we remove it as part of signalling. Maybe if we

stuck

...

the active process on the end as some 'helpful context' it would make life simpler. Or a subclass of Semaphore with a more explicit ivar(s) if we want

to

...

reduce confusion potential.

I think it's a little more complex than that.

Of course, no argument there. I was merely musing on a plausible trivial VM mechanism for passing up some helpful stuff to avoid try to infer the appropriate process to beat up. When we are signalling a very specific smeaphore like lowspace we can be quite certain that the active process when the allocation code gets upset is the one we want to deal with, right now, no excuses.

The user interrupt is a little different in that it is easy to say "stop the active process" but that isn't always what one really wants to do. As you mentioned earlier, sometimes one will get the weak finalization process, sometime the event tickler, or the main event loop etc. Generally in my experience I use the interrupt to try to stop the main user process to try to see WTF it is up to and why it hasn't done anything visible in a while. It would be nice to have some way to pop up a list of current processes and choose one to play with. I realise the process browser does some of that. I've noticed in Tweak that it is quite hard to get to interrupt anything except the event loop which rarely seems to lead me anywhere interesting when trying to debug TK4 stuff.

tim -- Tim Rowledge, tim@sumeru.stanford.edu, http://sumeru.stanford.edu/tim Useful Latin Phrases:- Mihi ignosce. Cum homine de cane debeo congredi = Excuse me. I've got to see a man about a dog.

On Fri, Apr 01, 2005 at 12:34:55AM -0800, Andreas Raab wrote:

I think it's a little more complex than that. It would be nice if we could define a few reliable test cases to spec out what it actually is that we think ought to work. I'll throw in the following:

• "[true] whileTrue" - we must be able to interrupt this
• "[[true] whileTrue] forkAt: Processor userSchedulingPriority + 1" -

again we must be able to interrupt that

• "Smalltalk stackOverflow" - we must be able to recover from that
• "[Smalltalk stackOverflow] forkAt: Processor userSchedulingPriority +

1" - again we must be able to recover from that

If there are more, please add.

Some variations to add the above list:

1) Each case must be interruptable in Morphic and in MVC. 2) Each case must be interruptable either with or without the event tickler running (a high priority process that may become schedulable while a low space condition is being detected in the VM). 3) Each case must be interruptable with a fixed size object memory, or with a dynamically expanding object memory (varies by platform, "-memory 10m" flag for a Unix VM). 4) Each case must be interruptable if the user hits the interrupt key multiple times in succession.

Variation #4 fails in Morphic for the "[true] whileTrue" tests going back to at least Squeak 3.6.

The "[[true] whileTrue] forkAt: Processor userSchedulingPriority + 1" fails in MVC going back to at least Squeak 3.6.

Dave

On Fri, Apr 01, 2005 at 06:44:35AM -0500, David T. Lewis wrote:

Thanks Andreas,

I will run some tests this weekend and see if I can confirm which process is being interrupted and why. One additional data point I can add is that the condition is 100% repeatable. That is, with the combination of Unix VM + fixed memory allocation + Morphic

• original #interruptName method + runaway recursion test, the

result will always be a VM crash. This should make it much easer to debug.

One additional hypothesis: If you were to disable the "grow object memory" capability within a Win32 VM, I would expect to see the same VM crash behavior.

I'll post any results by this Sunday.

As promised, here are some test results for the low memory handler problem. All tests were run on a Squeak 3.7 image with Unix VM set for fixed size object memory to force the out-of-memory condition. I used OSProcess and output tracing to the console in order to see what is happening in the interrupt handlers. The bad fix for #interruptName: is *not* applied in any of these tests, so this reflects behavior of the system prior to applying the bad fix. Some of the results are rather self-evident, but are included for completeness. For reference, I've also attached a change set with the hacks that I used for debugging.

Bottom line for the impatient: Andreas' analysis is correct, modulo a few details pertaining to MVC.

In Morphic, if I terminate the event tickler and start a dummyTickler process instead, I get a runaway overflow (VM crash) just as if the event tickler were running. If there is no event tickler, then no crash occurs.

dummyTickler _ [ [true] whileTrue: [(Delay forMilliseconds: 500) wait] ] forkAt: Processor lowIOPriority. Smalltalk createStackOverflow

If I do the same thing, but give the dummyTickler a longer time delay, then there is no failure. Presumably the dummyTickler was not ready to be scheduled, hence did not interfere with low space handling. This further supports the hypothesis that the event tickler is being incorrectly treated as the runaway process.

dummyTickler _ [ [true] whileTrue: [(Delay forMilliseconds: 50000) wait] ] forkAt: Processor lowIOPriority. Smalltalk createStackOverflow

In MVC, the following does *not* result in a VM crash. The low space watcher works fine despite having this process running. Also, if the event tickler process is terminated, this still works properly in MVC. In other words, in MVC we *always* handle the low space semaphore correctly (but see below), regardless of whether the event tickler or the dummyTickler (or both) is running.

dummyTickler _ [ [true] whileTrue: [(Delay forMilliseconds: 500) wait] ] forkAt: Processor lowIOPriority. Smalltalk createStackOverflow

Note however, in MVC if the stack overflow occurs in a background process, things go horribly wrong in ways I will not even try to describe.

[Smalltalk createStackOverflow] fork

I used OSProcess output tracing to identify the prempted process at the time of low space interrupt handling. In *both* Morphic and MVC, the event tickler is the preempted process (when the event tickler is running of course). See console trace logs below for the actual output.

This seems to bring us right back to the implementation of #interruptName:. There is an MVC implementation in ControlManager>>interruptName:, and a Morphic implementation in Project class>>interruptName:. The MVC implementation apparently handles the low space interrupt correctly, and the Morphic implementation does not.

What is different? The MVC implemention does not make any reference to the preempted process when figuring out which process to suspend. Instead it always suspends the ScheduledControllers activeControllerProcess process (see below, possibly this is why in MVC we cannot interrupt a high priority background process).

All of this supports Andreas' hypothesis that, when one process triggers the low memory semaphore, the higher priority event tickler process has become runnable by the time garbage collection completes and the semaphore has been signalled.

So this still leaves us with the problem of how to figure out which process actually was active at the time the low space interrupt was generated.

Just for completeness, in Morphic, the following *does not* hang the image, but in MVC it *does* hang the image.

[[true] whileTrue:[]] forkAt: Processor userSchedulingPriority+1.

So MVC apparently does not know how to handle either the low memory semaphore or the user interrupt semaphore if the offending process is running in background, separate from the main UI controller scheduling.

Following are copies of the console output obtained when running #createStackOverflow in both Morphic and MVC. In both cases, the #createStackOverflow method has been modified to produce console output, and to force termination if the method has been called more than 10 times after the low space semaphore was signalled. For each line in the trace output, the first number identifies the active process (the second number identifies the message receiver).

In Morphic the console output is:

lewis@dtlewis:~/squeak/squeak3.7> squeak -memory 7m squeak.7 1319:3840:UndefinedObject>>DoIt:about to run createStackOverflow from a workspace 430:3864:SystemDictionary>>lowSpaceWatcher:low space semaphore signal received, preempted process is [] in EventSensor>>eventTickler {[delay wait. delta := Time millisecondClockValue - lastEventPoll. (delta <...]} BlockContext>>on:do: [] in EventSensor>>installEventTickler {[self eventTickler]} [] in BlockContext>>newProcess {[self value. Processor terminateActive]}

430:3864:SystemDictionary>>lowSpaceWatcher:about to display low space notifier 430:2445:Project class>>interruptName::entering #interruptName, preempted process is [] in EventSensor>>eventTickler {[delay wait. delta := Time millisecondClockValue - lastEventPoll. (delta <...]} BlockContext>>on:do: [] in EventSensor>>installEventTickler {[self eventTickler]} [] in BlockContext>>newProcess {[self value. Processor terminateActive]}

1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 0 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 1 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 2 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 3 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 4 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 5 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 6 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 7 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 8 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 9 1319:3864:SystemDictionary>>createStackOverflow::low space signal already received, continuing with recursion depth: 10 1319:3864:SystemDictionary>>createStackOverflow::terminate recursion at depth: 10 3886:3864:SystemDictionary>>lowSpaceWatcher:restarted low space watcher 3886:3864:SystemDictionary>>lowSpaceWatcher:install low space semaphore 3886:3864:SystemDictionary>>lowSpaceWatcher:enable low space interrupts 3886:3864:SystemDictionary>>lowSpaceWatcher:wait on low space semaphore 2954:3864:SystemDictionary>>lowSpaceWatcher:restarted low space watcher 2954:3864:SystemDictionary>>lowSpaceWatcher:install low space semaphore 2954:3864:SystemDictionary>>lowSpaceWatcher:enable low space interrupts 2954:3864:SystemDictionary>>lowSpaceWatcher:wait on low space semaphore

In MVC the console output is:

lewis@dtlewis:~/squeak/squeak3.7> squeak -memory 7m squeak.7 1898:3840:UndefinedObject>>DoIt:about to run createStackOverflow from a workspace 2069:3864:SystemDictionary>>lowSpaceWatcher:low space semaphore signal received, preempted process is [] in EventSensor>>eventTickler {[delay wait. delta := Time millisecondClockValue - lastEventPoll. (delta <...]} BlockContext>>on:do: [] in EventSensor>>installEventTickler {[self eventTickler]} [] in BlockContext>>newProcess {[self value. Processor terminateActive]}

2069:3864:SystemDictionary>>lowSpaceWatcher:about to display low space notifier 2069:347:ControlManager>>interruptName::entering #interruptName, preempted process is [] in EventSensor>>eventTickler {[delay wait. delta := Time millisecondClockValue - lastEventPoll. (delta <...]} BlockContext>>on:do: [] in EventSensor>>installEventTickler {[self eventTickler]} [] in BlockContext>>newProcess {[self value. Processor terminateActive]}

1794:3864:SystemDictionary>>lowSpaceWatcher:restarted low space watcher 1794:3864:SystemDictionary>>lowSpaceWatcher:install low space semaphore 1794:3864:SystemDictionary>>lowSpaceWatcher:enable low space interrupts 1794:3864:SystemDictionary>>lowSpaceWatcher:wait on low space semaphore

Dave