Hi Eliot, thank you for the analysis. The question is why the code works with the VM, but not if we step through it. What does the VM do that makes this behave correctly, although the temps-array is removed from the stack prior to returning from #contextOn:do:?

I ask because I need to recreate that behaviour in case it also appears in other places. The general project is creating a squeak research VM. That is also why I try to refrain from changing the image, retaining compatibility with Pharo/Squeak trunk.

Thank you, Lars

2013/05/17 8:08 pm Eliot Miranda eliot.miranda@gmail.com:

Hi Lars,

On Fri, May 17, 2013 at 9:36 AM, Lars Wassermann <lars.wassermann@googlemail.com mailto:lars.wassermann@googlemail.com> wrote:

Hello VM-dev,
I habe problems understanding what exactly happens when I step
through:
(ContextPart contextOn: Error do: []) halt
in a workspace.
When I step over the parenthesis in the debugger, everything works
kind of like expected (the context of contextOn:do: misses one
temporary but the returned value is what the methods description
promised). When I step into or through the code, an error is
raised when executing bytecode 54 of #contextOn:do:, because the
temps-array illegally has been poped (by bytecode 53).

My question now is:
Why does the normal vm not only break when runing this code, but
also return the correct value (the first element of the last
temporary).
I appended a graphic trying to visualize what happens up to the
miracle of three skipped bytecodes.

Ugh. I hate this method. Actually I hate ContextPart>>jump, which is the really evil method.

So let's take a look at the methods:

ContextPasrt class>>contextOn: exceptionClass do: block "Create an #on:do: context that is ready to return from executing its receiver"

| ctxt chain | ctxt := thisContext. [chain := thisContext sender cut: ctxt. ctxt jump] on: exceptionClass do: block. "jump above will resume here without unwinding chain" ^ chain

answers a context that is an activation of on:do: ready to catch the supplied exception. It is used by ContextPart>>runUntilErrorOrReturnFrom: to insert an exception handler for UnhandledError immediately beneath the current context being executed by the debugger.

The on:do: argument is a little unclear. It is easier to understand if it reads

ContextPasrt class>>contextOn: exceptionClass do: block "Create an #on:do: context that is ready to return from executing its receiver"

| ctxt onDoContext | ctxt := thisContext. [onDoContext := thisContext sender. onDoContext cut: ctxt. ctxt jump] on: exceptionClass do: block. "jump above will resume here without unwinding chain" ^ onDoContext

So the idea is to run until in the on:do:'s receiver (the [onDoContext := ...] block), grab the sender (the on:do: activation), remove intervening contexts (onDoContext cut: ctxt) so that on:do:'s handlerActive doesn't get set, resume executing in the contextOn:do: context (ctxt jump) and answer the onDoContext.

The horror is in ContextPart>>jump:

ContextPart>>jump "Abandon thisContext and resume self instead (using the same current process). You may want to save thisContext's sender before calling this so you can jump back to it. Self MUST BE a top context (ie. a suspended context or a abandoned context that was jumped out of). A top context already has its return value on its stack (see Interpreter>>primitiveSuspend and other suspending primitives). thisContext's sender is converted to a top context (by pushing a nil return value on its stack) so it can be jump back to."

| top | "Make abandoned context a top context (has return value (nil)) so it can be jumped back to" thisContext sender push: nil.

"Pop self return value then return it to self (since we jump to self by returning to it)" stackp = 0 ifTrue: [self stepToSendOrReturn]. stackp = 0 ifTrue: [self push: nil]. "must be quick return self/constant" top := self pop. thisContext privSender: self. ^ top

This works by returning to self (a MethodContext), but return pushes a result. So the code pops the top thing off the stack and returns it. This is evil. If the stack is empty then this will pop the receiver, return it and push it back in the return (!!). If the stack contains only temps then the last temp is popped and pushed by the return (!!). This is what you're seeing. The stack's last temp is an indirection vector. It gets popped off the stack by top := self pop. at which point any attempt to look at self's temps in a debugger will cause an error because the indirection vector is no longer there. It then gets pushed back by= ^ top

Horrible. It all depends on being able to create a temporary invalid execution state in a context.

A better way to do this is by process switch, avoiding the return. For example, in my basic block profiler I handle an unknownBytecode error caused by executing the unknownBytecode that overwrites the bytecode at the beginning of each basic block, replaces the unknownBytecode with the correct bytecode, and continues:

ContextPart>>unusedBytecode "Handle unusedBytecode by replacing the bytecode with the correct one found in the coverage property and continuing. Continue via wait/signal since return would push a result." | coverage semaphore process | self assert: (method at: pc) = method encoderClass unusedBytecode. coverage := method propertyValueAt: #coverage. self assert: coverage notNil. self assert: (coverage includesKey: pc). semaphore := Semaphore new. process := Processor activeProcess.

[method at: pc put: (coverage removeKey: pc). process suspendedContext unwindTo: self. process suspendedContext: self. semaphore signal] fork.

semaphore wait

So that could become something like

jump "Abandon thisContext and resume self instead (using the same current process). You may want to save thisContext's sender before calling this so you can jump back to it. Self MUST BE a top context (ie. a suspended context or a abandoned context that was jumped out of). A top context already has its return value on its stack (see Interpreter>>primitiveSuspend and other suspending primitives)." | semaphore process | semaphore := Semaphore new. process := Processor activeProcess.

[process suspendedContext unwindTo: self. process suspendedContext: self. semaphore signal] fork.

semaphore wait

but perhaps the process switch will introduce other problems. I don't know.

Hope this helps.

I replicated the behavior in Squeak 4.0 and 4.4 images on
interpreter VMs 3.7.7, 4.10, and a newer cog VM.


A fix to the problem of stepping and comprehension as well as the
missing temp in the frame would be changing the method to:

contextOn: exceptionClass do: block
        "Create an #on:do: context that is ready to return from
executing its receiver"

        | ctxt chain |
        ctxt := thisContext.
        [chain := thisContext sender cut: ctxt.
        ctxt push: nil. ctxt jump] on: exceptionClass do: block.
        "jump above will resume here without unwinding chain"
        ^ chain

The difference is that after cutting off ctxt, we ensure that it
is a top context by pushing a meaningless value. But this still
doesn't help me understanding the original behaviour.

Thank you and all the best
Lars

-- best, Eliot

On 05/19/2013 07:29 PM, Eliot Miranda wrote:

On Sat, May 18, 2013 at 3:49 PM, Lars <lars.wassermann@googlemail.com mailto:lars.wassermann@googlemail.com> wrote:

Hi Eliot,
thank you for the analysis.
The question is why the code works with the VM, but not if we step
through it. What does the VM do that makes this behave correctly,
although the temps-array is removed from the stack prior to
returning from #contextOn:do:?

the vm doesn't do anything special (* with one exception) because even though the image code pops off the temps array the return pushes it back almost immediately, before the temps array is used.

Well, the temps array is poped by #jump, then pushed back due to the return, ... but then poped again because the returned to context is not a top context. It's pc points to the pop bytecode (54?) which is there because the return value of the #on:do: message send in #contextOn:do: is not used. And at this point, the stepwise debugging breaks. But the VM just skips over it and returns with the correct value. Why?

• the exception is that when a context is created for a stack frame the

code miust handle the fact that the stack may be empty and so the proxy context must have a nil where the missing stack entry is, not an uninitialized field. See the comment in updateStateOfSpouseContextForFrame:WithSP:.

I ask because I need to recreate that behaviour in case it also
appears in other places. The general  project is creating a squeak
research VM. That is also why I try to refrain from changing the
image, retaining compatibility with Pharo/Squeak trunk.

Thank you, Lars

2013/05/17 8:08 pm Eliot Miranda <eliot.miranda@gmail.com>
<mailto:eliot.miranda@gmail.com>:

...

Hi Lars,

On Fri, May 17, 2013 at 9:36 AM, Lars Wassermann
<lars.wassermann@googlemail.com
<mailto:lars.wassermann@googlemail.com>> wrote:


    Hello VM-dev,
    I habe problems understanding what exactly happens when I step
    through:
    (ContextPart contextOn: Error do: []) halt
    in a workspace.
    When I step over the parenthesis in the debugger, everything
    works kind of like expected (the context of contextOn:do:
    misses one temporary but the returned value is what the
    methods description promised). When I step into or through the
    code, an error is raised when executing bytecode 54 of
    #contextOn:do:, because the temps-array illegally has been
    poped (by bytecode 53).

    My question now is:
    Why does the normal vm not only break when runing this code,
    but also return the correct value (the first element of the
    last temporary).
    I appended a graphic trying to visualize what happens up to
    the miracle of three skipped bytecodes.


Ugh.  I hate this method.   Actually I hate ContextPart>>jump,
which is the really evil method.

So let's take a look at the methods:

ContextPasrt class>>contextOn: exceptionClass do: block
"Create an #on:do: context that is ready to return from executing
its receiver"

| ctxt chain |
ctxt := thisContext.
[chain := thisContext sender cut: ctxt. ctxt jump]
on: exceptionClass
do: block.
"jump above will resume here without unwinding chain"
^ chain

answers a context that is an activation of on:do: ready to catch
the supplied exception.  It is used by
ContextPart>>runUntilErrorOrReturnFrom: to insert an exception
handler for UnhandledError immediately beneath the current context
being executed by the debugger.

The on:do: argument is a little unclear.  It is easier to
understand if it reads

ContextPasrt class>>contextOn: exceptionClass do: block
"Create an #on:do: context that is ready to return from executing
its receiver"

| ctxt onDoContext |
ctxt := thisContext.
[onDoContext := thisContext sender.
 onDoContext cut: ctxt.
 ctxt jump]
on: exceptionClass
do: block.
"jump above will resume here without unwinding chain"
^ onDoContext

So the idea is to run until in the on:do:'s receiver (the
[onDoContext := ...] block), grab the sender (the on:do:
activation), remove intervening contexts (onDoContext cut: ctxt)
so that on:do:'s handlerActive doesn't get set, resume executing
in the contextOn:do: context (ctxt jump) and answer the onDoContext.

The horror is in ContextPart>>jump:

ContextPart>>jump
"Abandon thisContext and resume self instead (using the same
current process).  You may want to save thisContext's sender
before calling this so you can jump back to it.
Self MUST BE a top context (ie. a suspended context or a abandoned
context that was jumped out of).  A top context already has its
return value on its stack (see Interpreter>>primitiveSuspend and
other suspending primitives).
thisContext's sender is converted to a top context (by pushing a
nil return value on its stack) so it can be jump back to."

| top |
"Make abandoned context a top context (has return value (nil)) so
it can be jumped back to"
thisContext sender push: nil.

"Pop self return value then return it to self (since we jump to
self by returning to it)"
stackp = 0 ifTrue: [self stepToSendOrReturn].
stackp = 0 ifTrue: [self push: nil].  "must be quick return
self/constant"
top := self pop.
thisContext privSender: self.
^ top

This works by returning to self (a MethodContext), but return
pushes a result.  So the code pops the top thing off the stack and
returns it.  This is evil.  If the stack is empty then this will
pop the receiver, return it and push it back in the return (!!).
 If the stack contains only temps then the last temp is popped and
pushed by the return (!!).  This is what you're seeing.  The
stack's last temp is an indirection vector.  It gets popped off
the stack by
top := self pop.
at which point any attempt to look at self's temps in a debugger
will cause an error because the indirection vector is no longer
there.  It then gets pushed back by=
^ top

Horrible.  It all depends on being able to create a temporary
invalid execution state in a context.

A better way to do this is by process switch, avoiding the return.
 For example, in my basic block profiler I handle an
unknownBytecode error caused by executing the unknownBytecode that
overwrites the bytecode at the beginning of each basic block,
replaces the unknownBytecode with the correct bytecode, and continues:

ContextPart>>unusedBytecode
"Handle unusedBytecode by replacing the bytecode with the
 correct one found in the coverage property and continuing.
 Continue via wait/signal since return would push a result."
| coverage semaphore process |
self assert: (method at: pc) = method encoderClass unusedBytecode.
coverage := method propertyValueAt: #coverage.
self assert: coverage notNil.
self assert: (coverage includesKey: pc).
semaphore := Semaphore new.
process := Processor activeProcess.

[method
at: pc
put: (coverage removeKey: pc).
 process suspendedContext unwindTo: self.
 process suspendedContext: self.
 semaphore signal] fork.

semaphore wait

So that could become something like

jump
"Abandon thisContext and resume self instead (using the same
current process).  You may want to save thisContext's sender
before calling this so you can jump back to it.
Self MUST BE a top context (ie. a suspended context or a abandoned
context that was jumped out of).  A top context already has its
return value on its stack (see Interpreter>>primitiveSuspend and
other suspending primitives)."
| semaphore process |
semaphore := Semaphore new.
process := Processor activeProcess.

[process suspendedContext unwindTo: self.
 process suspendedContext: self.
 semaphore signal] fork.

semaphore wait

but perhaps the process switch will introduce other problems.  I
don't know.

Hope this helps.


    I replicated the behavior in Squeak 4.0 and 4.4 images on
    interpreter VMs 3.7.7, 4.10, and a newer cog VM.


    A fix to the problem of stepping and comprehension as well as
    the missing temp in the frame would be changing the method to:

    contextOn: exceptionClass do: block
            "Create an #on:do: context that is ready to return
    from executing its receiver"

            | ctxt chain |
            ctxt := thisContext.
            [chain := thisContext sender cut: ctxt.
            ctxt push: nil. ctxt jump] on: exceptionClass do: block.
            "jump above will resume here without unwinding chain"
            ^ chain

    The difference is that after cutting off ctxt, we ensure that
    it is a top context by pushing a meaningless value. But this
    still doesn't help me understanding the original behaviour.

    Thank you and all the best
    Lars


--
best,
Eliot

-- best, Eliot