[Vm-dev] Techniques for debuggin plugins

[David T. Lewis]

On Thu, Apr 03, 2014 at 01:44:42PM -0700, gettimothy wrote:
>  
> Howdy all.
> 
> I will be debugging the SqueakSSL plugin on Linux tomorrow or so and am planning on using DDD to connect to a running process and see what I can see.
> 
> Is this the approach you folks would use? or are there better methods for approaching this.
> 

Sure, connecting the debugger to the running VM process will let you debug the primitive.
Compile it with CFLAGS=-g (debugger symbols, no compiler optimization). It can be tricky
because the primitive function will not be visible until the plugin has been loaded, so
you can't set the breakpoint until after the plugin module is loaded, which can be annoying
if the primitive causes the VM to crash.

Here is another debugging technique, but I have to warn you that it is a super top secret
technique known only to VM gurus, so don't let anybody know I told you about this. After
you generate the sources for the plugin, put some printf's in the source code to print
out the values of the variables or addresses or object pointers that you think may be
causing a problem. It's crude, but sometimes this is easier than trying to capture the
problem in a debugger.

If you are debugging a plugin that works when compiled for 32-bits and crashes the VM
when compiled for 64-bits, the problems are usually associated with somebody trying to
stuff a 64-bit C pointer into a 32-bit sqInt variable. Look for this kind of problem,
and print the object pointers and variables with "%lx" in the printf statements.

Put a "fflush(stdout)" after your printf statements to make sure your debugging output
gets flushed out to the console right away.

Dave