As I begin to grasp the scope of the optimizations in the interpreter and C-Translator, I am filled with a sense of awe. These tweaks were obviously done with a deep knowledge of Smalltalk, C, the behavior of C compilers, and assembler. At the same time, I'm filled with horror as I realize the people who wrote these must of have sold their souls, left kidneys, and first born offspring in order to pull it off.
While the VM, as it exists today, is adequate for conventional Squeak as it is presently used. Unfortunately, however, the scope and nature of its optimizations make it difficult to modify and extend. The non-obvious things the C-translator does make VM hacking extremely inhospitable to newcomers. For example, I would look at the class definition and would think to myself: OK, these "instance" variables will be put into a structure that can be instantiated by appropriate calls to C, and the class variables will probably end up being static variables in the same source file. -- thereby approximating the behavior of Smalltalk classes.
Instead, class variables are optimized as constants, and while instance variables are put in a structure, there is no way to instantiate the structure or manage such instances should they be created. Much much worse, however, is that the C translator will behave differently depending on which class it is processing and apply special hacks where it thinks nobody will notice. -- The hapless newbie probably won't discover these hacks until he has read at least 500 messages...
I'm going to try to fork my own version of the vmmaker intended to be much more flexible and robust to experimentation though, perhaps, a bit slower... (The current version of my custom VM is half as fast but still usable...)
The current issue that I'm having trouble with is figuring out exactly how the CCode generator interfaces with the system wide compiler. I am especially curious as to how to implement scope-reduced variables. (variables declared within a block instead of a function...)
How exactly are syntax elements mapped onto the translation classes?
In any event, I hope to have an interesting variation on the VM technology someday...