Adding Modules to Squeak; a modest proposal

Joerg Beekmann jbeekmann at attglobal.net
Mon Mar 12 04:25:32 UTC 2001


Normally I would have tried to get a bit more working before sharing this,
but as Modularity is such a hot topic right I thought I'd go ahead and brave
it. I'll start by saying I believe a modular Squeak is possible within the
context of the current Squeak community (at least nothing I've seen so far
tells me otherwise). The attached document is my vision of what the means
and how it can be done. Small summary is below.

Joerg

-----------------------------------------------------
Joerg Beekmann
jbeekmann at attglobal.net


Document summary
================
The attachment proposes a Module infrastructure for Squeak. The goal of this
proposal is to come up with a system that will allow the modularization of
Squeak "in-situ" and over time. Modules are similar to Environments in that
they define a name space. Unlike Environments the track their own version
and prerequisite versions. Also unlike Environments Modules implement a flat
multi source import scheme rather than a deep single inheritance based
import scheme. I see four challenges to getting this done: 1) impementing
Module class extenstions; 2) Making the tools Module aware, even figuring
out how they should behave; 3) Modularizing Morphic and all that goes with
that. As I mention I'd like to see Morphic hooked into the primative
graphics and event mechanism as just another UI; 4) Obtaining consensus that
this should be done at all.

Additional feature seen as required are:
** Module based class extensions
** Automated Module loading using the image segmentation technology
** A very simple notion of "Module Path"
** Module based sources files
** Notion of "runable" for modules.
** Support for standard I/O

Table of Contents
=================
0) Basic Ideas
I) Basic modules
II) Module versioning
III) Module specific class extensions
IV) Module loading
V) Modules have a standard startUp method
VI) Potential image partitionings
VII) The plan
VIII) The vision && Dream
IXX) Status
IX) Some rational for decision.
X) Ideas considered but rejected
XI) Relationship to existing modular Squeak work
XII) The biggest challenges
-------------- next part --------------
Proposal for adding Modules to Squeak
===========================
Author: Joerg Beekmann <jbeekmann at attglobal.net>
Version: 0.1.0
Date: March 11, 2001


Table of Contents
=================
0) Basic Ideas
I) Basic modules
II) Module versioning
III) Module specific class extensions
IV) Module loading
V) Modules have a standard startUp method
VI) Potential image partitionings
VII) The plan
VIII) The vision && Dream
IXX) Status
IX) Some rational for decision.
X) Ideas considered but rejected
XI) Relationship to existing modular Squeak work


0) Basic Ideas
==============



I) Basic modules:
=================
** The system is partitioned into modules and each class in a module looks up global symbols in its Module. In this way modules are very much like Environments. 
** All modules are maintained in a ModuleRegistry defined in the root module 'Smalltalk'.
** Each module has a name and it would be nice if this were unique in the Squeak world.
** Each module is a subclass of Module and is defined via a special subclassing message that lists imports and exports.
** Modules declare the symbols they export or may specify to export all symbols. Exports are a list of Smalltalk symbols, which must be defined by the module.
** Modules may import any number of other modules. Imports are defined as a list of ModuleDescriptions that are resolved to modules at symbol lookup time.
** I'm assuming lookup performance is not critical. It is done at compile time and occasionally during runtime.
** If a module is imported all symbols that module exports are imported.
** Importing a module DOES NOT import its imports. (Unlike Environments no hierarchy of modules). 
** Mutual dependence between two modules should be allowed unless we can't figure out how to load such a cycle (will need to allow this initially in any case until the image is partitioned).
** Order of import makes no difference and search order is undefined (although it makes sense to search locally first).
** Same symbol may not resolve to two values, either in two imports or locally and in an import (no shadowing). 
** Respect Smalltalk philosophy of allowing user to create inconsistent image but provide tools to clean up.... undefined, doubly defined etc.
** Each module has programmatic access to the registry via the message 'registry' and thus all symbols.


II) Module versioning
=====================
** Each module has a versions and earliest compatible version. 
** Each ModuleDescription has a versions and a flag indicating if that exact module is required or any compatible module is acceptable.
** Versions are in three parts major.minor.patch and are Magnitudes
** When looking up a symbol in the imports look for the module in the registry. This is a module of the correct name compatible with the Version requested.
** Two versions of the same package may co-exist in image.


III) Module specific class extensions
=====================================
** Classes may be extended with methods by modules in which they are not defined. This provides a home for methods such as isMorph.
** The methods a class understands consists of the methods defined in it and its super classes as well as the extensions defined in the module corresponding to the sender (that is the senders view of the object).
** Importing a module DOES import extensions it imports or defines. This is required to ensure that derivations of imported classes function correctly. For example module MyViewers imports module MorphicViewers that imports Morphic.  Module MyViewers uses ObjectView defined in MorphicViewers to view morphs and non-morphs. Module Morphic extended Object with the message isMorph module MyViewers needs to import this extension in order for ObjectViewer >> isViewingAMorph invoked from module MyViewers to respond correctly. This may seems inconsistent with not importing classes imported by an imported module but its' not. This rule simply ensures that objects behave consistently with the way they behave in an imported module. It does not import additional symbols.
** Look for extension methods after failing (does not understand). i.e. extensions are in addition to existing protocol.


IV) Module loading
===================
** If a module description can't be resolved in the image attempt to load module from VM directory if not present there from the module directory (see below)
** Module filenames are of form name-version.smd (webserver-1.2.65.smd)
** If can't find a file matching the module either prompt or throw an exception depending on if this is a development or runtime image.
** Modules may be loaded form source or binary. 
** Binary modules are loaded using the image segment technology. 
** There is support for each module having its own sources file. The changes file is shared. However it is still possible to combine all sources.
** Binary module files contain their sources file in them (if sources are included).
** Include standard pre and post load methods for each module.
** Squeak modules are shareable. Multiple running VMs may use same copy of modules.
** Modules are self-describing; should be possible to have the VM print a description of a module if it is passed the module name as an argument (squeak.exe -describe module name).
** Note a simple "make module" may do nothing more than startup load modules and save the fully loaded image. This represents a full distribution. Option to combine sources from the various modules into a single sources file?
** Need a central WebDAV based repository of modules & their versions. Searchable using the self-describing capabilities of modules.
** Should allow for signing of modules via same mechanism as signing of Projects. Are projects subclasses of modules?
** Should be able to validate that loaded Modules still as when loaded (need to has only on classes, methods and byte-codes?
** Be able to browse differences between loaded module and source of module, be that from the web or wherever.


V) Modules have a standard startUp method
=========================================
** Modules define a method called run. This is the startup method of a module.
** VM takes module description as a parameter and this can include a path. This path is the module directory
** With WebDAV support this can be an Internet directory so the module and its requisite modules may be anywhere on the internet.


VI) Potential image partitionings
=================================
** Image should be partitioned into functional units not by groups of like classes.
** Should consider breaking some of the larger functional areas into two modules allowing choice in how complete a system someone wants. Especially the Kernel and graphics modules.
** I imagine that some standardized support for communication between modules may be required. Check out Channels as defined by Inferno (successor to Plan9). For example Kernel should put all events into a Channel that will have 0 or more consumers unknown to it. Ned Konz has mentioned other areas where this would be required.
** The base module should contain a functioning Smalltalk image with event dispatch, Segment loading capability and standard I/O support. Should not contain much else, not even a compiler. Everything else should be loadable on top of this. Given the advertised segment loading speeds may not even be a need to create a single image?
** Some module can act as image builders, simply startup, load imports, initialize and save as a single image with a single sources file, no changes.


VII) The plan
=============
** Implement the basic mechanism as described in I.
** Put all of Squeak into a single module and then experiment with adding additional modules.
** Partition Squeak into multiple modules but import and export all symbols to all modules. Play with additional modules.
** As above but play with restricting the exports of some of the squeak modules.
** Implement some more of above especially the ModuleExtensions and binary loading
** Get additional feed back and decide if it was all worth it and should become part of Squeak.


VIII) The vision && Dream
=========================
** Squeak as a base module and a collection of other modules.
** Able to dynamically load all modules required to run a module onto the base module.
** Version requirements are taken care of.
** Works for everything from a command line Smalltalk script interpreter to a full Squeak environment.
** Really dreaming in Technicolor I can imagine a base image & VM acting more like an OS with multiple programs running on them. 


IXX) Status
===========
** Currently I'm very early in all this. I laid out my goals and a rough feature set as above.  
** I've defined the some of the core parts of I) and can replace Smalltalk with a module (big deal).

IX) Some rational for decision.
===============================
** Why not Environments? For several reasons. 
1) I don't believe the model of inheriting name space along a single arbitrarily deep chain is the correct approach. Inheritance is both too complex and insufficiently powerful. The need to resort to compile in "special references" is an indication of lack of expressive power. What I mean by too complex is the inheritance chain itself. Especially if it becomes long.
2) I felt more than namespace partitioning was required (loading unloading, extensions, versions etc.).  
** Others??

X) Ideas considered but rejected
================================
** Ability to import only some subset of an Imports exports.
** Ability to import a module giving it a different name. 
** Ability to import a symbol under a different name. 
** Ability to import a module giving all symbols a prefix.
** Ability to shadow imports.
** Ability to inherit imports (multi level import)
** A more general 'module path' mechanism. This is the road to hell; at least I don't know how to do this correctly and so I deferred it. 

XI) Relationship to existing modular Squeak work
================================================
** I've looked at environments
** I'm aware of the image segmentation work and wish to incorporate this. 
** I don't understand how isolated projects work.
** In general I haven't make up my mind about how Projects fit into all this. 
** I don't know how these image modules will relate to VM modules (plugins)


More information about the Squeak-dev mailing list