Hi Craig et al,
I have read this, and its a very interesting read!
For me the main issue is the protocol that is used
between the two images (subject and history).
There is little written about it.
Just for thought, what if the history memory would
be a web server. What would the protocol look like?
Can the low-level protocol be hacked to support this?

And one thing I am suspicious is that there is so
much knowledge in the IDs. And limits to the maximum
number of editions etc. I'd rather have proper
objects that those IDs, with LargeIntegers :-).

Just my 2 cents,
thanks,
Wolfgang


Craig Latta wrote:
> 
> Hi--
> 
>      This is another call for feedback on the design of Naiad[1], a 
> Smalltalk module system I'm writing for Squeak as part of the Spoon 
> project[2].
> 
>      On the theory that I'll get more of a response by including the 
> whole text rather than a link to it, here it is... :)
> 
> ***
> 
> 2008-10-20, 1946 GMT
> 
> Copyright (c) 2008 Craig Latta. All rights reserved.
> 
> 
> Hi--
> 
>      I've been on a quest to make Squeak smaller and more modular, the
> Spoon project[1]. Part one was making the object memory small. Part
> three is about making the virtual machine small. This message is about
> part two, making a module system suitable for adding new behavior to a
> minimal system in an organized way, and for transferring behavior
> accurately between running systems.
> 
>      Spoon's module system is called "Naiad", which is an acronym for
> "Name And Identity Are Distinct". It keeps track of the development
> history of a system (what the "sources" and "changes" files are for
> now), and makes it available for exchange with other systems. I think
> keeping classes' names and identities separate is critical for
> this. Following are some notes on its design and use, including the
> object model[1].
> 
>      At this point I'd like to emphasize I am the author of this
> design, that I intend to release its implementation under an MIT-style
> license, and that I'd like to pursue a graduate degree with it (I'm
> open to invitations :).
> 
> ***
> 
> motivation
> 
>      A traditional Smalltalk system uses source code to express both
> development history and changes exchanged between systems. The precise
> meaning of source code depends on the current state of the system
> compiling it. Since a Smalltalk system is dynamic, source code is an
> inherently ambiguous medium across time.
> 
>      The most problematic system artifacts in light of this ambiguity
> are classes. All activity in a Smalltalk system is the result of
> sending messages to objects. The sending of a message invokes the
> execution of a method, a sequence of instructions for a virtual
> processor. Some of these instructions manipulate the state of the
> object receiving the message. Classes define the structure of that
> state. Therefore, when those class definitions change, the source code
> for the methods of those classes may become meaningless.
> 
>      One may confront this situation when trying to recompile source
> code for an old version of a method whose class definition has changed
> in the meantime. Similarly, source code from one system may not be
> meaningful on another, since corresponding class definitions on each
> system may change independently (or be removed entirely).
> 
>      This means that the accurate exchange of behavior requires manual
> labor, hindering the propagation of useful fixes and new code. It also
> means that interpretation and use of historical code is more difficult
> than necessary. So we pay twice for this problem: when learning the
> system, and when trying to share our work with others. By separating
> class name from identity, Naiad makes Smalltalk more approachable for
> newcomers, and more productive for developer and user communities.
> 
> 
> editions
> 
> 
>      Using Naiad, each development system consists of two object
> memories: one containing developed code, and another containing
> "editions" which describe that code. I'll call the first one the
> "subject memory" and the other the "history memory".
> 
>      An Edition is a description of some artifact in the subject
> memory at some point in time, currently an author, comment, tag,
> class, method, module, checkpoint, or edit. Each edition has a
> reference to that artifact's next state in the future (the next
> edition) and in the past (the previous edition), as well as an author
> edition, a collection of licenses, and a timestamp.
> 
>      An Edit represents the activation of some edition at a point in
> time. For example, there may be a method created in 2005 that is
> removed in 2006 and reactivated in 2007. There would be an Edit for
> each of those three events, but only two method editions (one
> representing the method becoming active, and one representing it being
> removed).
> 
>      The history memory replaces the current changes and sources
> files. It has an instance of EditHistory corresponding to the subject
> memory, which records the active (current) editions for the classes,
> method, modules, and authors in the subject memory. It also keeps the
> subject memory's id and the last Edit made to the subject memory.
> 
>      Every time the subject memory adds, changes, or removes a class
> definition, method, author, comment, tag, or module, or makes a
> checkpoint (i.e., makes an edit), it adds the appropriate editions to
> the history memory via remote messages. The history memory snapshots
> itself after every edit, so as to provide crash recovery support.
> 
>      The subject memory keeps a remote reference to the history
> memory's instance of EditHistory as a class variable of the local
> EditHistory class, and interacts with it using utility messages sent
> to the local EditHistory class. The history memory also keeps that
> EditHistory instance as a class variable of its local EditHistory
> class, but as a local reference.
> 
>      An edition typically elides some of its references when it is
> transferred out of a history memory. For example, a transferred
> edition will usually omit the references to its next and previous
> editions. The requesting subject memory can calculate the ID of those
> editions and obtain them with a separate request, if necessary.
> 
>      A subject memory may elect to keep its EditHistory instance as a
> local object, such as in a situation where one wants some limited
> immutable history for debugging purposes, and no crash recovery
> support. Whether in this scenario or in normal development the same
> EditHistory utility messages suffice, since no special code need be
> written to support remote objects. If no edits will be made during
> deployment, and no history retrieval is required, one may simply
> jettison the history memory. One may always reconnect the subject and
> history memories at a later time and continue development.
> 
>      The subject memory has tools for browsing and activating the
> editions, wherever they are located. This means that no special tools
> are needed to browse the artifacts of multiple subject systems; one
> uses the same tools as for browsing the artifacts of the local subject
> memory. Each subject memory may connect to multiple history memories
> concurrently (if allowed).
> 
>      For that matter, the history memories of multiple systems may
> connect to each other directly, to aggregate editions from multiple
> people, for example.
> 
> 
> class and method IDs
> 
> 
>      Each class in the subject memory has a universally-unique
> identifier[3], or UUID. The classes in the minimal subject memory are
> assigned UUIDs before the initial release, and all subsequent classes
> are assigned UUIDs when created. Rather than use the single word
> "class" to refer to either a metaclass or to its sole instance, Spoon
> introduces the term "protoclass". For example, (Array class) is a
> metaclass, and its sole instance, Array, is a protoclass. Each
> metaclass and protoclass has its own UUID, called a "base ID". This is
> supported by a new instance variable in ClassDescription.
> 
>      Each version of each class is identified by a ClassID, a byte
> array with segments for the class's baseID, author UUID, and a
> sixteen-bit version. This means we can uniquely identify, for each
> author, 65,535 versions of each class in the system. Since we identify
> authors by UUID, the number of possible authors is very large.
> 
>      Each version of each method is identified by a MethodID, a byte
> array which contains a ClassID and segments for the method's selector,
> author UUID, and a sixteen-bit version. This means we can uniquely
> identify, for each author, 65,535 versions of each method in each
> version of each class in the system.
> 
> 
> method editions and method literal markers
> 
> 
>      Each MethodEdition holds a reference to the corresponding
> ClassEdition, the method source code, and the information needed to
> reconstruct the corresponding CompiledMethod directly, without need of
> the compiler (the method header, initial and final program-counter
> values, method literal markers, and instructions). If one will never
> use the history memory to install methods in a subject memory that
> lacks a compiler, one could drop the compiled method information to
> save space.
> 
>      Method literal markers are used to transmit a compiled method's
> literal frame values between object memories. There are method literal
> marker classes to support references to classes, class variables,
> other pool variables, and literal objects, and to support methods
> which perform class-side super-sends. Each method literal marker
> instance knows how to serialize itself as part of Spoon's remote
> messaging system. In particular, when a method literal that refers to
> a class transmits itself, it transmits the ClassID of that class, not
> the name of the class.
> 
>      This gets at the namesake concept of Naiad, "Name And Identity
> Are Distinct". When referring to a class, we never need to use its
> name. Each version of each class is an object with a distinct
> identity. By using ClassIDs to refer to each of them, we can avoid
> using class names at all when storing history or distributing
> code. This means that name of each class can be anything, as far as
> the system is concerned.
> 
>      With every class name unconstrained, there is no need for
> "namespaces" to distinguish between classes which happen to have same
> name at some point in time. Each class effectively has its own
> namespace, since it is uniquely identifiable regardless of its
> name.
> 
>      Developer tools armed with this information can resolve ambiguity
> for humans browsing and changing the system. If a developer writes a
> method which uses a name shared by multiple classes, the system can
> present more information about each of those classes (such as the
> author, time of creation, version, and module association), so that
> the developer can choose the intended one. When browsing such a
> method, the system can distinguish the aliased class name visually,
> indicating that there is disambiguating information available.
> 
> 
> class editions and shared variables
> 
> 
>      Each ClassEdition holds the editions for all the method versions
> currently active in the corresponding class in the subject
> memory. Since every edition keeps a reference to its previous and next
> editions, one can trace the history of any method by starting at the
> active edition. Removed methods are represented by method editions
> which have the same MethodID as a normal previous method edition, but
> with the rest of the fields set to nil.
> 
>      Each ClassEdition also holds the information needed to
> reconstruct the corresponding class directly, without need of the
> class builder. For all classes, this includes the format, instance
> variable names, and superclass ID. For protoclasses, it also includes
> the class pool keys, class name, and received pool IDs.
> 
>      In Spoon, every shared variable pool is the responsibility of
> some class in the system. There is no global variables pool ("system
> dictionary"). Each class that defines a pool is said to "publish" that
> pool; classes which use that pool "receive" it. Spoon adds an instance
> variable to Class to map published pools to their names. Each
> ReceivedPoolID that a protoclass edition uses is a byte array which
> contains a class ID and a published pool name.
> 
> 
> checkpoints and modules
> 
> 
>      A Checkpoint edition is simply a named marker of a particular
> point in time. A developer may use checkpoints to indicate various
> interesting states of development, and use the tools to regress or
> replay edits made before or after that time.
> 
>      The largest unit of work is represented by module editions. They
> are named collections of method IDs, indicating the specific versions
> of methods which comprise a module, along with sets of child, parent,
> prerequisite, and postrequisite module editions. When a module edition
> is transferred out of a history memory, those edition references are
> transmitted as ModuleIDs. Each module edition also has an
> "antimodule", a module edition calculated at installation time by a
> receiving system which, if applied, would undo the changes made by
> installing the original module. Finally, each module edition has a URI
> by which someone at a remote site may install the module.
> 
>      That URI represents a command to a Spoon system running on a
> requestor's local machine; it refers to a standard port on
> localhost. Its path is a text-encoded action, containing an
> instruction (in this case "install a module"), the hostname and port
> of a Spoon system providing the module, and the module's ID. The
> receiving system uses this information to request the module from a
> providing history memory, which then transmits editions as
> necessary. Exactly which editions are transmitted depends on the state
> of the receiving system; this is a two-way conversation between the
> providing and receiving systems. This is often more time and space
> efficient than simply providing all of a module's code, which is what
> happens with traditional static representations like change sets.
> 
>      The URIs may be cited on ordinary webpages, which are indexed by
> search engines like Google. A person in search of a module for a
> particular purpose can search for it with a web browser, using those
> search engines. Having found a module's URI, the person can click on
> it, establishing a connection to an embedded webserver in their local
> Spoon system, which carries out the URI's command.
> 
>      This mechanism for code distribution avoids storing code in
> static files. It's a deparature from Smalltalk's traditional "fileout"
> mechanism.
> 
>      The encoded URIs can serve other functions as well, such as
> listing a system's installed modules, removing an installed module,
> making a snapshot, and quitting the system. In this way one can use a
> web browser to interact with a Spoon system for several basic tasks;
> this is especially useful when the system is headless (e.g., in its
> initial minimal state).
> 
> 
> comments and tags
> 
> 
>      Editions for authors, classes, methods, checkpoints, edits, and
> modules each have their own comment and tag editions. This means each
> one of those artifacts has a comment and tags, and the changes in both
> are recorded over time. Comments are as we've already been using them:
> they're explanatory prose about the artifacts. Tags may be familiar to
> you from the web; they are short semantic markers used for grouping
> similar artifacts.
> 
>      I intend for tags to replace class and method
> categories. Nominally, we've been using class and method categories to
> establish semantic hierarchies, but the hierarchies have turned out to
> be quite shallow. Although we can form hierarchies with tags as well,
> I think we would do better to apply the sorts of algorithms that
> search engines use, and not concern ourselves with memorizing an
> artifact's semantic markers. The computational cost this incurs for
> the tools might have been high in the early days of Smalltalk, but it
> is quite modest now.
> 
> 
>      Thanks for reading! Please let me know of any questions or other
> feedback, and feel free to discuss this on the Spoon and Squeak-dev
> mailing lists.
> 
> 
> -C
> 
> [1] http://netjam.org/spoon/naiad
> [2] http://netjam.org/spoon
> [3] http://en.wikipedia.org/wiki/Universally_Unique_Identifier
> 
> -- 
> Craig Latta
> improvisational musical informaticist
> www.netjam.org
> Smalltalkers do: [:it | All with: Class, (And love: it)]
> 
> 
> 
>