second call for feedback on Naiad design

[Craig Latta]

Hi--

      Thanks for the comments! I'm responding to the comments so far in 
this single message. I see no reason to restrict Naiad-related 
discussion to a single thread; hopefully threads will emerge around 
particular specific issues, rather than particular people. :)  Please 
feel free to break issues out into new threads... for this message, 
there's such a grab-bag going that I decided to deal with it all in one 
place.

      Karl writes:

 > When this system works, won't image size be an issue, like an
 > ever-growing web browser cache that has no size limit?

      I imagine the history memory will have various utilities, like:

-    dumping all the compiled method info, because the subject memory
      will always have a compiler

-    dumping all the method source, because the subject memory will
      never have a compiler :)

-    storing its less-frequently-accessed editions in one or more
      separate history memories, which spend most of their time as
      suspended snapshot files, but which can be activated when
      necessary. Remote message-sending is a fundamental part of
      Spoon; there's no inherent reason why the history memory can't be a
      federation of history memories instead.

      Of course, one might decide to put editions in another object
      database at any point instead (e.g., Magma or Gemstone). I just
      want to provide something that provides the bare minimum
      functionality "out of the box".

-    purging certain editions entirely (rather like when we made new
      sources files with the traditional setup)

***

      Wolfgang writes:

 > For me the main issue is the protocol that is used between the two
 > images (subject and history). There is little written about it.

      This is true, I haven't finished that documentation yet. One can 
look at the implementation of remote message-sending from the last Spoon 
release, but I haven't described it in prose yet, and the Naiad design 
document is the most prose I've written about how the subject and 
history memories communicate at a higher level. Eventually all this 
stuff will be in the Spoon book[1].

 > Just for thought, what if the history memory would be a web server.
 > What would the protocol look like?

      Well, there is already a (tiny) webserver in the subject memory, 
to  provide the initial user interface when first run. One could load 
its conveying module into the history memory and do lots of interesting 
things with it, yes.

 > Can the low-level protocol be hacked to support this?

      Yes.

 > And one thing I am suspicious is that there is so much knowledge in
 > the IDs.

      Since they're going to be flying back and forth over sockets, 
sometimes in large numbers, they need to be as small as possible; so 
I've thought carefully about minimizing them. At this point I'm simply 
open to discussion about what anyone would leave out. :)  I think have a 
good argument for every bit in every ID (likewise for every bit in the 
minimal subject memory).

 > And limits to the maximum number of editions etc.

      So far I've decided that it's not worth any extra bits expressing 
variable-length sizes, but again I'm open to discussion about that.

 > I'd rather have proper objects that those IDs, with LargeIntegers :-)

      (The size argument applies here, too.)

***

      Michael writes:

 > I think the main reason people aren't commenting is because that's a
 > lot of reading!

      Sure, I'll just keep asserting that the importance justifies the 
time. :)

 > Perhaps "versions" is a better name than "editions"? That's the name
 > we're more familiar with.

      In this case I think the familiarity is a disadvantage; "version" 
has multiple strong meanings to people. A "version" is sometimes an 
artifact which has multiple interesting states over time, and sometimes 
it's an identifier used to refer to such an artifact. I think it's 
better to use a less-used term here, and I like the resonance between 
"edition" and "edit".

 > Do we need to run two instances of Squeak to edit code, one for the
 > current version and one for managing the edit history? I assume that's
 > what you mean by needing two object memories.

      That's right. The typical case is one person using one subject 
memory connected to one history memory that is mostly that person's 
editions, over a localhost socket connection.

 > If so, is it intended for the edit history object memory to be a live
 > central repository shared by developers?

      That's also an option, yes; it's just not the default.

 > Does the system work if it can't contact the edit history object
 > memory?

      Yes, but the tools would show decompiled method source, and some 
history features like regression would be unavailable, (similar to what 
happens if you don't have the changes/sources files with the current 
setup). But the typical case is that you have the history memory 
snapshot on the local machine, so it seems no more likely this would 
happen than it would be for one to lose the old changes/sources files, 
or indeed the subject memory itself.

 > What do your remote references look like?

      Each one is an object which holds a special hash for a remote 
object, and stream on a socket connected to the remote system. So...

 > How stable are they? Do they rely on, e.g. IP address to find a remote
 > object memory? If somebody changes IP, are the remote references still
 > valid?

      ...currently, they do not survive suspension or termination of the 
object memory in which they live. They are *not* like URLs, as your 
comment implies. They are not a description of how to reach a remote 
object, they are an active connection to a remote object which behaves 
in all ways like the remote object. In general, they are created by 
sending messages to other remote objects. The first remote objects in a 
session are created specially as part of the connection handshake 
between object memories.

      If the object memory of the reference is suspended (saved and 
quit), the reference is nilled on resumption of the memory.

      I implemented this part of the system in 2003; it's been in all 
the Spoon releases so far.

 > I assume a class now contains a ClassID and a collection of MethodIDs?

      No, a class has a "base ID", which is a UUID. The subject memory 
as a whole also has a UUID. The history memory knows the UUID of the 
subject memory it is tracking, and has "class editions" for each of the 
classes that have ever existed in the subject memory. Each class edition 
has "method editions" for all of the methods which have ever existed for 
that class as defined at a certain point in time.

 > Why is ClassID so complex?

      It's complex? It's just a base UUID, an author UUID, and a version 
number. I think if it were any simpler we'd lose something important.

 > Why not just assign each class a new UUID for each new version of that
 > class, with authorship and versioning being metadata of that class?

      It seems to me that it would be useful to have a single unique 
identifier that can refer to the definition of a class at all points in 
time, as expressed by all authors. When you want to get more specific as 
to author and point in time, you can append additional bits to it.

      Also, I explicitly want to keep history information separate from 
the artifact objects they describe, so that they may be easily left 
behind during production.

 > Limiting to 65,536 versions per author is going to create problems in
 > 10 years time.

      I disagree. Remember, these are editions of a class *definition* 
(instance variable format, etc.). If you add a method to a class, you're 
not creating a new edition of that class, you're merely creating a new 
method edition. From my experience (which encompasses more than ten 
years ;), authors tend to create entirely new classes much more often 
than they revise class definitions, and they simply use the classes as 
they exist a lot more often than that. Frankly, I'd expect 1,024 to 
suffice here. Sixteen bits is simply the first sufficient number of 
bytes, so it's convenient as well.

 > Isn't having the author and version in the [class] IDs going to cause
 > conflict problems? What happens if the author is careless and ends up
 > with two different versions of a method with the same unique
 > identifier?

      Another good reason for keeping the history information in a 
separate (and headless) object memory is so it can take of itself 
without most developers bothering with it. :)  The typical developer 
uses tools in the subject memory. Those tools only make requests to the 
history memory for new editions to be added; they have no say in how the 
corresponding IDs are made. In particular, the history memory decides 
what the next available version number is for a particular combination 
of class base ID, author, and selector.

 > Are author UUIDs going to be able to be looked up to get email
 > addresses and names somehow?

      Each history memory stores author editions; each author edition 
associates an author UUID with all that info and more (see the class 
tree at [2]). When you receive a module from another author's system, 
you get the relevant author editions as well. When you use a new system 
for the first time, you can create an author edition for yourself.

 > Methods shouldn't have an author. The changes between methods
 > versions/editions should have an author.

      I disagree. I think it's less work over time to figure out those 
changes when necessary.

 > I think you're taking the "minimal memory usage" idea too far.

      I think it's necessary to make the system as easy to learn and 
maintain as I want it to be.

 > In my design for distributable packages... packages (cf: classes in
 > Naiad)...

      I would expect them to correspond to Naiad's modules, not classes.

 > ...they need to (deep-)copy it...

      Uh-oh... "deep copy" is one of those phrases that immediately 
makes me suspect something is wrong (almost as bad as someone saying 
"dude" ;).

 > I've separated source from bytecodes.

      Naiad does that, too.

 > I'm not sure it's a good idea to propose an unstable system as the
 > next version of Squeak though.

      Well, this is two major versions out, not one. I think we can get 
plenty of testing in. And I think we're in serious danger of stagnation 
as it is. For better or worse, I think this history stuff is the sort of 
thing that has to be done with a relatively provocative step. Sometimes 
this is good (insert your favorite Alan Kay quote here ;).


      thanks again!

-C

[1] http://netjam.org/spoon/book
[2] http://netjam.org/spoon/naiad