On maintaining a consistent update stream

goran.krampe at bluefish.se goran.krampe at bluefish.se
Fri May 14 13:25:02 UTC 2004


Hi people!

Ned Konz <ned at squeakland.org> wrote:
> Recently I had the opportunity to speak in person with Michael Rueger, 
> Scott Wallace, Yoshiki Ohshima, and Diego Gomez Deck about a number of 
> things.
> 
> One of the items that we discussed was the nature of the update stream.
> 
> There have been several updates in the update stream that were change 
> sets that loaded (or tried to load) packages from SqueakMap.
> 
> This has several negative consequences:
> 
> * It is possible to have a reliable connection to an update server but 
> not be able to load an update (if either SqueakMap or the server that 
> holds the package is down).

Yes. One measure is to place the actual release on the SqueakMap server
itself. As was done with VersionNumber for example. When we discussed
this last I think we agreed that a package release that is installed
using an update *should* be physically placed on the SM server. Thus we
only depend on two servers, which is vastly better than x > 2. :)

And the SM server itself should be made as stable as possible. Currently
we don't have mirrors, but it is very hig up on my todo list. ;-) (help
is always appreciated etc etc)

> * There is the risk that an external package may be replaced by another, 
> thereby introducing the possibility of there being differences between 
> to images that purport to be the same version. Though we've done this 
> from time to time in the update stream to fix bugs, (I recall that) the 
> replacements have been made quickly after the original was posted, and 
> only when the original update itself was not able to be fixed with 
> another update.

This is IMHO not a risk. The packages that we do install/upgrade using
updates should only be "base" packages that are under the "control" of
the community if you know what I mean. And there is currently NO reason
to issue an update that does NOT specify a specific release - all
updates using SM **should** specify a specific release. Period.

Not also that the idea of "freezing" releases has been on my drawing
board but several prominent Squeakers thought it was a bad idea. Instead
I introduced the "published" checkbox which is more of a "promise" from
the package maintainer that such a release will not be deleted or
altered in the future.
 
> * It is impossible to collect a single updates directory on a disk and 
> then apply the updates from that directory without a network connection 
> (we have "Utilities applyUpdatesFromDisk" which does this easily, and 
> which has been useful to those of us who actually create updates for the 
> update stream).

Well, it IS possible if you first preload the cache directory using a
script like I posted recently. :)

> It also has a positive aspect:
> 
> * It is easier to take non-code items like fonts, etc. in SAR files.
> 
> By definition, the update stream is for packages that live in the image.

Ehum, right.

> There are a few such packages that also are available on SqueakMap 
> (MCInstaller, PackageInfo, SM2, and SARInstaller are the ones that come 
> to mind). All of these packages are just code, and so the packaging 
> advantages of (say) a SAR aren't important.

I disagree with the "also" part. Those packages do NOT live in the
image. IMHO.
So they should NOT be evolved using updates but should instead get new
releases on SM and the update should simply do something like:

"Installs MCInstaller version 10"
SMSqueakMap default installPackageReleaseWithId:
'64076024-22b8-4b0d-94dc-113623fc5ac2'

The update stream is IMHO meant to evolve the code that has not yet been
moved out into an external package.

> For my updates to SARInstaller I have published the same code to the 
> update stream as Smalltalk code. We published the Bitstream Vera fonts 
> to the update stream as Smalltalk code, even though it was a fair amount 
> of work to do so (loading a SAR would have been much easier).

I am not sure why you do it this way.

> However, there have been other updates that have gone to SqueakMap to 
> load packages.

Hopefully they specified a specific release. :)

> We felt after discussion that it would be a good idea to establish a 
> clear policy for the update stream to avoid the above problems.

I very much agree.

> Michael's summary of our discussion is:
> 
> "General assumption should be that updates can be filed in from disc 
> only, without connection to any server.
> Images produced by updating should always be exactly the same, not 
> depending on some external changes over time as is the case with 
> external packages."

I agree in principal, I do not agree with the "as is the case with
external packages" since it implies that you can't trust SM in this
regard. The whole idea with having releases in SM is that those DO NOT
CHANGE over time. That is also the whole basis of the SM cache. Sure, I
haven't made them immutable, but as I said, that was because people
thought it was a dumb thing to do.

But they should be immutable *in practice*. Just as the updates are. :)

> My additional suggestions (not necessarily shared by the above named 
> people):
> 
> * We *should* require that all components of an update stream be loaded 
> from the same server. Having to go to more than more server is a really 
> bad idea.

Well, the simplest solution would be to move the updates to the SM
master. :)
Or at least set up 1-2 SM mirror servers.
 
> * We *could* require that all the updates be Smalltalk code, as they 
> have been. In the past this has required hacks like MIME-encoded, 
> Gzipped files to load fonts. This does, however, allow easier analysis 
> of code (like Andreas' recent analysis of authorship of methods in the 
> update stream), as well as searching for code (since the files are all 
> text).

I have no real thought on that. You guys know that better.

> * To avoid such hacks, we *could* extend the in-image update loader to 
> use the SqueakMap installers for various formats (SAR, .cs, .mcz, 
> etc...) and thus avoid the need for a change set that then loads another 
> file

Sure. Agai, no real thought on that.

> If we want to unify SqueakMap with the update stream, the following 

Could you explain what "unify" means in this respect? My mind was pretty
clear on what is what:

- The update stream mainly evolves the image we call "Minimal". With
"mainly" I mean that it only includes code changes for "Minimal" (not
Basic). Minimal doesn't exist in downloadable form today, instead it is
Basic with a few packages removed (those that currently live on SM and
that you can see as "installed" in a fresh Basic image).

- The update stream also evolves Basic, in the sense that it also have
updates that install/upgrade those additional packages using SM.

Now, I do agree that this is a bit "unclear". For example, what happens
if I throw out some of these packages thus getting to Minimal and then
load updates? Those updates that has "SM doits" and that are meant for
those packages should in that case be ignored or something. So it isn't
really clear today, I agree.

Ideas?

> suggestion from Diego Gomez Deck (who is sitting next to me right now) 
> could be useful:
> 
> * We could take items that were downloaded from an update server as part 
> of the update process but are also available on SqueakMap and save them 
> in the local SqueakMap cache to avoid duplicate downloading.

I am not sure I understand what you mean.

> What do you think?
> 
> Guides?

I am listening, this is important. :)

> Thanks,
> Ned Konz

regards, Göran



More information about the Squeak-dev mailing list