InstallStream 1.0 - Cuis (and All Smalltalks)
Introduction
Cuis2.0 from Juan Vuletich is showing the way towards the goal of
working with a relatively small kernel image as the starting point, to
build evolutionary and revolutionary new Smalltalk platforms. (e.g.
Morphic3)
A true kernel image need have no package or source code management
tools out of the box, nor does it have network protocols loaded.
The idea of InstallStream is to enable the simplest possible mechanism
that could possibly work, for a broad range of tasks.
1) Loading kernel updates.
2) Publishing and collaborating on kernel fixes for approval and
inclusion in the release image.
3) Publishing and maintaining personal collections of fixes.
4) Publishing and collaborating on specific kernel innovations.
5) A very simple means to publish and reload mini-packages so that
more can be removed from the kernel
6) Publishing sequences of monticello packages and fixes to load and
use tools/frameworks e.g. Seaside
7) Building derived images using all of the above
8) Capturing exactly the content and installation process of a derived
image for customisation and cherry picking.
Example Usage:
The collaboration and distribution of files for use with the minimal
kernel image is managed using source code management tools external to
squeak, such as Bazaar2.0, Mercurial, or Git. The following assumes we
have checked out the projects we wish to include in the image we are
building.
The existing updates stream mechanism consists of a single numbered
series of changesets, serially applied to improve the released image.
InstallStream (IS) provides an extension of the updates idea, such
that you may publish or subscribe to a number of different
InstallStreams in local directories. These will be applied in a
prescribed order, to a known starting image.
1) Loading release image kernel updates.
"to view pending updates"
InstallStream kernel pending.
an OrderedCollection(
InstallChangeSet on: 'updates/0394-DummyUpdateForDemo.1.cs'.
InstallChangeSet on: 'updates/0395-2ndDummyUpdateForDemo.1.cs'.)
"to load these updates"
InstallStream kernel applyAll.
"to see the status"
InstallStream status ==> a Dictionary('updates'->395 )
2 3 4 & 5) Loading kernel fixes and updates
"to view all pending updates"
InstallStream updates collect: [ :e | e pending ].
an OrderedCollection(
an OrderedCollection() "kernel updates are already installed"
an OrderedCollection(
InstallChangeSet on: 'updates-System-InstallStream/01-System-
InstallStream.st'.)
an OrderedCollection(
InstallChangeSet on: 'updates-demo/01-InstallStreamTest1.st'.
InstallChangeSet on: 'updates-demo/02-Load2more/01-
InstallStreamTest2.st'.
InstallChangeSet on: 'updates-demo/02-Load2more/02-
InstallStreamTest3.st'.
InstallChangeSet on: 'updates-demo/04-InstallStreamTest4.st'.)
an OrderedCollection(
InstallChangeSet on: 'updates-kph/0001-Symbol-value.st'.
InstallChangeSet on: 'updates-kph/0002-CodeHolder-
HandleSelectedClassIsNil.1.cs'.))
In the above example output the following directories are seen:
updates-System-InstallStream - a mini changeset-based package for
keeping InstallStream up to date.
updates-demo - a hierarchically organised collection of fixes
updates-kph - a personal set of fixes in progress
6 7 & 8) Deploying Packages and Building Images
"to view all pending installs"
InstallStream installs collect: [ :e | e pending ].
an OrderedCollection(
an OrderedCollection(
InstallMonticelloPackage on: 'Installer/install.Installer/01-
Installer-Core-kph.325.mcz'.
InstallMonticelloPackage on: 'Installer/install.Installer/02-
Installer-Scripts-kph.24.mcz'.))
Use Cases
Personal Kernel Fixes and Personal Application Fixes
An IS may be used as a collection point for personal local fixes, and
as a publishing mechanism to share fixes with others, and to nudge
them upstream to the kernel maintainer.
Similarly an IS may be used by a group to collaborate on a suite of
fixes, such as an audit of mathematical functions provided by the
kernel.
An IS may be used to publish, collaborate on, and subscribe to
optional features to the kernel, or prospective upgrades, such as
Traits, or a Namespace implementation.
In connection with an existing application-framework such as Seaside,
an IS may be used to manage local patches required by that application-
framework (applied prior to loading the application), or local patches
to that application itself after it is loaded.
In addition an IS may be used to provide the equivalent of packages,
but with less imposed structure.
And finally IS may be used to deploy sequences of existing monticello
packages, to install applications such as Magma or Seaside.
A Matter of Record
When Monticello or a package management system is used to load a
number of packages, the record of the exact installation process is
not kept in a manner which is then easily available for use by others.
"Installer" is one option for sharing install procedures, since it is
used to write scripts for loading packages in a specific order, after
which scripts may then be exchanged, and customised.
Using IS to organise packages for loading into a base image, the
directories of files are in themselves the record of exactly what was
loaded and in which order.
Thus when updates.webapp.seaside3 is published for Cuis using Bazaar,
and you check it out locally, it comes with all the chosen packages in
the appropriate order. You may branch it if you need a custom version,
and when you publish your application, it may be published with the
whole updates/installs build tree, since the build tree is the
documentation of what happened in which order to build your
application, ripe for cherry picking, or branching by any other
subscriber.
Packages in InstallStreams
It is worth noting that existing Monticello packages may be put into
IS in order to be loaded sequentially. This means that a single update
stream may publish the equivalent of a MonticelloConfiguration, or a
Sake/Packages load sequence, or a Metacello package definition.
A complex package like Magma, can be delivered by checking out a
single install stream, of ordered packages.
The "layer-cake" Model of the Image
Within existing package managers, the idea that there are explicit
dependencies between packages is used to provide a safe load order.
IS does not have this mechanism, instead, the system is considered to
have a layered architecture, and packages are published with respect
to a specific layer. Notably they may be subscribed to at a different
layer if required.
The basic layering is provided by a dot notation, in the directory name.
Layer 3 - /updates.webapp.seaside - application frameworks
Layer 2 - /updates.network - tools and services
Layer 1 - /updates-system - the kernel and core sub-systems
Layer 0 - /updates - the kernel update stream
The fundamental purpose of layers 0 & 1, the kernel and core layers,
is to provide a standardised language for use by all the other layers.
If there is a change in language syntax in this layer all the other
layers are compromised.
The rest of the layering is intended so that each layer builds upon
the layer of the API below. For example, an application framework such
as seaside, may if it chooses, isolate its users from variations in
the layers above.
Within each layer there are packages which provide a base framework,
and packages which use the base framework. So we may optionally use
alphabetic sorting to provide explicit load ordering subdividing the
layers further as follows. (from top to bottom)
Post-Post-Layer 2 Users Fixes - /updates.~kph-network-patches - the
tilda is used to "sort last"
Post-Layer 2 - /updates.~01-network-patches
Within Layer 2 - /updates.network-02-shttp
Within Layer 2 - /updates.network-01-http
Ordered Pre-Layer 2 - /updates.02-network-http
Ordered Pre-Layer 2 - /updates.01-network-sockets - the digits are
used to "sort first"
You can have as many layers as you wish and IS is agnostic about how
the layers are used, apart from the bottom layer 0. This is the
default, a re-implementation of the updates directory of old, which
may be used in any manner that the kernel maintainer wishes. Note
there is no "pre layer 0".
We may define the layers by commonly accepted and useful norms like
so: (bottom to top)
1a. kernel - the absolute minimum e.g. compiler in /updates
1b. core - a working minimum e.g. /updates-morphic /updates-
collection
2a. system - interfaces /updates.00-sockets
2b. system - tool and service frameworks /updates.network /
updates.toolbuilder
2c. system-extensions - services servers e.g. /updates.network-smtp
3a. application frameworks - e.g. Seaside
3b. application frameworks - integration e.g Seaside-Magma
4. Generic Application Development
updates.webbapp.seaside.store
5. Specific Application Deployment
updates.webbapp.seaside.store.acmestores
Conclusion
InstallStreams 1.0 is looking pretty usable and looks to have a lot of
potential bang for its buck. This simple idea has, in this 1.0
release, a clean and extensible implementation. I trust that this
passes Juan's criteria for simplicity.
In conjunction with Bazaar (or other scm tools) IS can provide the
basis for a "trunk for kernel images", in which both parallel (the old
3.11 process) and sequential (the new trunk process) approaches may be
combined.
I think that Cuis2.0 is poised ready to make great strides.
best regards
Keith
postscript:
Next Steps
1. InstallStreams may be used to tidy and trim Cuis2.0 further, if
Juan adopts IS for himself for loading his favourite bits and pieces.
2. The classes InstallChangeSet, and InstallMonticelloPackage, could
be joined by an InstallAtomically.
3. Loading Monticello into Cuis
