InstallStream 1.0 - Cuis (and All Smalltalks)

Introduction
Cuis2.0 from Juan Vuletich is showing the way towards the goal of  
working with a relatively small kernel image as the starting point, to  
build evolutionary and revolutionary new Smalltalk platforms. (e.g.  
Morphic3)

A true kernel image need have no package or source code management  
tools out of the box, nor does it have network protocols loaded.

The idea of InstallStream is to enable the simplest possible mechanism  
that could possibly work, for a broad range of tasks.

1) Loading kernel updates.
2) Publishing and collaborating on kernel fixes for approval and  
inclusion in the release image.
3) Publishing and maintaining personal collections of fixes.
4) Publishing and collaborating on specific kernel innovations.
5) A very simple means to publish and reload mini-packages so that  
more can be removed from the kernel
6) Publishing sequences of monticello packages and fixes to load and  
use tools/frameworks e.g. Seaside
7) Building derived images using all of the above
8) Capturing exactly the content and installation process of a derived  
image for customisation and cherry picking.

Example Usage:
The collaboration and distribution of files for use with the minimal  
kernel image is managed using source code management tools external to  
squeak, such as Bazaar2.0, Mercurial, or Git. The following assumes we  
have checked out the projects we wish to include in the image we are  
building.

The existing updates stream mechanism consists of a single numbered  
series of changesets, serially applied to improve the released image.

InstallStream (IS) provides an extension of the updates idea, such  
that you may publish or subscribe to a number of different  
InstallStreams in local directories. These will be applied in a  
prescribed order, to a known starting image.

1) Loading release image kernel updates.
"to view pending updates"
InstallStream kernel pending.

an OrderedCollection(
	InstallChangeSet on: 'updates/0394-DummyUpdateForDemo.1.cs'.
	InstallChangeSet on: 'updates/0395-2ndDummyUpdateForDemo.1.cs'.)

"to load these updates"
InstallStream kernel applyAll.

"to see the status"
InstallStream status ==> a Dictionary('updates'->395 )

2 3 4 & 5) Loading kernel fixes and updates
"to view all pending updates"
InstallStream updates collect: [ :e | e pending ].

an OrderedCollection(
  an OrderedCollection() "kernel updates are already installed"
  an OrderedCollection(
	InstallChangeSet on: 'updates-System-InstallStream/01-System- 
InstallStream.st'.)
  an OrderedCollection(
	InstallChangeSet on: 'updates-demo/01-InstallStreamTest1.st'.
	InstallChangeSet on: 'updates-demo/02-Load2more/01- 
InstallStreamTest2.st'.
	InstallChangeSet on: 'updates-demo/02-Load2more/02- 
InstallStreamTest3.st'.
	InstallChangeSet on: 'updates-demo/04-InstallStreamTest4.st'.)
  an OrderedCollection(
	InstallChangeSet on: 'updates-kph/0001-Symbol-value.st'.
	InstallChangeSet on: 'updates-kph/0002-CodeHolder- 
HandleSelectedClassIsNil.1.cs'.))

In the above example output the following directories are seen:
updates-System-InstallStream - a mini changeset-based package for  
keeping InstallStream up to date.
updates-demo - a hierarchically organised collection of fixes
updates-kph - a personal set of fixes in progress

6 7 & 8) Deploying Packages and Building Images
"to view all pending installs"
InstallStream installs collect: [ :e | e pending ].
an OrderedCollection(
  an OrderedCollection(
   InstallMonticelloPackage on: 'Installer/install.Installer/01- 
Installer-Core-kph.325.mcz'.
   InstallMonticelloPackage on: 'Installer/install.Installer/02- 
Installer-Scripts-kph.24.mcz'.))

Use Cases
Personal Kernel Fixes and Personal Application Fixes

An IS may be used as a collection point for personal local fixes, and  
as a publishing mechanism to share fixes with others, and to nudge  
them upstream to the kernel maintainer.

Similarly an IS may be used by a group to collaborate on a suite of  
fixes, such as an audit of mathematical functions provided by the  
kernel.

An IS may be used to publish, collaborate on, and subscribe to  
optional features to the kernel, or prospective upgrades, such as  
Traits, or a Namespace implementation.

In connection with an existing application-framework such as Seaside,  
an IS may be used to manage local patches required by that application- 
framework (applied prior to loading the application), or local patches  
to that application itself after it is loaded.

In addition an IS may be used to provide the equivalent of packages,  
but with less imposed structure.

And finally IS may be used to deploy sequences of existing monticello  
packages, to install applications such as Magma or Seaside.

A Matter of Record
When Monticello or a package management system is used to load a  
number of packages, the record of the exact installation process is  
not kept in a manner which is then easily available for use by others.

"Installer" is one option for sharing install procedures, since it is  
used to write scripts for loading packages in a specific order, after  
which scripts may then be exchanged, and customised.

Using IS to organise packages for loading into a base image, the  
directories of files are in themselves the record of exactly what was  
loaded and in which order.

Thus when updates.webapp.seaside3 is published for Cuis using Bazaar,  
and you check it out locally, it comes with all the chosen packages in  
the appropriate order. You may branch it if you need a custom version,  
and when you publish your application, it may be published with the  
whole updates/installs build tree, since the build tree is the  
documentation of what happened in which order to build your  
application, ripe for cherry picking, or branching by any other  
subscriber.

Packages in InstallStreams
It is worth noting that existing Monticello packages may be put into  
IS in order to be loaded sequentially. This means that a single update  
stream may publish the equivalent of a MonticelloConfiguration, or a  
Sake/Packages load sequence, or a Metacello package definition.

A complex package like Magma, can be delivered by checking out a  
single install stream, of ordered packages.

The "layer-cake" Model of the Image
Within existing package managers, the idea that there are explicit  
dependencies between packages is used to provide a safe load order.

IS does not have this mechanism, instead, the system is considered to  
have a layered architecture, and packages are published with respect  
to a specific layer. Notably they may be subscribed to at a different  
layer if required.

The basic layering is provided by a dot notation, in the directory name.

Layer 3 - /updates.webapp.seaside - application frameworks
Layer 2 - /updates.network - tools and services
Layer 1 - /updates-system - the kernel and core sub-systems
Layer 0 - /updates - the kernel update stream

The fundamental purpose of layers 0 & 1, the kernel and core layers,  
is to provide a standardised language for use by all the other layers.  
If there is a change in language syntax in this layer all the other  
layers are compromised.

The rest of the layering is intended so that each layer builds upon  
the layer of the API below. For example, an application framework such  
as seaside, may if it chooses, isolate its users from variations in  
the layers above.

Within each layer there are packages which provide a base framework,  
and packages which use the base framework. So we may optionally use  
alphabetic sorting to provide explicit load ordering subdividing the  
layers further as follows. (from top to bottom)

Post-Post-Layer 2 Users Fixes - /updates.~kph-network-patches - the  
tilda is used to "sort last"
Post-Layer 2 - /updates.~01-network-patches
Within Layer 2 - /updates.network-02-shttp
Within Layer 2 - /updates.network-01-http
Ordered Pre-Layer 2 - /updates.02-network-http
Ordered Pre-Layer 2 - /updates.01-network-sockets - the digits are  
used to "sort first"

You can have as many layers as you wish and IS is agnostic about how  
the layers are used, apart from the bottom layer 0. This is the  
default, a re-implementation of the updates directory of old, which  
may be used in any manner that the kernel maintainer wishes. Note  
there is no "pre layer 0".

We may define the layers by commonly accepted and useful norms like  
so: (bottom to top)

1a. kernel  - the absolute minimum e.g. compiler in /updates
1b. core     - a working minimum e.g. /updates-morphic /updates- 
collection

2a. system - interfaces /updates.00-sockets
2b. system - tool and service frameworks /updates.network / 
updates.toolbuilder
2c. system-extensions - services servers e.g. /updates.network-smtp

3a. application frameworks - e.g. Seaside
3b. application frameworks - integration e.g Seaside-Magma

4. Generic Application Development
	updates.webbapp.seaside.store
5. Specific Application Deployment
	updates.webbapp.seaside.store.acmestores

Conclusion
InstallStreams 1.0 is looking pretty usable and looks to have a lot of  
potential bang for its buck. This simple idea has, in this 1.0  
release, a clean and extensible implementation. I trust that this  
passes Juan's criteria for simplicity.

In conjunction with Bazaar (or other scm tools) IS can provide the  
basis for a "trunk for kernel images", in which both parallel (the old  
3.11 process) and sequential (the new trunk process) approaches may be  
combined.

I think that Cuis2.0 is poised ready to make great strides.

best regards

Keith

postscript:

Next Steps
1. InstallStreams may be used to tidy and trim Cuis2.0 further, if  
Juan adopts IS for himself for loading his favourite bits and pieces.
2. The classes InstallChangeSet, and InstallMonticelloPackage, could  
be joined by an InstallAtomically.
3. Loading Monticello into Cuis




-------------- next part --------------
Skipped content of type multipart/mixed