[V3dot10] V3dot10 process and meta-problems

[Keith Hodges]

Dear Jerome,

I am all in favour of your suggested improvements to MC, and have been 
working on a few myself. Actually I have now refactored some of MC so 
much there is no going back!

Ideas include:

An MC equivalent of a changeset, so that changesets can be managed, 
browsed and now loaded (order-doesnt-matter) with the same tools as 
normal monticello files.
   

    I think this would be particularly useful for back porting, develop
    in the latest squeak, and have a managed patch for '3.7 can load 3.8
    packages compatability patch.mccs'. This patch can then be loaded
    "prior" to any packages which may use it. It may patch squeak or the
    package, either way.

Now Monticello  (1.5 ;-)  can  theoretically do things like, load a 
class, in the middle of a class hierarchy, rather than only as a leaf 
class (If I can debug it) the mechanism is in place. It can also 
add/remove instance vars (etc) to other classes as well as extending 
them. I haven't had time to test these out yet.

What I dont agree with you is on your incremental process.

The incremental process guarantees   that everyone who wants to 
contribute is working on a moving target. This leaves one person in the 
driving seat to guide the continuous integration, and that is why there 
is so much pressure involved.

My suggestions would be to plan a number of items and see them all 
worked on together. Integrating all of the tasks simultaneously and 
regularly. This is effectively what I have done on 3.9.1

I will attempt a diagram!

Your process:

Squeak + MCUpdate1 + SMUpdate2 + KernelFix (breaking MC1 and MC2) + 
MCUpdate1b + SMUpdate2b.

My Process.
   
(fixed point)        + MC Latest
Squeak-stable   ->   + SM Latest         -> Squeak-unstable  (test, fix 
and iterate)
                     + KernelFixLatest
 


The aim is a list of tasks to be completed, and have a script that loads 
all of the results of the tasks in.

For example: 3.11

    "Project to harvest bug fixes"
    "Project to make all UI elements Scriptable" via Scripter
    "Project to make a nice UI display for interrupting a progress bar 
in progress"
    "Project to implement proper logging facility"
    "Project to migrate the kernel to use the logging facility"
    "Project to mark up packages, to generate minimal functional 
versions automatically".
    "Project to develop further tests for the base image"
    "Project to integrate flow for the primary stream support"
    "Project to replace FileDirectory"
    "Project to get  and keep old packages loadable"
    add Compiler, Tools, Morphic3, etc etc etc

All of these could be worked on simultaneously and continually 
integrated in a script to generate an 'unstable' release.

This was not viable before, because such a build script would constantly 
run into problems. For example, rather than tweak the TTFFont Speedup 
fix to make it loadable, I spent a couple of days fixing the loader so 
the problem should not reoccur. Removing loading order dependencies 
should also make this simultaneous-integration approach one step closer 
to reality.
> Several comments
>
> 1) time
> It takes a minute or two to download an image zip file
> on a high speed broadband.
>
> Compared to that 20-30 minutes for updating seems
> unacceptably long. 
>
>   
So one machine automatically runs the build integration script, and 
everyone else can download it.

Individual developers can have their own -unstable branches so as not to 
break other peoples work in progress.

I demonstrated this 4 months ago, and Installer supports the ability to 
have stable/unstable/personal versions of each script, via its search 
'path' parameter.
> 2) MC will have more problems on large chunks than on
> small. And it will take a good long time to report the
> problems.  The crash-fix-tryAgain-crash cycle is too
> long IMHO.
>
>   
MC only installs the changes, I guess it is the download and anaylsis 
that takes much of  the time.
> 3) With 1 and 2 being true the tendency to burn out
> the helpful image updater will be great.
>   
Lets automate the image updater, and the rest of us can work on Squeak.
> The problem with 7109 happened in a different way
> earlier (First try never got into the image.)
> So this is the second attempt. 
>
>   
Thats the problem with a moving target, multiple platforms etc.
> This multiple attempt is typical of worthy problems. 
> At the end of 3.9 finding a way to insure changeset
> current was not nil took more that half a dozen tries.
>  Each yeilding a small amount of information that got
> us closer to the solution.
>
> My impression is that Stef did a lot of work to get
> though each cycle because MC is heavy and slow
>   
Again I advocate using MC, but not for small incremental changes.

Suggest to use it as a publishing Source code control mechanism.

On every iteration, save all packages that have changed.

To final gm-release, load from repositories.
> a) Somehow it seems to me that not only should
> changeset be able to automatically become MC package
> updates. It should work the other way also.  You
> should be able to specify a specific diff and have mc
> create a change set that would recreate that
> difference.
>
>   
I have made MCDictionaryRepositories available , and 'System Changes 
Sets' is an in memory virtual Monticello package.
> Those MCdiff equivalent change sets should then be
> able to be archived.  and downloaded locally and
> installed or filed into an image.  This at least
> creates a ratchet process of small steps. Saving time
>   
> overall for test pilots. Producing and equivalent
> result to what we are attempting to do now.
>
> b) You might need to have two co-located people
> sharing the image creating task so they can spell each
> other.  And increase the possiblity of avoiding either
> of them burning out.  Colocation is to insure a wide
> stream of communication between the two of them.
>
> Finding this kind of resource would be maybe
> difficult. But it wouldn't hurt to ask.
>
> c) IMHO it needs to be admitted that the current
> choice for maintaining an image is not a good final
> choice. Some real deep though needs to be given to
>   
It is a good choice for partitioning an image, and source controlling 
those partitions.

It is just not a good choice for the little tweaks and fixes, or the 
broad across the board fixes that a release team needs to do.

My plan allows both. By allowing a Package to have a subset, being a 
minimal functional version, more and more stuff can be unloaded from the 
base image, and managed in packages. The remainder can be looked after 
any which way you like as long as the image can be saved to the 
repository(s) and loaded back in again successfully.

> I am pretty sure I have left out good and obvious
> ideas.  Its important to communicate what I have
> observed and thought of so far. Can you add something
> that might help?
>
>   
I think that the majority of the problems that previous teams had can be 
resolved by improving the tools.

best regards

Keith

 p.s. I did get your email
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/v3dot10/attachments/20070517/f6c86aba/attachment.htm