Update stream ideas for 3.8 (was Re: Squeak 3.8 status)

Doug Way dway at mailcan.com
Tue Aug 10 05:50:47 UTC 2004


There have been a number of suggestions lately on improving the speed 
of the harvesting process/update stream.

After thinking about it for a bit, there are a few things we could 
possibly try in 3.8alpha:

1. Start up the separate "internal" update stream as Michael suggested. 
  Except it wouldn't really be private/internal, it would just be very 
bleeding-edge... it would be readable by anyone who wanted to file in a 
special changeset to get access, or maybe just by switching a 
preference setting.  Periodically, these would be transferred to the 
external/relatively stable stream, as Michael described below, maybe 
every 2-4 weeks or so.  (Maybe come up with new names for the 
"internal" vs. "external" streams, maybe "unstable" vs. "stable"?  
Although the regular stream won't really be all that stable...  Maybe 
"unstable" vs. "regular", I don't know.)

There would need to be some conventions followed for the unstable 
update stream.  For example, bug reports shouldn't be posted to 
squeak-dev for something that's only in the unstable stream.  Perhaps 
there could be a separate mailing list with the very narrow topic of 
fixing immediate problems in the unstable stream.  And of course, Bruce 
O'Neel would only upload images to the ftp site when the regular/stable 
stream is updated.

The mechanism to do this is already in place, it just hasn't been used 
in a while.  I remember when the Guides started up we didn't want to 
have a private update stream a la Squeak Central, but I think this sort 
of semi-public/unstable stream could work.

2. (This one is my suggestion.)  If we have this unstable update 
stream, posting updates to this stream doesn't have to be as 
coordinated as with the regular stream.  So, we could allow harvesters 
who [approve] an item in the BFAV to also immediately "broadcast" that 
item as an update to the unstable stream.  (The broadcasting mechanism 
to do this is also already there from the SqC days, but needs some 
fixing.)

If all harvesters have the ability to broadcast updates, we'd want to 
make sure the same sort of automatic checks are happening before the 
update is broadcasted, as when I publish updates.  The broadcasting 
code could run the ConflictChecker, could check to make sure method 
timestamps aren't missing, check for linefeeds (I think it already does 
that), etc.  Plus it could be beefed up further, e.g. check to make 
sure any new classes have a class comment, etc.

We should probably require that all SUnit tests are run and passed at 
some point in this process, too.  If not when updates are broadcast to 
the unstable stream, then at least it should happen when they're moved 
to the regular stream.  (Something would need to be done with the 
current known-to-fail tests, either fix or remove them, or categorize 
them somehow.)

3. Let some harvesters/guides have direct "commit"/broadcast ability, 
without going through the BFAV approval process, as some have 
suggested.  However, I think this should be limited to only certain 
types of updates, probably just fixes and smallish 
refactorings/cleanups.  New enhancements or major redesigns would still 
need to go through the approval process.

So... how do these ideas sound?

#2 and #3 would hopefully remove me as the bottleneck at the update 
stream, which would be a bit of a relief at this point... ;)

There were also a couple of other things discussed:

- Enhancing BFAV to group posts by package and maybe by other 
groupings.  I think everyone agrees this is a good idea... I get sort 
of annoyed seeing all of the posts in BFAV which are for the SM 
package, or the VM, or Balloon3D... these posts can never proceed to 
Approved/Update.  I'm not sure exactly how this should be done, and I 
probably won't have time to work on this, but I wanted to offer support 
to the idea.

- Committing directly to a Monticello repository for the base image, 
instead of using updates.  This seems like more of a long-term goal, 
not really something which can be achieved during 3.8.  Actually, I'm 
not sure there's any point in doing this until the base image is staked 
out into packages (and stored on SM, with dependencies?)  But perhaps 
the goal was to stake out these areas relatively soon, even if many of 
the packages are "co-dependent" in the dependency scheme.  But in any 
case, I think this is all less urgent as far as speeding up the 
harvesting process goes.  (Although I think we should still add 
Monticello to Basic Squeak in 3.8 as discussed earlier... it is really 
an add-on which should change existing behavior.)

Whew, that's it for now,

- Doug


On Thursday, August 5, 2004, at 03:44 PM, Michael Rueger wrote:

> Marcus Denker wrote:
>
>> I don't think that this would work... at least not as long as we use 
>> an update-stream.
>> It's realy easy to destroy the system with an update. We had example 
>> of faulty updates
>> that manage to slip into the stream and replacing the changesets on 
>> the server is of
>> course possible, but it disrupts everything somehow...
>
> Back in the SqC days we had an internal and an external update stream.
> The internal one was pretty much unfiltered, living dangerous etc. 
> EVeryone living on that stream was used to having backups of their 
> image, ready to go back at any time. Then, when updates seemed ok for 
> a while, they were transfered to the external stream. This way it was 
> easy to retract updates, as they hadn't reached "everyone" yet, but 
> only people who knew how to, and were actually willing to, roll back 
> the changes.
>
> Re-establishing that separation might help speeding up the harvesting 
> process as it would spread the actual testing of an update across all 
> internal stream users.
> Maybe naming these streams differently could help with acceptance ;-)
>
> Michael
>
>




More information about the Squeak-dev mailing list