Good to hear from you Eliot, thanks for your long and considered email.
For me this raises the need to step back from the conversation about the development process and talk about the artifacts the community produces and who the consumers of those artifacts are.
Consumers can fall into some broad categories
- experienced squeakers who want to have at the system in all its
glory and program as the system was intended to permit programming
- new aspiring experienced squeakers who want to use the system in the
same way
- educators who wish to use the system to teach
- students who wish to be taught programming in the context of squeak
- people who want to run some application developed in squeak without
delving into the system other than superficially
- commercial entities through to hobbyists who want to deploy
applications they're written in squeak for profit or not There are probably other meaningful categories, and some people occupy more tan one category concurrently. But you get my drift. Not everyone wants to wrestle with the full glory, but those that do are likely to contribute to the system's subsequent evolution, fix bugs, etc.
It makes sense to me that all but the first want to start with a known quantity, something that is identified, has people who will try and help with problems with it, something around which a community is focussed, something with a hoped-for future, something documented and perhaps even written about. In short a distro. If that distro isn't in some sense a "closed shop", if its anything like a fast-moving target then it'll satisfy only the experienced few, the gurus, who can roll up their sleeves and have at it.
I agree with everything you have said.
So I see a contradiction in, Keith, your promotion of an open development model and at the same time an assumption of greater community participation. I suspect that for most new community participants the existence of a few well-differentiated and well-supported distros is a necessity.
This is how I expect it could work. If the "knowledge" (Sake/Packages) that goes into making distro's is captured, and is published and is part of automated builds (Bob-Releases). Then the "stable known quantity" can be updated and the distro's regenerated on top of a prospective new version of the core. Thus the core can actually be moved forward with the use-cases (the distro's) in mind, since the core can be automatically tested against the distro's. If a core change breaks a distro, then patches to the distro can be added to the "knowledge".
Thus the core may move forward, and it may even take on board radical changes, but it will not be able to move as fast as if it were the only entity, and compatibility is not being considered (i.e. the Pharo approach).
Also anyone can try a new core innovation, and test against the distro's, with the build system we are not restricted to building distro's on top of a single core.
So what would make sense to me is a development model which tries to gather maintennance contributions from those that are capable of and happy to fix problems with the system and harvest them to make stable distros, a clearing house that verifies or rejects potential bug fixes, filters potential candidates for inclusion, so that the maintainers of distros have an easier job in moving their slower-moving targets forward.
That is the current plan...
If we start with 3.10.2 then we have a batch of tested and approved fixes that apply to the current release. The derived image (distro) with all of these loaded is called 3.10.2-fixes. At some point a maintenance release of the most essential of these will be incorporated into 3.10.3.
The next release 3.11 is built upon 3.10-fixes + tasks which apply the design of 3.11. And next unstable release 3.11 is built upon 3.10-fixes + unstable fixes + tasks which apply the design of 3.11.
I want to say more about the kinds of artifacts the community might produce but the weekend is advancing and I'm about to dive into the bosom of my family for the weekend and leave the internet behind. So this is too brief. But I think people want to provide functionality in different forms.
One important form is the cross-dialect package, such as Seaside, which is a core package plus compatibility code to adapt it to various Smalltalks, not just various Squeak dialects.
Sake/Packages provides slots for tweaking the load (i.e. adding a dependency upon a particular bug fix/package) for the different squeak dialects, however supporting different smalltalks is a little bit more tricky, however Sake/Packages purposely has fairly minimal dependencies, so if you there is going to be any viable cross dialect package management solution Sake/Packages could be a front runner.
One form, of necessity, is the package imprisoned in an image, such as eToys and QwaqForums where on the one hand the lack of modularity means that its too difficult to extract a package and on the other hand where the necessities of making radical changes to move the artifact forward result in a fork.
This is where, for the purpose of applying fixes/patches at an integration level, we treat the image as a whole, simply saving the packages afterwards. The current technology for doing this is changesets, but one hopes that deltastreams will be better.
You could even load every package that will load, apply a patch/refactoring to the whole caboodle, and then save all the packages. (e.g. _ to :=) External maintainers would then be able to merge in the result to their externally maintained branch-head.
Other forms are kitchen-sink images appealing to power developers,
Squeak dev can now be built with
Packages load: 'Squeak-dev image'.
various forms of cut-down images attempting to get to a kernel as a suitable starting point for modular composition, and so on.
Sake/Packages supports unloading of packages. So as long as the package is correctly defined and unloads, there is no real need to unload it in core. However some may wish to make a distro which does unload everything that can be unloaded.
We called this the -minimum distro.
As long as no one unloads the ability to run scripts from the command line. Bob can run arbitrary scripts on an image, and then save the result. Distro's dont have to be created with Sake/Packages. However Sake/Packages is also designed to be easily unloadable, all of its data is in code, so you just have to delete the class category, it shouldn't leave unwanted artefacts behind.
All of these would benefit from a more modular system, especially one that had a small bootstrappable kernel that would allow the execution technology to move forward without breaking them, and without having to repeat bootstraps in place (yes I have a perspective here, porting closures to numerous images is not my idea of fun).
It would have been nice for the pharo crew to capture their what they learned while they were "boldy going....". One would also hope that atomic loading could help here.
Is there anything else you can think of that would make it easier?
So again I see issues with your development model. I think it is well-suited to the maintennance task,
Thank you.
and important in getting to the small kernel that we want.
Even better, since these have been the professed goals of the squeak community for several years.
Some appear to be forgetting is that as far as the board were concerned, 3.10 was essentially the end of the line. I encouraged/harassed them to continue 3.10 as essentially an ongoing maintenance release in which those of us using current stuff could continue to consolidate and improve our lot. i.e. I don't expect to see major innovations in the 3.x line, above and beyond the obvious improvements we would like, some of those (i.e. closures) are radical enough for now.
But it needs to dove-tail with a package-based model
Definitely, this is the role that Sake/Packages is intended to fulfil, and I would go so far as to say without a working package mechanism for loading things back in reliably, it's best not to take things out. 3.10 tried using Universes, but I regard that as a failed experiment.
Some of those clamouring for a new release are so keen to have a new image at any costs, they are not realsing that this new development process depends more upon a working Packages infrastructure, and that is one thing that we do now have which we didnt have a year ago.
If you look at how Damien produces the dev image...
http://damiencassou.seasidehosting.st/seaside/pier/Smalltalk/squeak-dev/Buil...
The fundamental line is...
Installer sake install: 'Squeak-dev image'.
And that is it - that is the only line that you have to give to Bob, to build developer images, apart from telling him the starting image, and the period.
because that's the model that gives most power to the community going forward in that it results in packages that can be deployed in different contexts and maintained to some extent independently of a specific substrate, and is the one that allows the community to evolve at its maximum rate. (It is still extraordinary the Squeak has had the same slow VM technology for 14 years).
Totally agree.
Anyway, my 2ยข.
Summary, can we not merge, or if not, marry, your auto-integration testing framework with a package-based approach
Yes by sitting down and carefully carving up the image into packages that are stand alone and have limited dependencies, so that a module is genuinely a package in its own right. This is the primary difference between 3.10 and 3.11 as planned.
the weekend beckons..
enjoy
Keith