Packages March 2005

packages@lists.squeakfoundation.org

10 participants
21 discussions

proposal for starting work on partitioning
by Avi Bryant 08 Mar '05

08 Mar '05

Cees and I just chatted on IRC about tools for supporting partitioning, and coordinating between the package splitting we do and the work done by the morphic splitters. Here's the simple proposal I have to let us get started as quickly as possible, hopefully with minimal conflict and minimal need for coordination: - I'll set up a page on the swiki with a list of packages in the image (if people have string preferences for how granular to make this list, or what should be on it, please say so). To begin with they'll all be up for grabs except the morphic ones. - If you want to do some work on splitting out a package you put a note beside it on the wiki saying that you've started working on it and what the current date is. - When you've done some work on it and are stopping, post a changeset with what you've done to the list, and take your name off of the page. - I'll post the changeset to a special update stream as soon as possible so that people can easily stay in sync. Always try to update from the stream before starting to work on your own splitting. - If someone's had a package "checked out" for a while without posting, use your judgement as to whether or not to override their lock and work on it yourself - maybe email them to see if they'll release the lock to you, or post to the list about it. - In terms of how we attack the splitting itself, it seems to me that it's easiest if people can really just focus on one package at a time - that is, I want to look through all the methods of a package and just decide "is it in this package or isn't it". So I suggest that we work in two passes: first, we try to kick everything out of packages that shouldn't be there, by going through all of the classes in the package and moving any of their methods that shouldn't be in the package to a special "Orphaned" package. Second, or in parallel, we go back through the Orphaned methods and adopt the relevant ones into our package. It would be nice if we had some tools to go through lists of methods in a package, or in Orphaned, with big "Mine" and "Not Mine" buttons, but we can probably do without that for now. If this approach seems sensible to people, I'll set up the small bits of infrastructure required right away. Avi

2 4

Re: Splitting tools
by Dan Ingalls 08 Mar '05

08 Mar '05

Daniel V. wrote... >Just a note here: >The tool Dan describes here, AFAIU, allows one to determine what can be removed after a named set of categories is removed. IIUC, this is useful after the named categories are themselves removable (not depended on by other stuff). > >MudPie can be used before that, to see what depends on what, and therefore what is preventing your package from being removable. > >So they seem to me to be complements. Yes, this was my sense as well. When I (wrote and) used these methods, I already had a sense that, for instance, Celeste, Scamper and MailMessage were the "roots" of most of the web classes [at that time]. Nowadays, you would want to include several other root apps, probably, to get the nice results I got. I believe that Daniel's MudPie would help a lot, both in identifying these natural roots, and in breaking any circularities. - Dan

1 0

Re: splitting tools
by Daniel Vainsencher 08 Mar '05

08 Mar '05

Just a note here: The tool Dan describes here, AFAIU, allows one to determine what can be removed after a named set of categories is removed. IIUC, this is useful after the named categories are themselves removable (not depended on by other stuff). MudPie can be used before that, to see what depends on what, and therefore what is preventing your package from being removable. So they seem to me to be complements. I've updated the SqueakMap version of MudPie to work on Squeak 3.9+Connectors 2.2. Daniel Dan Ingalls wrote: > [I accidentally sent this to Modules, and am cross-posting this to the Packages list, since it seems that similar work is going on there as well. My apologies if you get multiple copies -- Dan] > > >> On Tue, 1 Mar 2005 19:28:09 -0800, Dan Ingalls <Dan(a)SqueakLand.org> wrote: >> >> >>> Does such a tool exist right now? If not, I think it would be a great aid to the project of partitioning the image. >> >> >> It certainly would. When I worked with VW last year, we had such a tool which greatly helped in determining prerequisites. First it would tell you what packages where needed, but it had a detail view that would tell you why these were needed. >> >> With that, we could in theory point at ToolBuilder and some 'favorite morphs' and start hacking away. > > > > > Folks - > > I went back and tracked down my old code and messages to the Squeak list. It is from February 2001 (!!) and I attach the summary below. It turns out that most of the code is still in the image. I haven't tested it, but I'd be happy to help anyone to fix it or use it. > > The principle was extremely simple, and very few lines of code. > Here's an example of its use: > Start with a set of "package roots", such as {Celeste. Scamper. MailMessage}. > First establish the list of classes and methods that appear to be unused > in the full system > Then compute the same result but in the absence of the package roots. > The *newly* inaccessible classes and methods presumably are only > used by package roots, so they should be added to the package. > Iterate the previous two steps until the package does not grow any more. > > The comments with update #3735 below give the excellent results achieved in deriving a package for the three network applications of the time. > > This code all worked four years ago, and it was the basis for the preliminary partitioning in the Environments project. I mainly wanted to point it out, in case it might be of use to people in partitioning the current image. It may be entirely subsumed by Daniel V's MudPie but, if not, then it might save people some time. > > Hope this is of some use. > > - Dan > ----------------------------------------- > > > 3735BetterShrink-di -- Dan Ingalls -- 27 February 2001 > Introduces a new image partitioning tool that is essentially a local version of majorShrink. The idea is to pick some application like Scamper, or some cluster like all the 3D classes, and remove it, and also remove everything that is solely referred to by it. The way it works is: > Record all unsent messages and unused classes at the outset > Mark the application or cluster as removed > Note all *newly* unreferenced methods and unused classes > and iteratively remove them as well > For example, the expression > Smalltalk reportClassAndMethodRemovalsFor: #(Celeste Scamper MailMessage) > reports 63 classes and 155 other messages that can be removed. > Also, introduces a new method, fileOutAndRemove:... that will take such results, build a changeSet from them, fileOut everything that is about to be removed, and then remove all of the classes and related messages as well. For the above example, this method produces a 290k fileOut, and saves about 104k from the system. More importantly, if you fileIn the changeSet afterward, the system returns to approximately its former size, and you can run Scamper again. It's a way to fileOut a package that never existed. Tools for Pools yet to come. >

1 0

re: Splitting tools
by Dan Ingalls 07 Mar '05

07 Mar '05

[I accidentally sent this to Modules, and am cross-posting this to the Packages list, since it seems that similar work is going on there as well. My apologies if you get multiple copies -- Dan] >On Tue, 1 Mar 2005 19:28:09 -0800, Dan Ingalls <Dan(a)SqueakLand.org> wrote: > >>Does such a tool exist right now? If not, I think it would be a great aid to the project of partitioning the image. > >It certainly would. When I worked with VW last year, we had such a tool which greatly helped in determining prerequisites. First it would tell you what packages where needed, but it had a detail view that would tell you why these were needed. > >With that, we could in theory point at ToolBuilder and some 'favorite morphs' and start hacking away. Folks - I went back and tracked down my old code and messages to the Squeak list. It is from February 2001 (!!) and I attach the summary below. It turns out that most of the code is still in the image. I haven't tested it, but I'd be happy to help anyone to fix it or use it. The principle was extremely simple, and very few lines of code. Here's an example of its use: Start with a set of "package roots", such as {Celeste. Scamper. MailMessage}. First establish the list of classes and methods that appear to be unused in the full system Then compute the same result but in the absence of the package roots. The *newly* inaccessible classes and methods presumably are only used by package roots, so they should be added to the package. Iterate the previous two steps until the package does not grow any more. The comments with update #3735 below give the excellent results achieved in deriving a package for the three network applications of the time. This code all worked four years ago, and it was the basis for the preliminary partitioning in the Environments project. I mainly wanted to point it out, in case it might be of use to people in partitioning the current image. It may be entirely subsumed by Daniel V's MudPie but, if not, then it might save people some time. Hope this is of some use. - Dan ----------------------------------------- 3735BetterShrink-di -- Dan Ingalls -- 27 February 2001 Introduces a new image partitioning tool that is essentially a local version of majorShrink. The idea is to pick some application like Scamper, or some cluster like all the 3D classes, and remove it, and also remove everything that is solely referred to by it. The way it works is: Record all unsent messages and unused classes at the outset Mark the application or cluster as removed Note all *newly* unreferenced methods and unused classes and iteratively remove them as well For example, the expression Smalltalk reportClassAndMethodRemovalsFor: #(Celeste Scamper MailMessage) reports 63 classes and 155 other messages that can be removed. Also, introduces a new method, fileOutAndRemove:... that will take such results, build a changeSet from them, fileOut everything that is about to be removed, and then remove all of the classes and related messages as well. For the above example, this method produces a 290k fileOut, and saves about 104k from the system. More importantly, if you fileIn the changeSet afterward, the system returns to approximately its former size, and you can run Scamper again. It's a way to fileOut a package that never existed. Tools for Pools yet to come.

1 0

Get this party started...
by Avi Bryant 06 Mar '05

06 Mar '05

Hi all, I thought I'd kick off this mailing list with my (hopefully brief) thoughts on what our goals and priorities might be; hopefully y'all will post similarly, and we can start to reach a consensus on what it is we're trying to do. On the Teams page at http://anakin.bluefish.se/castaways/6, the Packages team is described as "Partitioning, PI, packages, TFNR, SM, Universes etc. Evolving the tools and assigning parts of the 3.9 image, and much more." So that's our scope. Personally, I read this mainly as two interlinked goals: 1. To separate the code currently in the image into discrete sections with minimal and well understood dependencies on each other. 2. To structure our community so that it's feasible and robust to manage what Lex talks about as the "commons" of Squeak as many small pieces of code rather than as one large one. Note that these are both primarily organizational or social goals rather than technical ones: the point is to know where the boundaries between packages are, and to know (as a user) which packages to load at any given point, not the particular mechanisms that enable this. In particular, I'd like to leave discussions about how to represent individual packages either in the image or on disk to other groups (like the Modules) group - we have ways to do this already (PackageInfo, SAR, .mcz), and although they may not be the best ways, my guess is that they will work as well as any to meet our goals. Even if someone does have a reason why a different technical approach will significantly change the way in which we can organize code, or make us much more productive in doing so (it's conceivable that MC2 would fit into this category, for example), I suggest that we plan to continue using the status-quo packaging systems at first: the main work of defining boundaries should translate easily to any new technology that comes along, and so we shouldn't sit around waiting for it. Where I do think we'll need to develop or advocate new technology is to better manage packages as groups: the more our world is split into packages, the more we will need to be able to specify the dependencies and compatibilities between sets of packages or package versions, and we don't yet have anything widely used that does this. With Göran increasingly busy with other stuff, I don't know how much we should expect him to do on SqueakMap to address this, but I suspect that one role of the Packages team will be to slowly take over the maintainance of that part of Squeak's infrastructure. We can't ignore the social component here too: the major thing that SqueakMap has going for it is how accessible and thus how widely used it is, and we have to make sure not to jeopardize that. The other major technological and organizational decision I expect we'll have to make is how to reconcile the update stream with a packaged image: do we lose the update stream altogether? Do we make some updates optional (only applied if package X is loaded)? How do we convert from snapshots of package versions to changesets appropriate for the stream? And so on. I don't have any immediate answers, but there are definitely a lot of questions there. So a very rough draft of a plan might include: 1. using PackageInfo (and maybe tools like Spaghetti Tracer) to split the image into minimal dependency chunks, which would allow: 2. for as many packages as possible, unloading them and maintaining them purely on SqueakMap, which would require: 3. enhancing SqueakMap to model sets of compatible packages, and dependencies between them, which leaves: 4. for the remaining packages, building technology that lets us convert between a view of the image as a set of versioned packages (for developers), and a view of the image as a monolithic application with incremental updates (for endusers). Those are the things that come to mind, but these are early thoughts and I hope and expect to change them once others have posted theirs. Cheers, Avi

2 4

Re: user types
by Cees de Groot 06 Mar '05

06 Mar '05

On Sat, 5 Mar 2005 16:59:46 -0400, Lex Spoon <lex(a)cc.gatech.edu> wrote: > There are several familiar user types: > We call them 'personas', and we often attach a full bio and even a (random ;)) mugshot to them. So in your project you can just say "Jim" wouldn't like this, and everyone would know that this would apply to "Jim Frensby, 28, CS PhD, wrote his thesis on event horizon user interfaces and is currently working on further research in this area. [etcetera...]". By developing personas you get 'live' customers to think about instead of some abstract group. Might be useful for our community as well... It is very important to keep driving the point home that this is not just for Joe Hacker :)

1 0

user types
by Lex Spoon 05 Mar '05

05 Mar '05

Avi points out that a packaging commons needs to support social mechanisms -- and perhaps even engineer new ones. This observation has a lot of ramifications. It means there is not an obvious Right Solution. It means that version 1 may well not work the way it was expected, and thus that we need to be prepared to watch how the tools actually get used and adapt to what people want. Some features, painfully, might not turn out to be useful at all. We should be careful not to get so attached to our theories, that we refuse to recognize what the users are actually doing. But we have to start somewhere! Let me try and sketch some user types that seem nice to support. This is by no means a final list, but we have to have *some* working model. Please add anyone I left out, or adjust any types you think are off the mark. There are several familiar user types: 1. J. Hacker goes to www.squeak.org, gets hooked, fools around, and is excited enough to develop some code that they'd like to share with the other Squeak users. It's important to support J. Hacker's, because they are the main source of all this great code that we'd like to share around. 2. Fred Exec likes golfing but hates computer innards. If someone gives him a CD with SqGolf 3000 on it, then Fred should be able to use it and to apply updates as Avi described in the kickoff email message. Let's keep in mind some less familiar types, as well: 3. The Scratch project is a local work group within a research lab, and they'd like to build something cool in Squeak. They would like to use stable packages that have been posted to the 'net, but they do not need the bleeding edge. They plan to eventually share their project on the web, but they would like in the meantime to keep their ugly looking code to themselves. Nevertheless, they are multiple people, and they can greatly benefit from having the repository tools still work for them. 4. Joe Porter might want to port BottomFeeder from VisualWorks to Squeak. Our infrastructure should allow redistributing stuff which is primarily maintained somewhere else. 5. HP Labs has some cool Squeak code that only makes sense for use within the organization. It would be nice if HP lab rats can use the package repository infrastructure to share this code amongst themselves. They should be able to do this regardless of which other Squeak projects they are involved in; e.g., they should be able to collaborate with the Scratch guys using an image that has HP Lab packages loaded. Some initial implications on the tools, based on these user types: There should be some very easy publishing mechanism available, so that J. Hacker can get their code seen by people who are interested. There is a big difference between posting something on a web site, and posting it to something like SqueakMap. For mysterious reasons, packages in the former category seem likely to wither compared to the latter. There should be a way to have local repositories, so that the Scratch group can make progress. One implication of this is that identifiers of all kinds are going to have conflicts between the local repositories and the well-known global repositories. It looks like mixin repositories would be useful, in order to support HP Lab's internal packages. Along these lines, it would be ideal if it is possible to manage a package manually, without reference to a repository. It should be possible for one Squeaker to send a newly created package file to the Squeaker sitting beside them on another computer, without invoking lots of infrastructure. -Lex

1 0

user types
by Lex Spoon 05 Mar '05

05 Mar '05

1 0

user types
by Lex Spoon 05 Mar '05

05 Mar '05

1 0

Untangling the code (was: Get this party started...)
by Daniel Vainsencher 05 Mar '05

05 Mar '05

One of the things you mention is the process of splitting up the existing code so that it can be managed and loaded in chunks. This is exactly the process I had in mind when I wrote SpaghettiTracer, since then renamed MudPie. First generalities just to put us all on the same page, then I'll describe MudPie, then some of my thoughts about what would help to make some progress on this. When I say progress, I mean on the following specific goal - *over time, the code in the squeak-dev released image, becomes divisible into smaller packages* I follow Avi in completely ignoring for now the code container technology, and further also ignore whether and how code actually is moved to packages. Problem: Squeak includes lots of code, and in terms of dependencies, not all code is born equal. A. There is some Kernel Stuff, which must be there for anything to run. Includes all the classes that the VM knows about for example. Everything depends on this. The kernel (shouldn't) depend on anything else. B. There are libraries that we are used to, for example Morphic, that is dependent on the kernel, and that applications are dependent on. C. Applications like Celeste depend on many different libraries. The most obvious thing about dependencies is that their direction should be A<-B<-C, and nothing else. We can't break Squeak into lots of little packages because this is not the case. In fact, Kernel classes depend on Morphic, Morphic knows intimately some things that most people would not call libraries. We have cycles. This means that after an application is removed, Morphic can break. MudPie: What MudPie does, is identify these cycles. It also allows us to write SUnit tests that say "X and Y should not be in the same cycle", so that once something like that is fixed, it stays fixed. MudPie at the moment, IIRC, works, but has zero UI. There are many kinds of loving it could use in fact, including a rewrite, but it can do what I said as is (modulo changes in PackageInfo since it was written). At a higher level, MudPie allows us to, at every moment, get an answer to: 1. "what can I break off and package now", and 2. "what is preventing me from breaking off and packaging <XYZ>". For something that is clearly an application, like Celeste, MudPie isn't really needed, because since we know it's okay that it depends on lots of stuff, and being an application, nothing should depend on it, we can just ask "what depends on Celeste", and PackageInfo does that nowadays. Same goes for the Kernel. I think MudPie should become useful when you want to: 1. Assert/see something about the global structure of the code. 2. Deal with the middle level, for example untangling libraries that are mixed into the whole system, like Morphic. Elements in a solution: 1. Making the situation visible. Right now it is hard to measure progress. People "know that Squeak has to much in it", but that is very vague. We can easily get the answer to the question "What code outside the Kernel does the Kernel depend on, transitively". This one question gives us lots of visibility into what is wrong, cheaply. Get the number of classes from that code, and we have a metric and a goal - 0. Later on, if we want more details, MudPie can give lots of them (how much of a suite of "modularity tests" is green, for example). Anyway, whatever metrics we choose to code up, should be widely available, and not require every person interested to learn to write some magic. We need a UI, or better, a web UI to this information, maybe with history. 2. Some big refactorings. I've seen lately calls for making Squeak's UI frameworks be pluggable components, so that for example, inform: does the right thing, whether Morphic, Tweak, or SeaSide is the current UI. This kind of refactoring would break the upwards dependencies that create most of the cycles. BTW, note that using MudPie doesn't require any further declarations or new constructs, so the actual work of untangling code is nothing more than refactoring (often just the renaming of method categories, and moving classes between class categories). As the packaging team, we should identify these refactorings, cooperate with people doing them to make sure the chosen solution actually breaks the cycles, and help get them into the image (reviews and so forth). 3. Maximize benefits: tools Regardless of whether the code is packaged separately, having the code dependencies be a DAG gives us various ways in which the tools can be smarter. For example, I wrote a little something for the star browser that does a topological sort on the Class Categories, according to the dependencies, so that applications end up above libraries, which end up above the Kernel, giving some meaningful clues about the meaning of the code to the user. Telling the user which SUnit tests to fix first is also an obvious application. 4. Maximize benefits: community Seems to me that untangling the code will give us more opportunities to swap code among the various Squeak subcommunities. Just saying that we should be alert to this, help it, and let it drive us. Looking forward to hearing your thoughts on this, -- Daniel Vainsencher

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Packages March 2005