From Dan at SqueakLand.org Tue Mar 1 07:11:53 2005 From: Dan at SqueakLand.org (Dan Ingalls) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready Message-ID: Folks - First of all, special thanks to those of you who took the time either to write a sort of position paper, or to fill out one of the summaries. This really makes a difference. It's exciting to embark on this project with such a knowledgeable crew. That said, I want to get the entries for Naiads and Islands filled out, and I'm hoping to get something on Universes as well, before we really blast off. Craig, Andreas, and Lex, if you are listening, would you be willing to take a crack at these? Anyone else? See the New Modules page, http://minnow.cc.gatech.edu/squeak/5608 Meantime, here are some appetizers to get you ready for the main course... Think about a little PDA with ROM and RAM. Suppose we blew the squeak image in to the ROM. Could we run with it in ROM? When we wanted to save, could we only save the differences from the ROM? If we started it up again, and made some more changes, could we just save those changes? What do these incremental snapshots suggest about modules? Consider Packages and Projects. We might well open a new project, build a package, and store it as a project. Or as a package. Compare and contrast these two modes of use, in particular noting how they might use the same underlying unit of modularity, and further noting in what other aspects they differ. Imagine a change to the VM which would allow message lookup to be intercepted. If we are writing some application and want to add a few methods to class String, then we put them in a StringExtensions class in our package (sometimes called a virtual class). A little table in the VM notes this and, when a message is looked up in class String, it is first looked up in StringExtensions. Things may go a little bit slowly on the first lookup but then, of course, the method goes into the cache and there is no performance penalty. But look -- we have not modified class String at all. Time to uninstall? Zero. Now imagine that the VM can have multiple method caches, one per, um, per process? per module? Think about it. And when you have figured it out, note that this system could run simultaneous multiple projects with conflicting changes to root classes with no performance degradation. Think about an image nicely partitioned into lots of little segments corresponding to different modules. Might it make sense to garbage collect them separately? And what about other aspects of storage management? Supposing we are concerned about security, and someone attacks the system by consuming more and more storage. How do we catch that? How do we make sure that it only affects the requestor, and not the rest of the system? More later - Dan From dfarber at numenor.com Tue Mar 1 16:48:31 2005 From: dfarber at numenor.com (David Farber) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready Message-ID: <3.0.6.32.20050301094831.01e0cb90@mail.frii.com> At 11:11 PM 2/28/2005 -0800, you wrote: >Now imagine that the VM can have multiple method caches, one per, um, per process? per module? Think about it. And when you have figured it out, note that this system could run simultaneous multiple projects with conflicting changes to root classes with no performance degradation. Dan - This is exactly what I want to accomplish with Sonora. Imagine a Workspace that understands an extended syntax that lets you define methods and classes in addition to the normal code snippets. ----SonoraWorkspace 'Hello!' asRot13. String>>asRot13 ^self collect: [:ch | ch asRot13] Character>>asRot13 ((self >= $a) & (self < $n)) | ((self >= $A) & (self < $N)) ifTrue: [ ^ (self asInteger + 13) asCharacter ]. ((self > $m) & (self <= $z)) | ((self > $M) & (self <= $Z)) ifTrue: [ ^ (self asInteger - 13) asCharacter ]. ^ self ---- If you select all of the code above and 'PrintIt' you should get 'Uryyb!'. The trick is, the two asRot13 methods should only be visible to messages and objects that originate from this invocation of the workspace. So far, only the code that understands the extended syntax has been written. To implement the "module" behavior I want, I was thinking about hacking in some ugly, behind-the-scenes, anonymous class manipulation. Obviously, if we can come up with a cleaner, more general approach, that would be better. david -- David Farber dfarber@numenor.com From craig at netjam.org Tue Mar 1 19:40:55 2005 From: craig at netjam.org (Craig Latta) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready In-Reply-To: References: Message-ID: <4224C547.3020403@netjam.org> Hi Dan and all-- > ...Naiads... "Naiad" :) > ...if you are listening, would you be willing to take a crack at > [a summary]? Will do... I've been busy the last few days with house stuff. > Think about a little PDA with ROM and RAM. Suppose we blew the squeak > image in to the ROM. Could we run with it in ROM? Having small object memories should help here. :) Good appetizers... -C -- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth! From Dan at SqueakLand.org Tue Mar 1 19:58:44 2005 From: Dan at SqueakLand.org (Dan Ingalls) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready Message-ID: [From the salutation, I think Trygve meant to CC this list. Here's his summary of BabyUML...] >Date: Tue, 01 Mar 2005 10:16:31 +0100 >To: Dan Ingalls >From: Trygve Reenskaug >Subject: Re: Getting Ready > >Greetings, modulers > >Your work is really exiting and highly relevant to what I am trying >to do. I expect to build on the emerging modules by encorporating >them into my components. It could even be that my components are >relevant to the modules effort (happy thought!). > >I am working on a BabyUML report and hope to release a first draft >in a few weeks. Here are some main points: > >* I need a high level programming language/environment for systems of objects. >* I need a stored program object computer (like Smalltalk/Squeak). >* I think and work in terms of systems of interacting objects, > classes are also objects in my descriptions. >* Object (sub)systems are encapsulated in components that are special objects. >* Components are characterized by their provided and required interfaces. >* A component appears as a black box object in its environment, > but can be opened up to reveal its internals. >* Components are connected by links to other components/objects > that use the required and supply the provided interfaces >* A component can encapsulate objects of any kind; > typically both application and class objects. >* Generalization/specialization between components is under study; > the starting point is the UML 2.0 merge operation. > >The main purpose of the component is to divide and conquer. It could >also be a unit of security/privacy and of persistence and reuse. > >The regular ST class with its metaclass siamese twin feels >obfuscating in this context. I want the class objects to be treated >and understood as regular objects that are visible parts of my >system if I need to see them; I will then probably see them as >factory objects. > >I am creating a new metaclass architecture: >- BMetaMetaclass is the mother of all classes. >- BMetaSimpleclass is the grandmother of regular objects > (and the mother of the regular classes). >- BMetaComponent is the grandmother of the components >- Each metaclass has its own IDE > and the high level program description, > possibly including graphics, IS the program >- The new objects and components interact seamlessly > with regular Squeak objects > >More later >--Trygve > >At 08:11 01.03.2005, you wrote: > >>Folks - >> >>First of all, special thanks to those of you who took the time >>either to write a sort of position paper, or to fill out one of the >>summaries. This really makes a difference. It's exciting to >>embark on this project with such a knowledgeable crew. >> >>That said, I want to get the entries for Naiads and Islands filled >>out, and I'm hoping to get something on Universes as well, before >>we really blast off. Craig, Andreas, and Lex, if you are >>listening, would you be willing to take a crack at these? Anyone >>else? See the New Modules page, >> >> http://minnow.cc.gatech.edu/squeak/5608 >> >> >>Meantime, here are some appetizers to get you ready for the main course... >> >>Think about a little PDA with ROM and RAM. Suppose we blew the >>squeak image in to the ROM. Could we run with it in ROM? When we >>wanted to save, could we only save the differences from the ROM? >>If we started it up again, and made some more changes, could we >>just save those changes? What do these incremental snapshots >>suggest about modules? >> >>Consider Packages and Projects. We might well open a new project, >>build a package, and store it as a project. Or as a package. >>Compare and contrast these two modes of use, in particular noting >>how they might use the same underlying unit of modularity, and >>further noting in what other aspects they differ. >> >>Imagine a change to the VM which would allow message lookup to be >>intercepted. If we are writing some application and want to add a >>few methods to class String, then we put them in a StringExtensions >>class in our package (sometimes called a virtual class). A little >>table in the VM notes this and, when a message is looked up in >>class String, it is first looked up in StringExtensions. Things >>may go a little bit slowly on the first lookup but then, of course, >>the method goes into the cache and there is no performance penalty. >>But look -- we have not modified class String at all. Time to >>uninstall? Zero. >> >>Now imagine that the VM can have multiple method caches, one per, >>um, per process? per module? Think about it. And when you have >>figured it out, note that this system could run simultaneous >>multiple projects with conflicting changes to root classes with no >>performance degradation. >> >>Think about an image nicely partitioned into lots of little >>segments corresponding to different modules. Might it make sense >>to garbage collect them separately? And what about other aspects >>of storage management? Supposing we are concerned about security, >>and someone attacks the system by consuming more and more storage. >>How do we catch that? How do we make sure that it only affects the >>requestor, and not the rest of the system? >> >>More later >> >> - Dan >> > >-- > >Trygve Reenskaug mailto: trygver ifi.uio.no >Morgedalsvn. 5A http://heim.ifi.uio.no/~trygver >N-0378 Oslo Tel: (+47) 22 49 57 27 >Norway -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.squeakfoundation.org/pipermail/modules/attachments/20050301/f104ddf8/attachment.html From mroos at roos.com Wed Mar 2 01:51:51 2005 From: mroos at roos.com (Mark Roos) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready In-Reply-To: Message-ID: Dan, I had no reason to limit the conversation to you, just an email error. In way of explanation on my question I have a product which is a single image consisting of 50 or so applications. Each application is built from 'modules' which are groups of behavior managed by an individual. Updates are performed by 'patches' which are autoloaded at image startup and are created by whoever discovers the problem. Some requirements were for distributed autonomous development. kibitzing of others code and a desire by our users to never have a bad experience due to upgrades. While this was a good approach when we were developing code now that we are in an incremental enhancement mode we are finding that keeping the applications running while updating the modules is becoming a pain. After looking at our options I am pursuing the concept that each application is its own image, modules namespace and patches thus moving away from the single image approach. Of course to make this work I need to keep the nice parts of a common image such as object communications between applications and sharing of the cpu. With this in mind I have some interest in your idea to have multiple projects ( similar to our applications ) in a single image. This appears to have some of the characteristics that I am looking for but it seems like the separate image approach offers a cleaner separation. Do you have any thoughts on the relative merits of the separate projects in a single image vs the independent project per image approach? mark Dan Ingalls 03/01/2005 12:46 PM To Mark Roos cc Subject Re: Getting Ready Dan stated. Now imagine that the VM can have multiple method caches, one per, um, per process? per module? Think about it. And when you have figured it out, note that this system could run simultaneous multiple projects with conflicting changes to root classes with no performance degradation. How do you envision this being different, or better, than several independent Squeak processes with some form of tight message based communications and a good scheduler? Hi Mark - I'm not sure what you're proposing as the thing to compare against. In Squeak there is no current way to have conflicting changes to root classes. Can you spell out a bit more what you mean by the above one-liner? p.s. I am liking a lot the idea of using the cache to handle overrides Me, too. We've already got most of the mechanism, and putting a level of indirection on it offers a lot of leverage. - Dan By the way, it's fine with me if you'd like to include the list on this. It's also fine to keep it offline. Just wanted you to know. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.squeakfoundation.org/pipermail/modules/attachments/20050301/3a481b8a/attachment.htm From Dan at SqueakLand.org Wed Mar 2 04:19:31 2005 From: Dan at SqueakLand.org (Dan Ingalls) Date: Wed Jul 19 10:23:23 2006 Subject: Modules In-Reply-To: <200502260012.j1Q0BhjP015155@squeakland.org> References: <200502260012.j1Q0BhjP015155@squeakland.org> Message-ID: Hi Jecel - [sorry it has taken me a while to catch up] >But back to "several problems at once", a trivial example is virtual >memory. Squeak doesn't have it which isn't a problem on most systems on >which it runs since they have their own paged virtual memory systems. >But if you can quickly load and unload image segments, that can be used >as an extremely simple virtual memory system which would come in handy >in a PDA which can only access the large Flash memory indirectly, for >example. Yes. You know, a couple of years ago, we got this actually working with Alan's big demo image. It had grown to about 200MB, and we got it back down to something like 50Mb by swapping out projects using ImageSegments. There was a little delay going into new projects, but usually nothing major. In fact the system was *freer* of long delays because full GCs went 4 times faster! That was all upside. The downside was that it took Ted and me a month to track down all the misbehaved pointers between projects, and we simply couldn't keep the thing well behaved. That is actually what inspired the Environments work -- an attempt to structure the world so that there were no pointers into the innards of other projects. Andreas's Islands are similarly motivated. > > I also feel that it is possible to keep the modularization nearly invisible to >> the casual user. > >True, and this is what I like about the Traits thing or G?ran's >namespaces - anyone who didn't have problems with the old system doesn't >see them at all. It is only when you try to do something different where >things would break down in plain Squeak that you have to learn the new >ideas. > >It isn't easy, however. The changes tend to leak out. For example: I am >building two Smalltalk systems. The 16 bit one is a traditional closed >one where things like "ClassX allInstances" make perfect sense. The 36 >bit one is modular and as an open system that expression's result is >less useful. In particular, you can never be sure that you have seen all >the instances "out there". An outdated explanation of my "modules" can >be found in http://www.merlintec.com:8080/software/8 if anyone is >interested, but a lot of that might not be relevant for Squeak. Yes! We had the same problem with enumeration in our module image. It was important if you wanted to reshape a class. Ted wrote special code to deal with this. >While I started seriously thinking about this in 1984 (I thought an >image based system was fine in a lab, but wouldn't be as nice in a >product) the only stuff in this area which I designed and was actually >implemented and tested was an OO OS for the PC AT (286, for you young >ones) in 1988. > >One thing that I see as a problem in the way of properly partitioning >Squeak is that we use a single mechanism to deal with different kinds of >coupling between objects. Fixing that would require changes visible to >the users, so this is probably not the way to go for this project but >I'll explain it anyway. As Alan Cox likes to point out, in the real >world you have objects that are soldered together, others that are >bolted together, that are wired together with connectors that are easy >to snap on/snap off and so on. But when I do > > inst := MyHelper new. > >and distribute this as text to be recompiled in another user's system, >the coupling is looser than it should be. Their environment is different >and the text could be ambiguous. I know exactly what object I am talking >about and having an intra-module pointer to it would be just right. The >other user shouldn't be bothered by the name I used at all, as I think >Craig Latta would agree. > >On the other hand, > > p := HTMLParser new. > >is far too tight a coupling for my taste. Replacing the text with an >inter-module pointer wouldn't really help. This kind of dependency might >allow the module I used while developing to be autoloaded into the >user's system when my module is, but is that what we want? As long as p >is the right type (understands a given set of messages) the job will get >done. If the user's system has a different, but compatible, parser then >I would like to use that instead. Some global Protocol (or Interface or >whatever) objects that acted as factories would be nice: > > p := HTMLParsingProtocol makeInstance. > >would check the loaded modules, fetch one according to the local >preferences if none of the loaded ones has the needed class and finally >would get the class to create a new instance for us. There is no reason >why this shouldn't be written exactly as the previous expression (with a >Protocol instance simply replacing the class and implementing #new), of >course, and then any such change would be far less visible to the users. > >In short: my suggestion is that checking all class references in the >image (and packages) and separating those that should be tightly coupled >from those that should be loose (by making the latter indirect) would >make partitioning the image much easier. Yes, I think we want something like that. In fact, I'm leaning toward all global refs being indirect. Then at least the binding mechanism is uniform, and dealing with near and far references is simply an administrative issue. >About the modules themselves, I like binary ones like imageSegments >better because they apply to all objects instead of just sources. My >preference is also to refer to modules by some universal ID and then >have a separate service to map that to a local file name or URL from >where it can be loaded. This allows me to move and rename them without >breaking stuff. For immutable modules a good ID is simply the hash of >the contents. A sequence of immutable modules can simulate one mutable >one (TeaTime?) and again I would rather have a separate service deal >with this than hard coding it in the modules themselves. I think I pretty much agree here. >The iAPX432 (Intel's first 32 bit processor, for you young ones) got me >interested in capability based security. Having different people see >different interfaces for the same object is the most flexible way to >express capabilities. I can imagine more than one way to implement that >and for all of them you would get the ability to put extensions to one >module inside another and to load incompatible stuff simultanously "for >free". > >One more thing ;-) (sorry that this is so long already) - in a binary >module system I would move the sources from external files into the >image (not really - into a module). So I would have a "main" module that >would include the class objects, the compiled methods and anything else >that is needed at run time. This shouldn't be too big and would be the >only thing that most users would ever load. I agree here in spirit -- I want the sources to go with the external form of the module -- on a server like SqueakMap. When you download it, it comes to your disk as is, with binary segment and sources segment. Then, when you load it into your image, the binary segment gets loaded, and a sources pointer gets established for the material in that segment or module. That way, it continues the tradition of binaries in memory, sources on the disk. >A "source" module would have >a set of string objects, which happen to be the sources for the stuff in >"main", which would have inter-module pointers to the strings so trying >to see the methods in a browser or debugger would automatically load >"source". I would include at least a third module, "docs". This would >have objects with far nicer documentation than just the sources, and I >would include in that a set of test methods. This documentation would be >more at a package level than method comments or class comments and could >include animations and all kinds of neat stuff, including pointers to >related objects in other modules. This would automatically be loaded >when my actions in the browser implied "serious" use of the module. At this point, I'm not sure. You've thought about it, and I'm sure others have. It's more into the packages discussion and, for a while at least, I'm going to focus on the runtime architecture. Thanks for all your comments. - Dan From goran.krampe at bluefish.se Wed Mar 2 11:37:24 2005 From: goran.krampe at bluefish.se (goran.krampe@bluefish.se) Date: Wed Jul 19 10:23:23 2006 Subject: A bunch of observations Message-ID: Hi all! I just (phew) read most of the "Modules" thread and took some notes that I would like to share with you. I am trying to be brief. First of all - some of the most profound postings came (not unexpectedly) from Colin, I think we all should read them one more time, this is the first one: http://lists.squeakfoundation.org/pipermail/squeak-dev/2005-February/08 9005.html The most important points here IMHO: - Colin supports Alan Lovejoys definition of "separately deployable code" (well, not "objects" perhaps). I do too. We should "choose" what a "module" is, and that is IMHO the overshadowingly most important attribute of it. Though "objects" is a better word than "code". - The decoupling of name resolving. This one is a no-brainer, we should just factor that out, regardless of how we intend to actually make it work. ..and then this one which is *very important* I think: http://lists.squeakfoundation.org/pipermail/squeak-dev/2005-February/08 9304.html - Here Colin captures very nicely a way to partition these matters. Agree? Now in random order: 1. Resolving names. A recent post pointing at how Forth does this reinforced the idea that unbound references (typically names of classes) should be resolved and bound at "load time" and then after that "be what they are". Again this is my perception, may be wrong. Colin is also leaning in that direction AFIACT, and instinctively I would agree. :) Note: In my namespace work I have always considered that resolving a name like HTML::Parser (or whatever) can be done using various techniques and mappings. Even if it is an absolute name (a prefix/namespace and a name in that space) there is nothing stopping the resolver to for example map to another namespace or whatever. It is only helpful that it doesn't "just" say Parser. 2. Dan talked about having a module hanging by a single reference and thus be ImageSegment-friendly and also be "user friendly" by saying that you have the module as long as it is "there". We could typically just have a Collection somewhere with the Module instances in it - and if you remove one, it is thus "uninstalled". I agree, it would be a *very* nice model - *if* we can pull it off. :) It reminded me of Gobo-Linux (http://www.gobolinux.org) - a new Linux distro which actually tries a similar thing, each installed piece of software lives in a single directory on disk. An uninstall is simply done by removing the directory (I think). 3. Colin and the others are working hard on MC2 etc and I just want to give a hint on something extraordinarily cool in that arena. Darcs (http://www.darcs.net). Darcs is a new distributed version control tool that has something that AFAIK no other tool has - a "Theory of Patches". The net result is remarkable smartness, bordering on magic. :) Slate just moved over to using Darcs and even though I have no current use of it, it is my preferred choice today over all the others (Arch, Svn, yaddayadda). Perhaps a few things in there to sneak a peek at? Perhaps Colin already has. :) 4. And finally a general observation - "Isolation considered harmful?". Many of the ideas and concepts in this area (modules, namespaces, versioning etc) are about further isolating things from each other - in the belief that more isolation improves stability and is only a good thing. Sure, it sounds reasonable on first sight - if you have very few dependencies, then the risk of breaking when the world around you changes is less. But... it also means that things that *should* be coupled are instead duplicated and forked off. One example is namespaces. People often advocate a total isolation approach - each developer gets his/her own little sandbox, and we all live happily ever after, right? Wrong IMHO. This is what I call "pessimistic" namespaces. If each of us are isolated from the rest, then we are bound to reimplement things over and over - a dozen SocketStream classes, multiple SMTPClients etc etc. So isolation is NOT only a good thing - we need to have processes that enable us to discover things - like for example "discover" that there already is an SMTPClient class and prevent us from making our own, and also encourage us to improve on the one that already exists. regards, Göran From avi.bryant at gmail.com Wed Mar 2 12:29:41 2005 From: avi.bryant at gmail.com (Avi Bryant) Date: Wed Jul 19 10:23:23 2006 Subject: A bunch of observations In-Reply-To: References: Message-ID: On Wed, 2 Mar 2005 12:37:24 +0100, goran.krampe@bluefish.se wrote: > 3. Colin and the others are working hard on MC2 etc and I just want to > give a hint on something extraordinarily cool in that arena. Darcs > (http://www.darcs.net). Darcs is a new distributed version control tool > that has something that AFAIK no other tool has - a "Theory of Patches". > The net result is remarkable smartness, bordering on magic. :) Darcs was, in a way, a major inspiration for the design of MC2 - as you may remember, I was looking into darcs around the same time that you were a few months ago. After studying it, the conclusion I came to was that the "theory of patches" is actually unnecessary in a Smalltalk context: its purpose is to figure out the dependency between versions of an individual piece of a file, given the dependency between versions of the file as a whole. Monticello has roughly the same problem, so it was familiar to me, but MC also suggests a simpler solution: since we already keep track of the individual pieces (methods), we can just record their dependencies directly, and the whole inference mechanism behind darcs can be thrown away. All you need is method-granularity versioning (as in ENVY), combined with knowledge of multiple ancestors (as in Monticello), and you get the same effect as Darcs, but more reliably. That's the basis of MC2. In terms of file-based versioning systems, the design ends up being closer to Codeville (http://codeville.org) - I wonder if Bram looked at Darcs too and came to the same conclusions. Avi From goran.krampe at bluefish.se Wed Mar 2 12:47:26 2005 From: goran.krampe at bluefish.se (goran.krampe@bluefish.se) Date: Wed Jul 19 10:23:23 2006 Subject: A bunch of observations In-Reply-To: References: Message-ID: Hi Avi! This was an interesting background - and if it is like you say then I really have expectations for MC2! :) regards, Göran From cg at cdegroot.com Wed Mar 2 13:30:03 2005 From: cg at cdegroot.com (Cees de Groot) Date: Wed Jul 19 10:23:23 2006 Subject: A bunch of observations In-Reply-To: References: Message-ID: On Wed, 2 Mar 2005 13:47:26 +0100, wrote: > This was an interesting background - and if it is like you say then I > really have expectations for MC2! :) > I have very high expectations for MC2 regardless ;) From jecel at merlintec.com Wed Mar 2 13:26:39 2005 From: jecel at merlintec.com (Jecel Assumpcao Jr) Date: Wed Jul 19 10:23:23 2006 Subject: Flash memories (was: Modules) In-Reply-To: References: <200502260012.j1Q0BhjP015155@squeakland.org> Message-ID: Dan, thanks for you examples and comments on my comments. Here are something things I thought while reading your "Getting Ready" message: > Think about a little PDA with ROM and RAM.¾ Suppose we blew the > squeak image in to the ROM.¾ Could we run with it in ROM? >¾When we wanted to save, could we only save the differences > from the ROM?¾ If we started it up again, and made some more > changes, could we just save those changes? What do these > incremental snapshots suggest about modules? When you used Compact Flash cards with the SqueakBox, you probably let the underlying Linux file system deal with the nasty details. And they are nasty: you can only erase whole blocks (typically 64KB) at a time and any given word can only be reprogrammed a certain number of times (10,000 or so). So you have to detail with parts slowly going bad and/or try to spread out writes as much as you can. Writing a new image, erasing the old one and then renaming the new to have the old name isn't a good way of dealing with Flash. You ping pong between two areas while leaving the rest of the memory untouched. Even worse: nearly all of the content that has just been written is identical to what was previously erased to make room for it. Just dividing the system into a bulk read-only part and a changes part would be a big help. The changes could grow and grow until there was not enough room left and then a new read-only part could be created compressing all the changes (with a new, empty changes part). This would reduce the write rate to individual words by a factor of ten or more. Splitting this further into modules would be even nicer, but you would have to take into account the the typical module would be smaller than the Flash block. > Now imagine that the VM can have multiple method caches, one per, > um, per process?¾ per module?¾ Think about it.¾ And when you have > figured it out, note that this system could run simultaneous multiple > projects with conflicting changes to root classes with no performance > degradation. One cache per PIE viewpoint would be great. A single process might want to switch viewpoints once in a while. -- Jecel P.S.: should I add something about Neo Smalltalk "modules" to the swiki page? All the other systems there are real and this is just a paper design, so it didn't seem proper to me. From jecel at merlintec.com Wed Mar 2 14:22:25 2005 From: jecel at merlintec.com (Jecel Assumpcao Jr) Date: Wed Jul 19 10:23:23 2006 Subject: secure inter-module references Message-ID: This is a bit off topic, but I thought it would be interesting to think about what kinds of issues can come up when dealing with security. For example, can you have unforgeable inter-module references if the system is open source and any given node might be running a version with arbitrary modifications? What if access to the actual bytes in a module is relatively easy to obtain? I think what follows is a correct solution to this problem, but not something we would ever want to actually do given the high overhead. First we somehow have a universal reference to the module itself, M. Then we associate a sufficiently large random number, K, with a given object within the module. An object in another module can refer to this object with the pair Cap := { M. K }. When we load M, it includes an object table (in-pointers table). We generate a tag Tag := K oneWayFunction. and then use K with a simple hash to index into the table looking for an entry with Tag. Here there are many interesting variations that could be done, but which don't really matter for this example. i := objTable probeFor: Tag startingAt: K simpleHash. Once we have the correct entry, it will also give us the first byte position and object length so we can fetch the content of the object. That has been encrypted with K, so we reverse the process and we are done. start := (objTable at: i) start. obj := (module from: start to: start + (objectTable at: i) length) decryptWith: K. So if you have the module itself you can see how many objects it has and what their sizes are but without a valid Cap you won't be able to read any of the objects. And you won't be able to guess K by looking inside the module - someone else will have to give it to you. Once you are able to read an object, of course, it might itself include remote references and so you will have one or more valid Caps that you can use to get even more objects. A nice property of this scheme is that a remote reference is not affected if we add or remove objects from M. But such details aren't important - I am just trying to see in general what the solutions we will need might look like. -- Jecel From ned at squeakland.org Wed Mar 2 23:40:03 2005 From: ned at squeakland.org (Ned Konz) Date: Wed Jul 19 10:23:23 2006 Subject: A bunch of observations In-Reply-To: References: Message-ID: <200503021540.03868.ned@squeakland.org> On Wednesday 02 March 2005 3:37 am, goran.krampe@bluefish.se wrote: > ????????- Colin supports Alan Lovejoys definition of "separately deployable > code" (well, not "objects" perhaps). I do too. We should "choose" what a > "module" is, and that is IMHO the overshadowingly most important > attribute of it. Though "objects" is a better word than "code". Yes. I think an important but perhaps overlooked point here is that we seem to be making the distinction already between 'code' and other objects. Perhaps this is one of the distinctions between a "distributable package" and a "deployed module". Currently, of course, when we load a package we compile source code into the image. So we could look at the source code that's in a distributed package as either: (1) a kind of inefficient serialization format (with the side-effect of holding comments) for certain kinds of objects like: - Classes and metaclasses, either new or modified - CompiledMethod instances that get stuck in method dictionaries somewhere - chunks of loose code that get executed on-the-fly or saved somewhere as blocks (we could think of these as objects that get created and saved for later execution, even though "later" is often pretty soon after their creation. or (2) a special kind of non-object thing that gets treated different from "real" objects. I think it's important to remember that deploying modules more generally means managing objects, *not* particularly compiling code. We know that we generally have other kinds of objects to deal with anyway when we're deploying modules (look at ImageSegments or Projects, or some of the preamble/postscript magic in change sets that could otherwise be done by holding pre-built objects). I think that source code isn't (or shouldn't be) particularly important to the modules project, while it might be to the packages project. -- Ned Konz http://bike-nomad.com/squeak/ From lex at cc.gatech.edu Thu Mar 3 00:50:27 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation Message-ID: There has been some murmering about full isolation, both pro and against. The idea is that modules will be totally insulated from each other and thus unable to cause each other any harm. It's a nice idea but terrible if you take it to the extreme. Instead, a module system should *allow* modules to hold stuff private without interfering with other modules. On the other hand, modules systems should also provide convenient ways for modules to interact with each other. After all, the whole point of loading a module into your image is to enhance what the image is already capable of. Besides, full isolation is a royal pain if you really try it. As a few examples of what it would mean: 1. Every module must have its own thread. Otherwise, 2. Modules must interact by passing messages or events, and not by simple call and return. Otherwise, when you call, the other module can simply choose not to ever respond. 3. Classes need to be modified. In particular, you can't let people send #class to get a real class object, and then use #superclass and #subclasses to browse around the class hierarchy. You also probably don't want modules calling #compiledMethodAt:put:. I am not sure this extremity is a good idea for a module system even if we had loads of time to implement it. If you start with full isolation, then it requires extra work to get modules talking to each other again -- and I am not sure that the benefits of isolation are always worth this extra effort. Regardless, it's not something that is achievable in 6 months, and also, it's something that will break most existing Squeak code. Thus, we all need to deeply accept that, when a module is loaded, it may well cause some damage to the image it is loaded into. Hopefully such breakage is unusual, but it cannot be avoided entirely. Instead, focus on trying to prevent common accidents, and to allow common desired interactions. Two modules accidentally defining a "Server" class is a common kind of accident, as are two modules trying to add a ">>" method to class Behavior. What should happen, in such a situation? -Lex From mroos at roos.com Fri Mar 4 06:18:35 2005 From: mroos at roos.com (Mark Roos) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: Message-ID: Your comments are similar to the issues that have led me to two use cases for "modules" in our environment. One of the cases, which I refer to as "Application", benefits from isolation from other modules. An application does not want to be affected by the loading, unloading or modification of other modules. It also has less need for the tight coupling provided by being in the same process space. The second use is for tightly coupled modules, which I will refer to as libraries. Libraries are used by developers to create applications. They benefit from the tight coupling, but suffer some of the issues you mentioned like unexpected uverlaps, modifications and load sequence dependent behavior. It would be nice to have a system which maintained the tight coupling while protecting the programmer from the problems the tight coupling allows. I would like to see an approach that supported both use cases. mark "Lex Spoon" 03/02/05 04:50 PM To modules@discuss.squeakfoundation.org cc Subject full isolation There has been some murmering about full isolation, both pro and against. The idea is that modules will be totally insulated from each other and thus unable to cause each other any harm. It's a nice idea but terrible if you take it to the extreme. Instead, a module system should *allow* modules to hold stuff private without interfering with other modules. On the other hand, modules systems should also provide convenient ways for modules to interact with each other. After all, the whole point of loading a module into your image is to enhance what the image is already capable of. Besides, full isolation is a royal pain if you really try it. As a few examples of what it would mean: 1. Every module must have its own thread. Otherwise, 2. Modules must interact by passing messages or events, and not by simple call and return. Otherwise, when you call, the other module can simply choose not to ever respond. 3. Classes need to be modified. In particular, you can't let people send #class to get a real class object, and then use #superclass and #subclasses to browse around the class hierarchy. You also probably don't want modules calling #compiledMethodAt:put:. I am not sure this extremity is a good idea for a module system even if we had loads of time to implement it. If you start with full isolation, then it requires extra work to get modules talking to each other again -- and I am not sure that the benefits of isolation are always worth this extra effort. Regardless, it's not something that is achievable in 6 months, and also, it's something that will break most existing Squeak code. Thus, we all need to deeply accept that, when a module is loaded, it may well cause some damage to the image it is loaded into. Hopefully such breakage is unusual, but it cannot be avoided entirely. Instead, focus on trying to prevent common accidents, and to allow common desired interactions. Two modules accidentally defining a "Server" class is a common kind of accident, as are two modules trying to add a ">>" method to class Behavior. What should happen, in such a situation? -Lex -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.squeakfoundation.org/pipermail/modules/attachments/20050303/8c4011ac/attachment.htm From craig at netjam.org Fri Mar 4 07:14:26 2005 From: craig at netjam.org (Craig Latta) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation Message-ID: Hi all-- If you've got both small core object memories and remote message-sending, why not just put things that need to be isolated in separate object memories? Besides being sufficient (?) and simpler than trying to cram everything into a single object memory, you get distribution across multiple machines "for free". I have to admit that, so far, I'm not a fan of "isolation" within a single object memory, or using image segments for anything not "below the line" in virtual-machine-land. Jecel's virtual memory suggestion seems fine, but throwing image segments around seems like an overcomplication to me. Why not keep it all live objects sending messages? -C -- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth! From goran.krampe at bluefish.se Fri Mar 4 10:12:13 2005 From: goran.krampe at bluefish.se (goran.krampe@bluefish.se) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: References: Message-ID: Hi guys! "Lex Spoon" wrote: > There has been some murmering about full isolation, both pro and > against. The idea is that modules will be totally insulated from each > other and thus unable to cause each other any harm. [SNIP of rest] I have been writing an "article" or "brainstorm" or whatever the last few days, I will make it available soon but need to think it through a little bit more. Anyway, it roughly describes an approach with these main points: - We separate between "loaded" and "activated" modules. This IMHO was a thing that Modules3.3 got Right. A module should ALWAYS be loadable (getting it into the image) but when it is only loaded it should just "be there" for tools to work at. It should not affect anything. - We use ImageSegments for modules. I have stepped through the ImageSegment code a bit to refresh my memory and while some of it can probably be cleaned up I think it is a really nice way to go with Modules, at least as the primary mechanism - nothing preventing other means of loading/constructing a Module of course. This would implicitly define a Module to contain objects - not source code. My sketch simply means we rip out a bunch of classes/globals in an ImageSegment with some extra info with it. So they are already compiled etc, just need to "hook them in". Kinda like a ".dll" for Squeak. :) - When activating a module it is "hooked up" using various mechanisms. This is when it will affect the image, and just as Lex says - it can't be totally isolated per definition. :) And to still hold it with a single reference when activated would mean making a lot of references that are pretty fundamental today to be dynamic lookups. I am leaning again to use something similar to Modules3.3 here - a batch become operation to hook things up. So an *activated* module would *not* only be held in a single reference - but a deactivated one or just a loaded one would. So if the Module/system keeps enough info around it should be able to deactivate too - thus disconnecting the objects in the Module from the rest of the image in a controlled way (think superclass, subclass references yaddayadda). Well, enough now - I need to finish the writeup. :) And no - this is not a "proposal" - this is "loud thinking". The writeup includes some explanations on how things work today - so even if you all guys think I am hallucinating - it should be nice to read anyway. regards, Göran From ducasse at iam.unibe.ch Fri Mar 4 10:48:43 2005 From: ducasse at iam.unibe.ch (=?ISO-8859-1?Q?st=E9phane_ducasse?=) Date: Wed Jul 19 10:23:23 2006 Subject: Modules In-Reply-To: <4228055B.5020700@gmail.com> References: <69c7042a05022222472b6a285c@mail.gmail.com> <8294600f17feddd620e5dc6ea66472a3@wiresong.ca> <42201AAD.1090108@gmail.com> <197c936e58501532247798b1fd6a5a8d@wiresong.ca> <42214A8B.5080102@gmail.com> <47f80989764dc64c674b5cbe036aae21@wiresong.ca> <42228E3D.1010200@gmail.com> <303f21fb222281179ebdaf5c823666dd@wiresong.ca> <4222A96D.6040008@gmail.com> <4ff2f8a1fe99f8114ab0a4793a348188@iam.unibe.ch> <4228055B.5020700@gmail.com> Message-ID: hi florin > Hi st?phane > > I have just read the papers on classboxes. > > I think they are an interesting and useful concept. I don't think they > represent all that we would want from a module system, but I will try > to present my view for where they would fit in. > > I too believe there is a strong case to be made for being able to > limit the capabilities of modules, especially untrusted ones (either > because of untrusted origin or because of the experimental nature of > their code). > In general though, as a way of organizing code, many modules that > would be dsitinct because of a logical separation/classification do > not need to be limited/isolated. Let's say we put collections in a > separate module. They will depend on some core module, but they will > enjoy the same level of trust, and if for example they add an > extension method to Object, there is no need to limit the visibility > of that method to the collections module. But this is an artificial > example. Let's take my most common use of "overrides": a separately > loadable module that contains bug fixes to the base image. Of course I > want those fixes visible to everything else in the image. True I always thought that we would like to have the two semantics > > On the other hand, if we let foreign Croquet code run in our image, we > would most likely want to limit its capabilities. > > With the above in mind, I think that it would make sense to assign > levels of trust to modules and to link the visibility of the changes > to the level of trust: if a module has a level of trust equal or > greater than that of the modified (or imported, in your terminology) > modules, its changes should be visible to the imported modules as > well. If not, its bindings should be local. Interesting idea.;) > > Obvioulsy this does not completely solve the security issue, but I > think it's one of the necessary steps. > > Florin > > From fmateoc at gmail.com Fri Mar 4 06:51:07 2005 From: fmateoc at gmail.com (Florin Mateoc) Date: Wed Jul 19 10:23:23 2006 Subject: Modules In-Reply-To: <4ff2f8a1fe99f8114ab0a4793a348188@iam.unibe.ch> References: <69c7042a05022222472b6a285c@mail.gmail.com> <8294600f17feddd620e5dc6ea66472a3@wiresong.ca> <42201AAD.1090108@gmail.com> <197c936e58501532247798b1fd6a5a8d@wiresong.ca> <42214A8B.5080102@gmail.com> <47f80989764dc64c674b5cbe036aae21@wiresong.ca> <42228E3D.1010200@gmail.com> <303f21fb222281179ebdaf5c823666dd@wiresong.ca> <4222A96D.6040008@gmail.com> <4ff2f8a1fe99f8114ab0a4793a348188@iam.unibe.ch> Message-ID: <4228055B.5020700@gmail.com> st?phane ducasse wrote: > Hi florin > > I really like the quality of your discussion with colin. Thanks for that. > >> Here, I am not so sure. >> As I see it, there are two reasons for having overrides. One is to >> offer an automatic resolution for accidental collisions, and as a way >> to guarantee that your package _can_ be loaded (without manual >> modifications) in its intended state, mentioned above. >> The other one, that I mentioned in my reply to Dan's message, is for >> intentional overrides of something that is known to exist and to be >> used in the pre-existing image (either because it is part of the base >> image, or because it is part of another, required package). Sometimes >> you do need hooks in other packages or in the base image, to attach >> yourself to a pre-existing state, and they are not general enough to >> justify a "fix" in the base image. Since they are specific to your >> package, they should belong as organizational structure to your >> package as well, just like normal (non-conflicting) extensions would. >> And it's not just about methods. As an example, your module wants to >> add some state to processes, so it needs an additional instvar in >> class Process. Shouldn't this change to a pre-existing class >> definition be contained in your module, be loaded with it, and be >> unloaded when the module is unloaded? This is clearly not an >> accidental collision, but it does not make sense by itself in the >> module originally defining class Process. > > > I agree with you. > In Classboxes (again I'm not saying that this is the solution) we took > the idea that overrides, extensions, state extensions > were local to the package doing them. I would really appreciate if you > could comment on our papers (on the list or privately). I'm not sure > that always having the semantics we gave to extensions is the one we > want but at least we have a consistent world. > > Stef > Hi st?phane I have just read the papers on classboxes. I think they are an interesting and useful concept. I don't think they represent all that we would want from a module system, but I will try to present my view for where they would fit in. I too believe there is a strong case to be made for being able to limit the capabilities of modules, especially untrusted ones (either because of untrusted origin or because of the experimental nature of their code). In general though, as a way of organizing code, many modules that would be dsitinct because of a logical separation/classification do not need to be limited/isolated. Let's say we put collections in a separate module. They will depend on some core module, but they will enjoy the same level of trust, and if for example they add an extension method to Object, there is no need to limit the visibility of that method to the collections module. But this is an artificial example. Let's take my most common use of "overrides": a separately loadable module that contains bug fixes to the base image. Of course I want those fixes visible to everything else in the image. On the other hand, if we let foreign Croquet code run in our image, we would most likely want to limit its capabilities. With the above in mind, I think that it would make sense to assign levels of trust to modules and to link the visibility of the changes to the level of trust: if a module has a level of trust equal or greater than that of the modified (or imported, in your terminology) modules, its changes should be visible to the imported modules as well. If not, its bindings should be local. Obvioulsy this does not completely solve the security issue, but I think it's one of the necessary steps. Florin From ned at squeakland.org Fri Mar 4 13:31:55 2005 From: ned at squeakland.org (Ned Konz) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: References: Message-ID: <200503040531.55987.ned@squeakland.org> On Friday 04 March 2005 2:12 am, goran.krampe@bluefish.se wrote: > - We use ImageSegments for modules. I have stepped through the > ImageSegment code a bit to refresh my memory and while some of it can > probably be cleaned up I think it is a really nice way to go with > Modules, at least as the primary mechanism - nothing preventing other > means of loading/constructing a Module of course. This would implicitly > define a Module to contain objects - not source code. My sketch simply > means we rip out a bunch of classes/globals in an ImageSegment with some > extra info with it. So they are already compiled etc, just need to "hook > them in". Kinda like a ".dll" for Squeak. :) Note that loading an ImageSegment back into the image it came from, or loading it twice, will result in multiple objects with the same object identityHash values. Not that there aren't collisions ordinarily, of course, but the fact that these collisions are on objects of the same type as the originals could cause some performance issues. Just something to think about. -- Ned Konz http://bike-nomad.com/squeak/ From jecel at merlintec.com Fri Mar 4 18:15:38 2005 From: jecel at merlintec.com (Jecel Assumpcao Jr) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: References: Message-ID: Craig Latta wrote on Fri, 04 Mar 2005 02:14:26 -0500 > If you've got both small core object memories and remote > message-sending, why not just put things that need to be isolated in > separate object memories? Besides being sufficient (?) and simpler than > trying to cram everything into a single object memory, you get > distribution across multiple machines "for free". Imagine that an acyclic directed graph describes the dependency between the modules (we should be so lucky!). You want a node to be isolated from some other random node, and also from everything that depends on it. But you don't want to isolate the node from the stuff on which it depends. A solution is to put each node plus a copy of everything on which it depends into a separate memory, though this only works as long at everything but the node itself is read-only. Another solution is to put each node into a separate memory and use remote messages exclusively for a given node to interact with the stuff it depends on (these are objects, after all, so we can do this). In this case writable shared stuff is allowed but performance might not be good. A third solution is like the first, but the copies are kept in sync. This would be TeaTime, right? > I have to admit that, so far, I'm not a fan of "isolation" within a > single object memory, or using image segments for anything not "below > the line" in virtual-machine-land. Jecel's virtual memory suggestion > seems fine, but throwing image segments around seems like an > overcomplication to me. Why not keep it all live objects sending messages? I would very much like to do it like that (there is an interesting Croquet thread about this) but one requirement I have is to be able to deal with people who are connected to the world via CD-ROMs sent by snail-mail. We can still think of the included files as messages in some protocol over a channel, but without a reasonable back-channel there can be no negotiations. If I could drop this requirement then things would be nicer. -- Jecel From lex at cc.gatech.edu Sat Mar 5 17:37:14 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: misc (was Re: full isolation) In-Reply-To: References: Message-ID: It will be great to hear the long version, Goran! > - We separate between "loaded" and "activated" modules. This IMHO was a > thing that Modules3.3 got Right. A module should ALWAYS be loadable > (getting it into the image) but when it is only loaded it should just > "be there" for tools to work at. It should not affect anything. Yes, that would be excellent. It is a little awkward to use a separate tool for browsing external code, like we do right now. Additionally, it seems like this will make the code browsers more modular and easy to work with. Instead of trawling through the system, the browser could ask questions to some sort of SourceCodePool object. (Chuck's name for this concept -- but feel free to improve on it!). I believe the refactoring browser has a similar concept floating around, named BrowserEnvironment perhaps? It's a little mind-bending to think about the details of this. > - We use ImageSegments for modules. ImageSegments are a fantastic piece of technology. Please see my next message, though, about potentially focussing too much on fully general objects. > - When activating a module it is "hooked up" using various mechanisms. > This is when it will affect the image, and just as Lex says - it can't > be totally isolated per definition. :) And to still hold it with a > single reference when activated would mean making a lot of references > that are pretty fundamental today to be dynamic lookups. Hmm, yes that seems true. The links can get reconnected as modules are loaded and unloaded. Semantically, that's a dynamic lookup. ImageSegments can, if I understand correctly, deal with both multiple roots and multiple out-pointers from a segment. So, yes, there are excellent bragging rights available if the Squeak module system can hang on to a module with just one reference! However, it is possible to muddle along with a merely fabulous module system that has 10 or even 100 references hanging on to it. There is some wiggle room here. -Lex From lex at cc.gatech.edu Sat Mar 5 17:58:20 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: focus on code? Message-ID: There has been some discussion about how awkward it currently is to ship around objects in packages, and how nice it would be if Squeak's module system could support including a fully general object. If this group figures out a way to do that, it would indeed be wonderful. Even so, there is nothing wrong with concentrating mostly on code. Such a focus does not prevent there being a hook in the module system to add general objects to a module. In fact, adding a hook to support general objects may turn out to be easy no matter what the rest of the module system looks like. I mention this, because code is a common case, and there is a lot that can be done for it to improve how development works in Squeak. How does a statement in one module, refer to classes (or method names, Dan?!) in a different module? How does one have a single method in a module, and then arrange for it to get attached to the correct class when it is loaded? In what circumstances can a method in one module override a method in another? These are all questions that are specifically about code, but they are well worth attacking. So how do you do all these things? I don't know! It's great that there are Squeak folks who are good at these things, and I thoroughly look forward to seeing what everyone comes up with. Please don't be too reticent about ideas that "only" make sense for code. A code-centric module system that is effortless for developers to use, is something to be proud of. -Lex From ned at squeakland.org Sat Mar 5 20:22:09 2005 From: ned at squeakland.org (Ned Konz) Date: Wed Jul 19 10:23:23 2006 Subject: misc (was Re: full isolation) In-Reply-To: <200503051942.j25JgJjP006526@squeakland.org> References: <200503051942.j25JgJjP006526@squeakland.org> Message-ID: <200503051222.10148.ned@squeakland.org> On Saturday 05 March 2005 9:37 am, Lex Spoon wrote: > Yes, that would be excellent. ?It is a little awkward to use a separate > tool for browsing external code, like we do right now. ?Additionally, it > seems like this will make the code browsers more modular and easy to > work with. ?Instead of trawling through the system, the browser could > ask questions to some sort of SourceCodePool object. ?(Chuck's name for > this concept -- but feel free to improve on it!). ?I believe the > refactoring browser has a similar concept floating around, named > BrowserEnvironment perhaps? Yes; there are various BrowserEnvironment subclasses that can enumerate their contents or answer inclusion tests (which is exactly what PackageInfo instances can do). They also provide convenient ways to get to the SystemNavigation, SystemOrganization and ClassOrganization services. And they can hold a set of search strings, which is nice in the browser. There is one flavor (ParseTreeEnvironment) which can be given patterns to match the parse tree against. So for instance you can have a BrowserEnvironment that includes only the methods that call menu construction operations with untranslated literal strings (a Lint rule that I just added). These are among the ways you can create them (note that you can do logical set operations on them too): & forCategories: forClass:protocols: forClass:selectors: forClasses: forPackage: forPackageContainingClassCategory: implementorsMatching: implementorsOf: instVarReadersTo:in: instVarRefsTo:in: instVarWritersTo:in: matches: not referencesTo: referencesTo:in: selectMethods: | There are also representations of instantiated and non-instantiated classes: RBAbstractClass #('name' 'newMethods' 'instanceVariableNames' 'model' 'superclass' 'subclasses' 'removedMethods' 'realClass') RBClass #('classVariableNames' 'poolDictionaryNames' 'category') RBMetaclass #() And RBMethod #('class' 'compiledMethod' 'source' 'selector') And there's RBNamespace: RBNamespace #('changes' 'environment' 'newClasses' 'removedClasses' 'changedClasses' 'rootClasses' 'implementorsCache' 'sendersCache') which gives a way to deal with multiple changes across a group of classes and methods. Its interface is more or less a duplicate of SystemDictionary, but it represents the future state of the contents of a BrowserEnvironment. At some point you can choose to go: someRBNamespace changes execute. and actually commit the changes all at once. There are also concrete representations of changes to the model (which provide execute and undo operations). These let you preview a set of changes. Actually, what I'm doing right now (since we already had the RefactoryChange #('name') CompositeRefactoryChange #('changes') RenameClassChange #('oldName' 'newName') RenameVariableChange #('className' 'isMeta' 'oldName' 'newName') RenameClassVariableChange #() RenameInstanceVariableChange #() RefactoryClassChange #('className' 'isMeta') AddClassChange #('definition' 'superclassName' 'instanceVariableNames' 'classVariableNames' 'poolDictionaryNames' 'category') InteractiveAddClassChange #('controller' 'definedClass') AddMethodChange #('source' 'selector' 'protocols') InteractiveAddMethodChange #('controller' 'definedSelector') RefactoryVariableChange #('variable') AddClassVariableChange #() AddInstanceVariableChange #() AddPoolVariableChange #() RemoveClassVariableChange #() RemoveInstanceVariableChange #() RemovePoolVariableChange #() RemoveClassChange #() RemoveMethodChange #('selector') I'd suggest that we consider using the RB model classes as a base for a future reflection and editing framework. They give a very clean interface to the system. I should also note that the 'Monticello-Modeling' and 'Monticello-Patching' class could be mostly replaced by (or at least derived from) the 'Refactory-Model' and 'Refactory-Change Objects' classes. -- Ned Konz http://bike-nomad.com/squeak/ From fmateoc at gmail.com Sat Mar 5 22:19:26 2005 From: fmateoc at gmail.com (Florin Mateoc) Date: Wed Jul 19 10:23:23 2006 Subject: focus on code? Message-ID: <422A306E.7080405@yahoo.com> I agree very much with Lex. Especially since, as is obvious from his message, he is not so much talking about source code, but about its in-image compiled representation (and its holders). In this interpretation, code (classes, with their definitions and methods) is a major invariant in a Smalltalk image, it is what makes the quasi-permanent part of the object memory, and for the most part, it constitutes the roots for holding everything else. The only other significant kind of root is non-class globals, which are indeed also worth focusing on, but which also have (or if they don't, they should :)), for the most part, a code definition ((re)initialization) somewhere. So it's not just that code is a common case. It is also the most likely candidate for being _the_ non-garbage collectable reference to all these other generic objects that we might want to include in a module. I would agree though that code-centered entities like Envy applications, or, closer to what we seem to want (as they are binary, independently deployable units), VW parcels, err by not paying enough attention to the non-code aspects (globals and other pre-initialized objects). We do need a solution (and re-initialization from code is not good enough in the general case) for loading/unloading globals. I have not used ImageSegments, but I assume they would fill that gap nicely. Florin From Dan at SqueakLand.org Sun Mar 6 00:13:03 2005 From: Dan at SqueakLand.org (Dan Ingalls) Date: Wed Jul 19 10:23:23 2006 Subject: Administrivia Message-ID: A couple of folks are participating on this list without being subscribed. I have no problem with this, but I want you to know that the turnaround can be slow because your message doesn't not get posted to the list until I (as moderator) have been asked to approve it by the mail bot. Sometimes I do this right away, but sometimes I'm away doing other things, so there's no guarantee of decent response. FYI - Dan From craig at netjam.org Sun Mar 6 06:21:17 2005 From: craig at netjam.org (Craig Latta) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: References: Message-ID: <422AA15D.90909@netjam.org> Hi Jecel-- > You want a node to be isolated from some other random node, and also > from everything that depends on it. But you don't want to isolate the > node from the stuff on which it depends. > > A solution is to put each node plus a copy of everything on which it > depends into a separate memory, though this only works as long at > everything but the node itself is read-only... [another] solution is > like the first, but the copies are kept in sync. This would be > TeaTime, right? Similar, yes. > > Why not keep it all live objects sending messages? > > I would very much like to do it like that (there is an interesting > Croquet thread about this) but one requirement I have is to be able to > deal with people who are connected to the world via CD-ROMs sent by > snail-mail... without a reasonable back-channel there can > be no negotiations. You could populate the CD-ROMs with object memory snapshots that, when resumed, start servers on 'localhost'. Message-based negotiation still happens, it's just that the clients and servers are all on the same machine. The CD-ROM object memories could have all the modules you want to convey in them. thanks, -C -- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth! From craig at netjam.org Sun Mar 6 06:51:00 2005 From: craig at netjam.org (Craig Latta) Date: Wed Jul 19 10:23:23 2006 Subject: focus on code? In-Reply-To: References: Message-ID: <422AA854.4040709@netjam.org> Hi Lex-- > There has been some discussion about how awkward it currently is to > ship around objects in packages, and how nice it would be if Squeak's > module system could support including a fully general object. If this > group figures out a way to do that, it would indeed be wonderful. With the distributed system I wrote, one can send any message with any parameters to any object in any other object memory, as if it were a normal send. I think this is a superset of the goal mentioned above. (And if you're limited to, say, CD-ROMs, then just ship entire "provider" object memories, since they start out minimal and only have what you want in them, and let negotiation take place all on one machine.) -C -- Craig Latta improvisational musical informaticist craig@netjam.org www.netjam.org [|] Proceed for Truth! From Dan at SqueakLand.org Sun Mar 6 22:39:37 2005 From: Dan at SqueakLand.org (Dan Ingalls) Date: Wed Jul 19 10:23:23 2006 Subject: Splitting Tools In-Reply-To: References: Message-ID: [I am cross-posting this to the Packages list, since it seems that similar work is going on there as well] >On Tue, 1 Mar 2005 19:28:09 -0800, Dan Ingalls wrote: > >>Does such a tool exist right now? If not, I think it would be a great aid to the project of partitioning the image. > >It certainly would. When I worked with VW last year, we had such a tool which greatly helped in determining prerequisites. First it would tell you what packages where needed, but it had a detail view that would tell you why these were needed. > >With that, we could in theory point at ToolBuilder and some 'favorite morphs' and start hacking away. Folks - Sometimes I worry that a lot of work can get lost with the passage of people and time. I went back and tracked down my old code and messages to the Squeak list. It is from February 2001 (!!) and I attach the summary below. It turns out that most of the code is still in the image. The principle was extremely simple, and very few lines of code. Here's an example of its use: Start with a set of "package roots", such as {Celeste. Scamper. MailMessage}. First establish the list of classes and methods that appear to be unused in the full system Then compute the same result but in the absence of the package roots. The *newly* inaccessible classes and methods presumably are only used by package roots, so they should be added to the package. Iterate the previous two steps until the package does not grow any more. The comments with update #3735 below give the excellent results achieved in deriving a package for the three network applications of the time. This code all worked four years ago, and it was the basis for the preliminary partitioning in the Environments project. I mainly wanted to point it out, in case it might be of use to people in partitioning the current image. It may be entirely subsumed by Daniel V's MudPie but, if not, then it might save people some time. Hope this is of some use. - Dan ----------------------------------------- 3735BetterShrink-di -- Dan Ingalls -- 27 February 2001 Introduces a new image partitioning tool that is essentially a local version of majorShrink. The idea is to pick some application like Scamper, or some cluster like all the 3D classes, and remove it, and also remove everything that is solely referred to by it. The way it works is: Record all unsent messages and unused classes at the outset Mark the application or cluster as removed Note all *newly* unreferenced methods and unused classes and iteratively remove them as well For example, the expression Smalltalk reportClassAndMethodRemovalsFor: #(Celeste Scamper MailMessage) reports 63 classes and 155 other messages that can be removed. Also, introduces a new method, fileOutAndRemove:... that will take such results, build a changeSet from them, fileOut everything that is about to be removed, and then remove all of the classes and related messages as well. For the above example, this method produces a 290k fileOut, and saves about 104k from the system. More importantly, if you fileIn the changeSet afterward, the system returns to approximately its former size, and you can run Scamper again. It's a way to fileOut a package that never existed. Tools for Pools yet to come. From goran.krampe at bluefish.se Mon Mar 7 14:01:44 2005 From: goran.krampe at bluefish.se (goran.krampe@bluefish.se) Date: Wed Jul 19 10:23:23 2006 Subject: Brainstorm Message-ID: Ok, here is my over-and-over-revised-brainstorm I mentioned. Yes, it is long but I think it has some thoughts and concepts worth thinking about. And remember, this is brainstorming and I have edited it so much there may be inconsistencies. :) regards, Göran ---------------- A brainstormed model based on: - Dan's idea of a module being held in a single reference. - Some memories from Modules3.3 - My Namespace implementation Sitting on the bus going home (funny, I think Namespaces was born that way too) I was wondering a bit on how this idea of holding-a-module-in-a-single-place could work. This is just an attempt at playing with that thought - don't consider it a proposal. I also mix the concepts a bit, but humour me. :) Logical reasoning 1. A module is a separately deployable unit of objects, typically including classes. I think this definition is a good one. 2. Given the above a module is maintained by one or several developers as a *unit*, then all names (classes typically) it defines can be considered to all be in the same namespace. In other words there is no need nor danger of getting a name conflict within the module itself. So a module could be a namespace on its own, or in other words - there is no need to have a module defining names in *multiple* namespaces. At least I have a hard time seeing that need, sure, some scenario could probably be cobbled up - but hey, seriously? (interested to hear about that) 3. But can a namespace be populated by more than one module then? For simplicity, let's say it can't. :) Then we have a one-to-one relation between namespace and module, but for reasons to unfold I am not suggesting an IS A relation just yet, because I think the *lifecycle* of these two beasts are different. :) 4. If we can only hold the module in a single reference, this means all external references to classes (or other "globals" in the module) must be late bound (looked up during execution), we can't bind them at compile time. Practical? I don't think so. For example, what about class A in module Z that subclasses class B in module W? Anyway, further down I am proposing something "in between". 5. If we get away with holding a module in a single reference, then using ImageSegment for a module should work just fine, which would be slick indeed. Loading ImageSegments is brutally fast and saving them is pretty nippy too. Let's say we have a global called LoadedModules. As in Modules3.3, we take a two-step approach, first you load a module into the image, and that should always work no matter what! That little fact opens up much better tool support btw. Then you "activate" it as a second step which can be more tricky. Modules3.3 boiled down to performing a batch become operation IIRC (quite slick if you ask me) and that may very well be the route to go, I think I will dust off Henrik's code to have a look. Imagine that the load is simply an ImageSegment load, so it already contains fully compiled classes and other objects. LoadedModules is a Dictionary. A Module has an instvar originalName (I think it should), but when we load the module we hang it in LoadedModules under a name we choose (inspired a bit by Craig that names shouldn't really be "in" the object), typically we choose the originalName, *if* that name isn't already taken. So *loading* a Module will *always work*, the only thing that can possibly happen is that the originalName is already taken - but then we just pick another. Now, a loaded Module is just hanging there - it doesn't do much more. :) Ideally we would now after the ImageSegment has been loaded be holding an instance of Module in a single reference. But... what about outpointers from the Module? Today all classes have an instvar called #environment and it points to the SystemDictionary instance. This is one of the outpointers when creating the ImageSegment. If you create a tiny ImageSegment of class "Baba" (and its methaclass) inheriting from Object - you get *at least* these outpointers: Object (the superclass of Baba), nil, #Baba, Smalltalk, Object class and Metaclass (superclass of Baba class, right?) And when we load the segment back in these things are typically hooked back up with the "correct" objects in the image. nil and #Baba is probably quite allright. :) But what about the other guys? We have 3 classes and Smalltalk. The environment instvar feels like we could have nil'ed before exporting, then when loading a Module we can set it to something suitable. This would mean that a loaded Module (ignoring Symbols and nil and other low level objects) would only have a direct reference to Smalltalk from inside, if we ignore the references to other classes. Say we have another global called Namespaces which is also a Dictionary. You activate a Module by plugging it "in a namespace", and you never remove it from LoadedModules. So loading a Module gives us one reference and activating it, yet another reference but we also hook into objects inside the Module in variois places. So an activated Module would have two references to itself and a number of references to objects contained inside it. But when deactivated those references are turned into something else (on both sides) and one handle on the Module (Namespaces) is removed. Then the module again is held only in LoadedModules, and to unload it you would simply do: LoadedModules removeKey: #MyModule. Smalltalk garbageCollect. ":)" The key in the Dictionary called Namespaces is the name we designated to the module as its "activated name". Again, typically the name used when loading it (which in turn typically is the originalName), but for various reasons we may choose another one. For example, if I reimplement the networking module I could call it "Network" (originalName) which is the same as the old Module, you could load it into your image, but you would have to call it NewNetwork if you have the old Network already in there. And then you can either decide to activate NewNetwork in parallell with Network, thus plugging the module in a new namespace called NewNetwork, or you could simply plug it in the existing Network namespace thus effectively replacing the old module with a different implementation. So given the one-to-one relation of Module<->Namespace we could have a Namespace instance having an instvar called #module but also hold the associations (bindings). But why this extra indirection? Why isn't Module and Namespace the same thing? Because we want to be able to create the Namespace instance and even populate it with bindings, without yet having the module hooked in there or even loaded! And the need for that is because the other activated Modules have references to objects in the not-yet-loaded-nor-activated Module. So a Namespace instance gets a similar role as Undeclared has. We can load and activate a Module that indeed has a class A that subclasses class B from Module X, without having loaded or activated module X! class A should then have a yet "unbound" reference to the superclass, meaning it should try to refer to a binding in Namespace X, and since there is none there - create it. Then later, when Module X is activated, we can plug in the correct value in that binding - class B. Let us try it. This means that Namespaces (a Dictionary) should in effect hold Namespace instances which in turn,. each one, refers to a Module or to a single instance we can call UndefinedModule (similar to UndefinedObject). This all is based on keeping the SystemDictionary Smalltalk as is, which IMHO is a good idea - I would like a new Modules approach to live side-by-side, just like Monticello does with the other tools - it makes it so much easier to introduce and thus we don't repeat the Modules3.3 debacle. Ok, how will textual references to the module and more importantly to its objects (classes) work? Most of us don't really know what a textual reference to a global ends up *being* when compiled. I am not even sure I am grokking it in full, but I *think* I do. :) Each global object today is hanging in an instance of Association (or a subclass thereof) which hangs in the SystemDictionary (Smalltalk). The association has the name (an instance of Symbol) in the instvar #key and the #value instvar refers to the object itself. When a textual reference to a global is compiled the CompiledMethod instance will end up with having in it a reference to this Assocation instance. This means CompiledMethods do not refer to the global object directly, but via the Assocation. PointerFinder pointersTo: AFreshNewSubclassOfObject ...will show you 4 hits. The Metaclass of the new class, the Array that holds subclasses in Object, the Association in Smalltalk and the ClassOrganizer which is held by the class itself. For each subclass you create for this new class you get one more hit - since the subclass refers to its superclass. But you do not get more hits when referencing your class in methods! I am not sure if these are all kinds of references we have to a class - but if it is then perhaps we can introduce a "dynamic binding" which is resolved fully when both Modules are activated, but is not resolved fully when only one Module (either side of the equation) is activated. A "well formed" Module would then not have any direct references to anything outside it when loaded, instead it contains instances of DynamicBinding, that in turn contains enough information to resolve themselves when activated. When we activate a Module we plug it into a Namespace instance, we take all bindings (defined names) in the Module and hang them into the Namespace instance - but if there are already one or more bindings there we simply put the value into them. For all DynamicBindings in the Module we resolve them but we also somehow need to be able to turn them back into DynamicBindings (?). A DynamicBinding typically has two Symbols - the namespace and the name to bind. So we just look up the namespace, and if there is a binding in there we use it, otherwise we create one. So how is such a DynamicBinding created then? Well, if we use a variant of my Namespace implementation then all globals in the source would be of the form "ANameSpace::AName". In practive the above scheme would mean that: Object "This is compiled and resolved just like before, it refers to an object in Smalltalk, which is not a Module. Remember - side by side." Network:: "This is a reference to a Namespace, and those are held in the global called Namespaces. So the Compiled looks in there to resolve it." Network::Socket "This is a reference to a named object in a namespace. When compiling this the Compiler first finds (or created on demand) the Namespace instance, then looks in there. If there is a binding we use it, otherwise we create one that is somehow marked -unbound-. (nil can not be used, because that is of course a valid value. :))" Summary We define a one-to-one relation between Module and Namespace (I think that is fair) but we do not unify the two concepts - because we want to be able to create Namespace instances "ahead" and then later plug in a Module inside it and let the Module populate it with bindings and reusing the unbound bindings that are already hanging there. So a Namespace acts almost like a ValueHolder for a Module. And the binding in there are ValueHolders for objects from the module. We differentiate between loading a Module and activating it in a Namespace. Given the above you should be able load and activate modules with classes having references to classes/objects in Modules that aren't even loaded yet. If we totally disregard the "class extension problem" this means Modules can be loaded and activated in *any order*. Well, that was my thoughts. :) From bergel at iam.unibe.ch Mon Mar 7 18:11:17 2005 From: bergel at iam.unibe.ch (Alexandre Bergel) Date: Wed Jul 19 10:23:23 2006 Subject: Getting Ready In-Reply-To: <3.0.6.32.20050301094831.01e0cb90@mail.frii.com> References: <3.0.6.32.20050301094831.01e0cb90@mail.frii.com> Message-ID: <20050307181117.GO7871@iam.unibe.ch> Hi David! Sorry to answer so lately to your email. I was away until today. I see that assigning one method cache per project/module/workspace/whatever is only as an implementation trick to implement a model. What is the model behind Sonora ? how do you handle method redefinition ? With classboxes, I have the following SUnit assertions: cx1 _ self createClassboxNamed: #CX1. cx2 _ self createClassboxNamed: #CX2. "---------------------------" cx1 import: #Object from: Classbox system. cx1 createClassNamed: #TestCXA. cx1 createClassNamed: #TestCXB superclass: #TestCXA. cx1 addMethod: 'foo ^ 1' for: #TestCXA. cx1 addMethod: 'bar ^ self foo' for: #TestCXB. "---------------------------" cx2 import: #TestCXA from: cx1. cx2 import: #TestCXB from: cx1. cx2 addMethod: 'foo ^ 2' for: #TestCXA. "---------------------------" self assert: (cx1 evaluate: 'TestCXB new foo') = 1. self assert: (cx1 evaluate: 'TestCXB new bar') = 1. "---------------------------" self assert: (cx2 evaluate: 'TestCXB new foo') = 2. self assert: (cx2 evaluate: 'TestCXB new bar') = 2. Other tests are available on http://kilana.unibe.ch/classboxes/tests/ and a tutorial containing screenshots is available on http://kilana.unibe.ch/classboxes/classboxtutorial/ Alexandre On Tue, Mar 01, 2005 at 09:48:31AM -0700, David Farber wrote: > > At 11:11 PM 2/28/2005 -0800, you wrote: > >Now imagine that the VM can have multiple method caches, one per, um, per process? per module? Think about it. And when you have figured it out, note that this system could run simultaneous multiple projects with conflicting changes to root classes with no performance degradation. > > Dan - This is exactly what I want to accomplish with Sonora. > > Imagine a Workspace that understands an extended syntax that lets you define methods and classes in addition to the normal code snippets. > > ----SonoraWorkspace > 'Hello!' asRot13. > > String>>asRot13 > ^self collect: [:ch | ch asRot13] > > Character>>asRot13 > ((self >= $a) & (self < $n)) | ((self >= $A) & (self < $N)) > ifTrue: [ ^ (self asInteger + 13) asCharacter ]. > ((self > $m) & (self <= $z)) | ((self > $M) & (self <= $Z)) > ifTrue: [ ^ (self asInteger - 13) asCharacter ]. > ^ self > > ---- > > If you select all of the code above and 'PrintIt' you should get 'Uryyb!'. The trick is, the two asRot13 methods should only be visible to messages and objects that originate from this invocation of the workspace. > > So far, only the code that understands the extended syntax has been written. To implement the "module" behavior I want, I was thinking about hacking in some ugly, behind-the-scenes, anonymous class manipulation. Obviously, if we can come up with a cleaner, more general approach, that would be better. > > david > > -- > David Farber > dfarber@numenor.com -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.iam.unibe.ch/~bergel ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. From bergel at iam.unibe.ch Mon Mar 7 18:44:41 2005 From: bergel at iam.unibe.ch (Alexandre Bergel) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: <20050304003525.A3F0425BAF2@mailhub01.unibe.ch> References: <20050304003525.A3F0425BAF2@mailhub01.unibe.ch> Message-ID: <20050307184441.GQ7871@iam.unibe.ch> I have been away these last days, that's why I was pretty silent... Would it be possible to set up this list in such a way that when one answers, the answer is sent to the list rather than privately. > There has been some murmering about full isolation, both pro and > against. The idea is that modules will be totally insulated from each > other and thus unable to cause each other any harm. One serious issue concern meta programming. I believe that achieving a descent isolation mechanism require a complete overhaul of the reflective facility. It seems that the Self mirror mechanism would provide an interesting solution... > 1. Every module must have its own thread. Otherwise, Why ? Classboxes do not impose such a constraint. > 2. Modules must interact by passing messages or events, and not by > simple call and return. Otherwise, when you call, the other module can > simply choose not to ever respond. ??? Can you provide a concrete example ? > 3. Classes need to be modified. In particular, you can't let people > send #class to get a real class object, and then use #superclass and > #subclasses to browse around the class hierarchy. You also probably > don't want modules calling #compiledMethodAt:put:. yep, metaprogramming and reflective features should be reshaped... > Instead, focus on trying to prevent common accidents, and to allow common > desired interactions. Two modules accidentally defining a "Server" class is > a common kind of accident, as are two modules trying to add a ">>" method to > class Behavior. What should happen, in such a situation? Look at the Selector Namespace and Classboxes... Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.iam.unibe.ch/~bergel ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. From ned at squeakland.org Mon Mar 7 21:36:28 2005 From: ned at squeakland.org (Ned Konz) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: <20050307184441.GQ7871@iam.unibe.ch> References: <20050304003525.A3F0425BAF2@mailhub01.unibe.ch> <20050307184441.GQ7871@iam.unibe.ch> Message-ID: <200503071336.28130.ned@squeakland.org> On Monday 07 March 2005 10:44 am, Alexandre Bergel wrote: > One serious issue concern meta programming. I believe that achieving a > descent isolation mechanism require a complete overhaul of the reflective > facility. It seems that the Self mirror mechanism would provide an > interesting solution... I agree with this, partly because of the necessity to have reflection on as-yet-unloaded modules. Taking the responsibility out of Class/Behavior and moving it into model classes that represent the classes (and other artifacts) would be considerably cleaner. I'd rather ask a mirror of (say) a package or module about its contents, etc. than have to go to a (global) SystemNavigation or whatever it's called today. -- Ned Konz http://bike-nomad.com/squeak/ From bergel at iam.unibe.ch Tue Mar 8 13:55:17 2005 From: bergel at iam.unibe.ch (Alexandre Bergel) Date: Wed Jul 19 10:23:23 2006 Subject: Brainstorm In-Reply-To: References: Message-ID: <20050308135517.GW7317@iam.unibe.ch> Just some quick comments: > 1. A module is a separately deployable unit of objects, typically > including classes. I think this definition is a good one. What a module is for? When one would choose to use it? Is it for deploying application? Is it to share source code? Is it to save/share EToys stuff and other scripting mechanism? What is the model behind the technical details that you gave? Me, as an end-user, I do not really have to care that ImageSegment is used or not, or that a module can be easily unloaded by putting a ref to nil. The system should do that for me. For me, it is really not clear why and for what we need modules. To share source code, I think that Monticello is not bad, regardless the lack of first class package that implies some naming convention. > and Metaclass (superclass of Baba class, right?) Not really. (Baba class) superclass == (Baba superclass) class Baba class class == Metaclass Cheers, Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.iam.unibe.ch/~bergel ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. From goran.krampe at bluefish.se Tue Mar 8 14:22:21 2005 From: goran.krampe at bluefish.se (goran.krampe@bluefish.se) Date: Wed Jul 19 10:23:23 2006 Subject: Brainstorm In-Reply-To: <20050308135517.GW7317@iam.unibe.ch> References: <20050308135517.GW7317@iam.unibe.ch> Message-ID: Hi! Alexandre Bergel wrote: > Just some quick comments: > > > 1. A module is a separately deployable unit of objects, typically > > including classes. I think this definition is a good one. > > What a module is for? When one would choose to use it? Is it for deploying application? Is it to share source code? Is it to save/share EToys stuff and other scripting mechanism? I just followed the proposed "definition" by Alan Lovejoy which a few other people also thought sounded good - incuding Colin Putney IIRC. The emphasis is on *deployable* and *objects*, meaning not only source and not typically partitions made for SCM (like Monticello etc) but for actual deployment. I still think it is a good start and the answer to your questions would be: - It is for deployment and distribution of applications and libraries. Since it is not constratined to only source code it could apply for eToys too I guess or any other "part of the image that you would like to distribute". - It is not primarily for SCM management, we have Monticello and various other tools that are more capable with merging mechanisms etc etc. A Module is in this aspect more "low level". My current belief is that a Module would simply be a "chunk of the image" with defined bindings in it (=named objects), and most often it would contain initialized class objects but not necessarily only so. > What is the model behind the technical details that you gave? Me, as an end-user, I do not really have to care that ImageSegment is used or not, or that a module can be easily unloaded by putting a ref to nil. The system should do that for me. You seem to miss that I posted a "brainstorm" - I am expecting you to storm with me, not just shooting it down :). And yes, the end user wouldn't care - but I am deliberately talking about implementation in order to make this much more tangible and concrete. In the end I want a clear cut easy to understand Module concept - because if it is not easy to understand then it will fail, mark my words. > For me, it is really not clear why and for what we need modules. I can see several reasons, but claiming it is "clear" to me, I do not. :) This Team is exploring the area, so let us explore. > To share source code, I think that Monticello is not bad, regardless the lack of first class package that implies some naming convention. I agree. > > and Metaclass (superclass of Baba class, right?) > > Not really. > (Baba class) superclass == (Baba superclass) class > > Baba class class == Metaclass Yes, I think that was a typo, I meant "class of Baba class, right?". > Cheers, > Alexandre cheers, Göran From jecel at merlintec.com Tue Mar 8 09:30:14 2005 From: jecel at merlintec.com (Jecel Assumpcao Jr) Date: Wed Jul 19 10:23:23 2006 Subject: focus on code? Message-ID: Lex Spoon wrote on Sat, 5 Mar 2005 13:58:20 -0400 > So how do you do all these things? I don't know! It's great that there > are Squeak folks who are good at these things, and I thoroughly look > forward to seeing what everyone comes up with. Please don't be too > reticent about ideas that "only" make sense for code. A code-centric > module system that is effortless for developers to use, is something to > be proud of. The code-only 3.3 module system had my full support. Any step forward is to be encouraged. But my programming style is such that I can't use this kind of system to distribute my stuff to others. I nearly always create instances as well as classes and methods and use various editors and tools to shape them. These are then stuck in the image in which they were born (though I can use projects to export them). This really hasn't been a problem since nobody else is really interested in an application to edit and upload a font to some hardware than only I have. Some folks would rather see a more declarative way of coding in Smalltalk. And they have some very good points. I want to play with live objects. But Squeak isn't for me - it is for all of us. -- Jecel From jecel at merlintec.com Tue Mar 8 09:17:11 2005 From: jecel at merlintec.com (Jecel Assumpcao Jr) Date: Wed Jul 19 10:23:23 2006 Subject: modules on CD (was: full isolation) In-Reply-To: <422AA15D.90909@netjam.org> References: <422AA15D.90909@netjam.org> Message-ID: Craig Latta wrote on Sat, 05 Mar 2005 22:21:17 -0800 > > [need to ship modules on a CD] > > You could populate the CD-ROMs with object memory snapshots that, when > resumed, start servers on 'localhost'. Message-based negotiation still > happens, it's just that the clients and servers are all on the same > machine. The CD-ROM object memories could have all the modules you want > to convey in them. I guess there are two ways this could be done: 1) for each module we would have a memory snapshot containing the module itself, any modules it depends on including the very basic stuff 2) a single memory snapshot containing all the modules we want to ship, any modules they depend on including the very basic stuff Option 1 wastes some of the CD-ROM, just like in Unix before dynamic linking each executable file had a copy of the basic libraries. Running each image separately doesn't take up much memory. Option 2, however, does take up quite a bit of (virtual) memory... if less than 650MB can be considered quite a bit in this era of 1GB RAMs. In any case, after message-based negotiation the local Squeak(s) get the modules they need and that will be saved to local disk automatically when the image is saved. This system is certainly usable and has many advantages, but takes up more RAM and disk than alternatives which allow images to be split into chunks which can then be lumped together again. -- Jecel From craig at netjam.org Wed Mar 9 00:27:00 2005 From: craig at netjam.org (Craig Latta) Date: Wed Jul 19 10:23:23 2006 Subject: modules on CD In-Reply-To: References: <422AA15D.90909@netjam.org> Message-ID: <422E42D4.10505@netjam.org> Hi Jecel-- > > > [need to ship modules on a CD] > > > > You could populate the CD-ROMs with object memory snapshots that, > > when resumed, start servers on 'localhost'. Message-based > > negotiation still happens, it's just that the clients and servers > > are all on the same machine. The CD-ROM object memories could have > > all the modules you want to convey in them. > > I guess there are two ways this could be done: > > 1) for each module we would have a memory snapshot containing the > module itself, any modules it depends on including the very basic > stuff > > 2) a single memory snapshot containing all the modules we want to > ship, any modules they depend on including the very basic stuff > > Option 1 wastes some of the CD-ROM, just like in Unix before dynamic > linking each executable file had a copy of the basic libraries. This doesn't seem significant to me, given that an object memory with the usual "basic libraries" is from 90k to 200k on disk. Also note that a module doesn't have to have all its prerequisites in the same object memory, so I don't think we'd be duplicating modules much (just the ones needed to start the minimal snapshot). > Option 2, however, does take up quite a bit of (virtual) memory... if > less than 650MB can be considered quite a bit in this era of 1GB RAMs. Surely one could make snapshots which are somewhere in beween 90k and 650MB. :) I think that's Option 3: a combination of large and small snapshots. We have the flexibility of grouping modules according to considerations other than size. It might be moot anyway; once you get a group of object memories started, they can form a single distributed object memory. > This system is certainly usable and has many advantages, but takes up > more RAM and disk than alternatives which allow images to be split > into chunks which can then be lumped together again. To summarize, I don't think either the additional RAM (~200k per object memory currently) or additional disk (~90k per snapshot currently) is significant. More importantly, this system is *accurate*, by virtue of the live negotation that takes place, whereas the non-message-based ones are not. thanks, -C -- Craig Latta improvisational musical informaticist craig@{netjam.org, weather-dimensions.com, appliedminds.com} www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)] From lex at cc.gatech.edu Wed Mar 9 14:57:33 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: Brainstorm In-Reply-To: <20050308135517.GW7317@iam.unibe.ch> References: <20050308135517.GW7317@iam.unibe.ch> Message-ID: Alexandre Bergel wrote: > For me, it is really not clear why and for what we need modules. The main pressing needs for the Squeak group seem to be: 1. Independent maintenance of parts of the Squeak-iverse. 2. Having load options, so that everyone's image can load just the things they care about The "independent" part of independent maintenance seems to be where most of the design activity is on this list. How do you keep the Tetris module from making SmaCC unusable? Lex From lex at cc.gatech.edu Wed Mar 9 14:30:25 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: full isolation In-Reply-To: <20050307184441.GQ7871@iam.unibe.ch> References: <20050304003525.A3F0425BAF2@mailhub01.unibe.ch> <20050307184441.GQ7871@iam.unibe.ch> Message-ID: Alexandre Bergel wrote: > I have been away these last days, that's why I was pretty silent... > Would it be possible to set up this list in such a way that when one answers, the answer is sent to the list rather than privately. This is the Squeak standard, though I have railed against it in the past. Here's what convinced me: http://www.unicom.com/pw/reply-to-harmful.html I would request that people do send emails to the list only, and not CC me. But if you want to CC me, no technology is going to prevent that! > > There has been some murmering about full isolation, both pro and > > against. The idea is that modules will be totally insulated from each > > other and thus unable to cause each other any harm. > > One serious issue concern meta programming. I believe that achieving a > descent isolation mechanism require a complete overhaul of the reflective > facility. It seems that the Self mirror mechanism would provide an interesting > solution... Hmm. In practice, it seems like any near-term isolation mechanism is going to only go so far. It is okay if certain reflection mechanisms break isolation, so long as those mechanisms are treated as meta-tools and not as everyday programming tools. That said, it's great if you can make more reflective tools follow isolation restrictions, so long as there is some mechanism around for programmers to look under the hood when they need to. > > 1. Every module must have its own thread. Otherwise, > > Why ? Classboxes do not impose such a constraint. > > > 2. Modules must interact by passing messages or events, and not by > > simple call and return. Otherwise, when you call, the other module can > > simply choose not to ever respond. > > ??? > Can you provide a concrete example ? Here is what I am thinking of: EvilWebComponent>>process: webRequest [true] whileTrue. What do you do about this, other than give the thing its own thread? And if it has its own thread, you still cannot simply do RPC to that thread, because the same attack is possible. These are not my ideas, by the way, even though I am convinced. This is the rationale behind E's "vats" system. I am not convinced that you have to use something as complicated as E's promise pipelining, but I do beileve you need to have separate threads and that the underlying mechanism needs to be based on something asynchronous like messages or promises. > Look at the Selector Namespace and Classboxes... I have looked at it already, and plan to read more about it. It sounds like a great way to do things, and I really don't care whether it achieves full isolation or not. I remember liking the overall idea behind it... but I won't try to summarize that now because I'll botch it! I simply worry that if you chase the grail of full isolation, you may be forced into some unpleasant choices, and it's not necessary anyway. I don't think the EvilWebComponent above is a common enough case that it is necessary to prevent that attack at *every* module boundary. Preventing a large class of accidental conflicts, is already a wonderful improvement; it is not necessary to prevent all conflicts including malicious ones. -Lex From lex at cc.gatech.edu Wed Mar 9 14:39:51 2005 From: lex at cc.gatech.edu (Lex Spoon) Date: Wed Jul 19 10:23:23 2006 Subject: modules on CD (was: full isolation) In-Reply-To: <200503081531.j28FVkIN017852@sark.cc.gatech.edu> References: <422AA15D.90909@netjam.org> <200503081531.j28FVkIN017852@sark.cc.gatech.edu> Message-ID: I feel like such a grunt reading some of these threads. I don't want to interfere with any of the beautiful ideas flying around, but please notice that there are simple "hammer the nail in, already" solutions to distributing packages to people not on the 'net. The Stable 3.7 Universe is one such example: if you unzip Squeak3.7u1.zip onto a CD-ROM, you are done. Additionally, look to almost any Linux distribution to see how they do it. The decision about what goes on the CD, is less elegant than the ideas being tossed around. In the universes example, the CD easily holds everything. (If you think that all the Squeak code in the world takes more than one CD, well... ahem. Don't worry!) Linux distributions have to pick and choose what goes on the initial CD, and there is sometimes work to break dependencies or provide simpler alternatives of common programs, in order to fit a good set of things on that important first CD. That said, please carry on. :) Lex