What *does* matter is to begin seeing what is going on when you try to herd all those cats into a nice orderly modular world. And it would be very helpful if you were cut a bit of slack in the early stages. A lot of slack would be better. In particular, I've been thinking that it might be useful if you were to go ahead and introduce modules now, but in a fashion where they are more or less just decorations. That is, go ahead and use the existing environment variable on the classes to point to bona-fide modules in whatever scheme you pick. But don't let it mean too much just yet- the classes would still be reachable through the Old Reliable means of the SystemDictionary Smalltalk, and your existing code tools would work no differently than before. Think of the module system as being no more valuable to you than a thin layer of paint- don't get too attached to it just yet. But go ahead and build new tools that use it, tools that realize that the older non-modular tools may do things behind their back.

Les -

This is a GREAT perspective on getting started. When I first started on Environments, I spent a day just reworking the systemOrganization to where it began to show some of what we are after. The I started attributing meaning to the major categories...

I have several reasons for suggesting this. One, obviously, is that it would be nice to try out different modularity schemes over time. But that isn't the most important reason at all. Easily the most important ( well it comes to mind quickly in the middle of the night, at any rate ) is to avoid the Class Initialization Problem. You don't want to tackle this yet. It will have to be handled, but not today.

Yes. But even if we go a bit farther this need not be a problem. I would be quite happy to have a module system that organized Squeak in a way that we liked, but where a couple of core modules were still incapable of self-initialization. For those, for now, they can remain part of the release, and we'll care more about unloading them (for really tiny kernels) than about loading them (since they are shipped as part of the image).

So this brings me to a second suggestion- just as we have an environment variable for classes ( and rightly so, AFAIK ), if we add an environment variable to CompiledMethods ( or a subclass which can be used by recompiling all methods in the system ) then you have a mechanism to keep track of which methods in a given class are "native", and which ones are really there because some other module wanted them there. This would be a crutch- I am not seriously suggesting this as a long term solution. Perhaps it is, but I don't have any insight to that at this stage. But if you did this, and had the tools to help you discern whether a given method belonged to the module in which its class is defined, or should instead be listed as an extension method provided by another module, I think you could get rolling relatively quickly.

Les, I don't think you need anything this deep. I think it would suffice to put this info in the nascent modules. It would not be hard to keep it up to date with a little hook in Behavior>>compile, and it wouldn't be all that hard to sweep the system again to bring things back in line until we have it all right. We'll be doing this anyway while we juggle things around into a nice factoring.

So, I guess what I'm proposing for this early stage of the game is a form of weak modularity, hopefully one that can grow gracefully into a strong modularity.

I think this is a wonderfully liberating approach. Merely installing such a weak module system enables us to then work on

A good factoring of the monolithic image

Enhancing our tools to work with modules

Shrink methods that really work (think of this as unloading weak modules ;-)

Various candidates for strong modules

Thanks for an empowering middle-of-the-night point of view.

- Dan

Dan Ingalls wrote:

Just as a PS, there is already code in the image that makes a reasonable attempt at this, even without knowing any module structure. If you run

Smalltalk reportClassAndMethodRemovalsFor: #(Celeste Scamper MailMessage)

[snip]

Going a bit further, there is a humungus cruncher in

Smalltalk computeImageSegmentation

Which image? I have Squeak 3.0, but these methods are not there.

- les

Henrik,

Just one small comment: It's simple. I like it ;-)

Cheers, - Andreas

-----Original Message----- From: squeak-dev-admin@lists.squeakfoundation.org [mailto:squeak-dev-admin@lists.squeakfoundation.org]On Behalf Of Henrik Gedenryd Sent: Saturday, August 25, 2001 11:00 AM To: squeak-dev@lists.squeakfoundation.org Subject: Re: [Modules] From here to there

...

it would be very helpful if you were cut a bit of slack in

the early stages.

...

A lot of slack would be better. In particular, I've been

thinking that it

...

might be useful if you were to go ahead and introduce

modules now, but in a

...

fashion where they are more or less just decorations.

Les,

this is an excellent point! The 'modularity of modules' faction awards you an instant honorary doctorate. (It's probably one of those dubious degrees that spam emails want to sell you, but hey, it's the thought that counts.)

In bringing this idea forward, I think the crucial point is to decide just how much 'real meaning' (vs. just decorations) these lightweight modules should have. There is a trade-off here: functionality that we leave out will not be debugged and improved to working quality. So a Goldilocks balance would be desirable.

So what we need next is to figure out just what would go into such a solution. If you recall my posting "[Modules] Let's get things rolling soon", much of it applies to this question. The idea was to put out something soon that contains just precisely what has to be there at first, and then collectively build from there.

I have been hacking up a Modules prototype derived from Dan's Environments, and after thinking just a little about just what would go into your lightweight scheme, and with the KISS approach I took, I don't think they are very far apart.

I'd like to see others' thoughts as well about what should go into this lightweight scheme. But I think as much of the modularity functionality as possible should be placed there, but be designed with a consistent 'opt-in', non-forcive policy.

And we could use some code analysis tools rather early on, too. (There are some seeds already.)

Ok, I have more comments but I'll post the code now for your weekend hacking pleasure, and return with more comments later. This is Squeak3.1a-4261 code, it should work with 4282 but I just haven't tried it.

A couple of notes:

1. This is a prototype/hack-up, don't take it too seriously.

It is highly incomplete, buggy, unfinished, etc etc. Beware of the two-headed calf!

2. Compare it with my writeup of a modularity proposal. This

follows to the proposal but does not cover it all yet though. There are some hints for an all-encompassing name space, such as Andrew Black suggested. These are just hints so far.

3. Instructions in the first preamble.

4. Note Dan's rudimentary code analysis tools, and his

beautiful solution that hunts down and rewrites global references in the source code.

5. The core is in the category System-Modules, it is rather

small. The rest is a bunch of patches to make the rest of the system follow this regime (but just barely so far).

Henrik

Joerg Beekmann wrote:

Is there a way to embed morphs into text so they behave as part of the text during resizing? The effect I'm after is demonstrated by the entry field in the HTML at the end of this message. I've tried various table layouts without success. The problem I'm having is that the layout considers each morph added as a cell and then arranges those cells. But what I want to do is have a morph be treated as a character in the text.

Joerg

<html> <body>

            This is a bunch of text in the body of my document.
            The text contains an embedded entry field that behaves like part of the text.
            <input type="text" value="the initial contents">
            This is some text following the entry field.

</html>

This is exactly what I am trying to figure out how to do in table support for Scamper. The solution is TextAnchors, but I don't know how to use them yet... Look at TextMorph>>addMorphFront: aMorph fromWorldPosition: wp "Overridden for more specific re-layout and positioning" | i | self addMorphFront: aMorph. i _ (self paragraph characterBlockAtPoint: (self transformFromWorld globalPointToLocal: wp)) stringIndex. self paragraph replaceFrom: i to: i-1 with: (Text string: '*' attribute: (TextAnchor new anchoredMorph: aMorph)) displaying: false. self fit

This method is used for drag and drop I guess because it refers to world position. The HtmlFormatter produces a text looking like this:' * * * ' where each asterisk is a text anchor, but when I initializes a TextMorph with this as string it will loose the anchor info. The anchor info is stored in a additional collection in the formatter.

Hope this helps. If you figure it out, let me know. I will report any progress I make to the list :-)

Karl

Joerg Beekmann wrote:

Karl; thanks for the tip. I've played with TextAnchors a bit and by running with the class comment for TextAnchor the following seems to get me close to what I (you?) want:

BUT then substituting a TextMorph for a PluggableTextMorph as below does not work. I get a window only showing the TextFieldMorph. Any ideas?

Both examples work for me( in a 3.1-42xx image). The second one shows up as white text on black background in a very wide window. From my experiments I found that TextMorphs were picky and could screw up layout totally Another issue is in this method that gets called from TextMorph>>contentsWrapped:

TextMorph>>newContents: stringOrText "Accept new text contents." | newText embeddedMorphs | newText _ stringOrText copy asText. "should be veryDeepCopy?" text = newText ifTrue: [^ self]. "No substantive change" text ifNotNil: [(embeddedMorphs _ text embeddedMorphs) ifNotNil: [self removeAllMorphsIn: embeddedMorphs. embeddedMorphs do: [:m | m delete]]].

text _ newText.

"add all morphs off the visible region; they'll be moved into the right place when they become visible. (this can make the scrollable area too large, though)" newText embeddedMorphs do: [:m | self addMorph: m. m position: -1000 @ 0]. self releaseParagraph. "update the paragraph cache" self paragraph. "re-instantiate to set bounds" self world ifNotNil: [self world startSteppingSubmorphsOf: self]

I don't really understand why submorphs get stripped out of the text and added back in at -1000@0 ? I comented out that part of the code and got more reliable results:

TextMorph>>newContentsWithEmbeddedMorphs: stringOrText "Accept new text contents." | newText | newText _ stringOrText copy asText. "should be veryDeepCopy?" wrapFlag _ false. container ifNotNil: [container fillsOwner ifTrue: [wrapFlag _ true]]. text = newText ifTrue: [^ self]. "No substantive change" text _ newText. self releaseParagraph. "update the paragraph cache" self paragraph. "re-instantiate to set bounds" self world ifNotNil: [self world startSteppingSubmorphsOf: self]

Karl

On Saturday, August 25, 2001, at 02:00 PM, Henrik Gedenryd wrote:

... I have been hacking up a Modules prototype derived from Dan's Environments, and after thinking just a little about just what would go into your lightweight scheme, and with the KISS approach I took, I don't think they are very far apart.

I was just curious about how you were handling the default single-path inheritance aspect of namespaces in Environments. I agree that it should be possible to delegate name lookup to multiple imported modules, as you mention in your proposal. (ModSqueak and other module proposals seem to assume this as well.) Would this require abandoning the Environment-style "MyModule MyClass" message-send name lookup (as opposed to VisualWorks-style "MyModule.MyClass" syntax)?

(Apologies if this has already been worked out in an obvious manner. I confess to not being an expert on Environments...)

I'd like to see others' thoughts as well about what should go into this lightweight scheme. But I think as much of the modularity functionality as possible should be placed there, but be designed with a consistent 'opt-in', non-forcive policy.

Also, I'm not sure if there's any consensus yet on whether supporting "loose methods" (a.k.a. class extensions) should be part of an initial lightweight scheme... they're not specifically mentioned in your proposal. I notice that Dan's #reportClassAndMethodRemovalsFor: utility does identify loose methods, for example. (ModSqueak and most other non-lightweight proposals support them.) Whether or not they're supported would certainly have an effect on how the initial breaking up of the image would be handled. (e.g. solving the ol' String>>asHtml problem by storing that method in its appropriate html-related module) I guess I would personally like to see loose methods supported, unless it turns your lightweight proposal into a heavyweight one. :-)

Good work on the stuff you've done so far,

- Doug Way dway@riskmetrics.com

Well, the weekend didn't turn out to be so productive for me, so at this time I have no comments on this. However, this evening I have made some progress on starting a few new tools. More on that in a little while- basicallly the idea is to construct tools to support the initial generation, modification, saving, and resurrection of a modular partitioning of the Smalltalk system. If you have these four things, plus something that is watching what is going on behind your back as the regular tools change things, then it should be possible to avoid the problem of having to re-generate the modular world from scratch every time you try something new.

I'll try to get up to date on everything soon, though I'll be primarily focusing my attention on tools for transformation and analysis of modular ( and non-modular ) systems. I've been doing that for quite a while, so this should be straightforward.

- les

I would be especially interested at this point to hear from Les, Hans-Martin, and Joseph, regarding whether

such a target structure in the image could work reasonably with their existing bodies of code, or whether we should be considering a different kernel or at least some changes to this one.

• Dan

--

I'll try to be a little more clear:

Java has 4 access rules for class members (instance variables):

public -this has no equivalent in Smalltalk private - this has no equivalent in Smalltalk protected -this is the standard for instance variables in Smalltalk package access -this has no equivalent in Smalltalk

What I am suggesting is that two new types of access be defined for Smalltalk:

Module instance variables -accessible only by objects of a class or sub-class defined within the module -roughly equivalent to package access in Java

Module global variables -accessible via the pool dictionary strategy ONLY to classes defined within a module (and to any subclass?) -roughly equivalent to package access for pool variables.

Module instance variables can be handled in the class definition:

moduleInstanceVariableNames''

Module global variables are handled by using the dummy dictionary name "Module" which defaults to "Smalltalk" if the package is used by itself.

All classes in a given module should have automatic access to the "Module" pool, just as they do to "Smalltalk" but ONLY classes defined in a given module (and their sub-classes?) should have access to the Module dictionary.

Hope this makes more sense.

on 8/29/01 2:26 PM, Lawson English at english7@mindspring.com wrote:

I haven't seen it mentioned before, so i'll throw this in:

should there be a new kind of instance variable defined that is associated with a given module? Likewise, a "global" module variable might be defined as well.

This might be a variant of pooled variables where only classes defined within the module know about and can use variables in the specific module pool, and using the module-pool dictionary is implicitly managed since there is only one of them per module and the name of the dictionary is redundant and probably better left unspecified anyway.

on 8/29/01 8:04 PM, Mark van Gulik at ghoul6@home.com wrote:

Hm. The concept you (or someone else) seem to be proposing regarding module instance variables is that of a state extension to a class. This would be entirely superfluous in Avail because of my concept of Power Set Inheritance, but I can see how it would be useful in Squeak.

Perhaps. I merely wanted to create an extension to Squeak's variable handling that was based on the existing one. Note that the default is what you already get from Smalltalk: Module pool = Smalltalk pool & moduleInstanceVariableNames = instanceVariableNames if the module concept hasn't been implemented at the lowest level or if there is only one module used in a project.

It also sidesteps questions concerning namespaces. If there WAS a naming conflict between a subclass's module pool and the superclass's pool, you could resolve it by prepending "supermodule::" to the global variable name (or some other equivalent syntax more in keeping with Smalltalk) without having to have resolved the general namespace issue.

["supermodule::variable_name" would allow the sub-class to access the global module pool of the superclass rather than the global module pool of the sub-class, just to make things clear]

Les,

The only logic I don't follow is why you think the module reference needs to be physically contained in the class or CompiledMethod object.

Wouldn't it be easier to have a parallel structure that recorded for each class or method, it's defining module. To save space, at the method level this could be an exception table. A method is assumed to be defined by the same module as its class, unless a different module is explicitly listed for it. Also, it should be easier to deal with global variable and other edge artifacts using a scheme like this.

There are lots of good reasons for this separation. First, it separates the essential runtime meta structures that are required by the virtual machine from the development/management meta data that may only be needed by development tools. This makes life much easier for people who want to generate minimal-size application deployment images that don't need any development support.

Another advantage of such separation is that, in theory, you could define multiple modular views upon the same image. This might be quite interesting during the phase when people are experimenting with alternative ways to modularize the system.

Finally, this type of independent representation of the structure of the image is essentially the basis of the "semantic model" used by ModSqueak, Team/V, and other similar systems. Once you take a little step in that direction, its easy to take additional steps and do a lot of neat things.

Allen

At 10:40 PM 8/23/2001 -0700, Les Tyrrell wrote:

The last few weeks we've seen a tremendous amount of discussion on bringing modularity to Squeak, and I know that some of you are really itching to get on with the show. The last few weeks I've been dusting off a few things, reconsidering others, and generally getting back into the swing of things. I've thrown out a few cautions, but hopefully these are not overly pessimistic. The inventory of working code waiting just offstage is actually rather impressive, and I'm not just tooting my own horn here ( the same would be true if none of my stuff had ever been built ). I wish I were in a position to throw out Oasis to the masses and say "here's your answer, live long and prosper!", but that won't happen for a while yet- I still have some things to explore there before I'll have time to release it. So, out of practical concerns, I'm basically limited to making a few comments here and there.

The last couple of days I've been pondering how to move Squeak forward, but perhaps without getting bogged down in some of the potholes that I've encountered while building Oasis. Bear in mind that I am not claiming that there are any quick and easy shortcuts- I shall judiciously avoid using the word "just", for instance- I've come to really find that word to be annoying.

In the early stages, it will not much matter how you build modules, or what characteristics they have. If you have *some* kind of module system, the code it handles can be transferred to a different module system at a later date when the issues are better understood and you can see where the deficiencies really lie, not just where you anticipate them lying.

What *does* matter is to begin seeing what is going on when you try to herd all those cats into a nice orderly modular world. And it would be very helpful if you were cut a bit of slack in the early stages. A lot of slack would be better. In particular, I've been thinking that it might be useful if you were to go ahead and introduce modules now, but in a fashion where they are more or less just decorations. That is, go ahead and use the existing environment variable on the classes to point to bona-fide modules in whatever scheme you pick. But don't let it mean too much just yet- the classes would still be reachable through the Old Reliable means of the SystemDictionary Smalltalk, and your existing code tools would work no differently than before. Think of the module system as being no more valuable to you than a thin layer of paint- don't get too attached to it just yet. But go ahead and build new tools that use it, tools that realize that the older non-modular tools may do things behind their back.

I have several reasons for suggesting this. One, obviously, is that it would be nice to try out different modularity schemes over time. But that isn't the most important reason at all. Easily the most important ( well it comes to mind quickly in the middle of the night, at any rate ) is to avoid the Class Initialization Problem. You don't want to tackle this yet. It will have to be handled, but not today. Example: in Oasis I can and do load the entire class libraries of many different Smalltalk dialects. But to no avail- you can't initialize them because they are way too deeply non-modular. Just can't get there from zero, or so it seems, certainly not without some doing some pretty fancy stuff, or just accepting the fact that you have to start with a completely new class library. If I could, then I could run into the next problem on the list, which is along the lines of "What to do about nil?" and so forth. But Allen says I can get around that, so I won't worry about that for now. ( actually, I can get around this by using PocketSt, but that's not my point ).

If you side-step the CIP, you can have modules as soon as you define class Module in your likeness. The first thing you will notice is that there certainly are an awful lot of methods out there that your modules are providing for classes that they don't define. So this brings me to a second suggestion- just as we have an environment variable for classes ( and rightly so, AFAIK ), if we add an environment variable to CompiledMethods ( or a subclass which can be used by recompiling all methods in the system ) then you have a mechanism to keep track of which methods in a given class are "native", and which ones are really there because some other module wanted them there. This would be a crutch- I am not seriously suggesting this as a long term solution. Perhaps it is, but I don't have any insight to that at this stage. But if you did this, and had the tools to help you discern whether a given method belonged to the module in which its class is defined, or should instead be listed as an extension method provided by another module, I think you could get rolling relatively quickly. You would have to be tolerant of the Old Reliable tools creating these new CompiledMethods without giving them a reference to a module, but that would be easy to accomodate with the new tools.

So, I guess what I'm proposing for this early stage of the game is a form of weak modularity, hopefully one that can grow gracefully into a strong modularity. I'll give it some more thought, and will try to give this a try both in Squeak and in my Oasis development image to see how well it works sometime this weekend. (These things almost never work so easily)

Take care!

• les