In what follows, my answers are phrased in terms of the question which
spoke about macros per se, but should be understood as applying to
generative or meta-level programming, which has a more understandable
relationship to Smalltalk than macros do per se. (For this reason, I found
the specific questions to be inappropriate, but still worth answering.)

This is a long email, to my regret. My request is that replies to this not
be long-winded or expect long answers, because I just don't have time to
deal with professors needing answers from a mailing list. Being unemployed
(or self-funding, as it were), I would appreciate basic self-
sustainability before someone decides that I should be posting several
pages of content per day without a fee. (Maybe I should preface all my
emails with links to my resume? :)

On Thu, 18 Sep 2003, ducasse wrote:

> Hi brian
>
> What I would like to know is much more at a meta level. For example
> what are the problem that we can encounter with a macro systems?

Avi Bryant asked this question recently:
http://www.cincomsmalltalk.com/userblogs/avi/blogView?showComments=true&entry=3239986014

and more significantly in a followup:
http://www.cincomsmalltalk.com/userblogs/avi/blogView?showComments=true&entry=3240052865

Basically pattern-matching macros are much more tractable in terms of
consistency for IDE UI purposes than the basic procedural macro.

> How do we manage the fact that everybody can define its own.... have you
> any experience wtih that. I remember visiting a company developing in
> lisp and they wanted to get rid of macros that were generating
> macro....10 lines was creating 6 pages of code. So on one hand this
> would be interesting but what is the price (not the technical one
> because we are good to access and solve it but the scalability and use
> one).

The way to deal with or comprehend macros is to grab the output or some
intermediate form of it (using full or step-wise macro-expansion.. which
is a standard lisp tool) and insert that output into your code. Of course,
a good Lisp programmer never does this as such, preferring to write new
macros for a refactoring and using the macro-expansions' comparison to
debug/unit-test. Or, to write a new common macro that lets you express
both the new and old pattern in it, and implement them both on top of
that. It's just refactoring and XP applied at the meta-level.

One reason that Lisp's tools aren't up to par with Smalltalk's (aside from
the big economic collapse) is that the output code of macro-expansion is
just a bunch of lists usually, so the important meta-information is not
explicitly transmitted with it. The reasons that are often cited about
IDE's suffering because macros (or any interesting meta-level code) break
regularity are missing the fact that any meta-level regularity and a
single substrate IDE framework for managing generative code would be able
to augment and grow the IDE as necessary. (Caveat: this doesn't really
exist, yet, except in piecemeal among several different research systems.)

At some point, though, you are just talking about generative programming
or meta-programming, and you just won't get a magical ability to translate
between different custom extensions, just as you can't magically press a
button and transition from one O/R database mapping to another. The one
important relation that is added is creator/created code consistency:
there are very few integrated development environments which as a whole
can know about these effects and handle them well. Dylan's Functional
Developer seems to have done the best in the tools area.

(If you want a deeper answer about the state of the art, you need to start
mining resources outside of Squeak and Smalltalk, perhaps into work that's
done in Haskell on monadic staged compiler phases or somesuch. The LL1
mailing list or Citeseer should be good enough for that.)

One thing I am intending to do with Slate is to use the prototype nature
to actually communicate macro information with its generated code as
annotations on the output source trees: it's really simple to say that
"this code was generated by a program that preserves these invariants", so
that's that much more information that the compiler/optimizer gets for
free, or the versioning system. Another thing is to have a mode for the
IDE where it can show the transition from generating code to generated
code, in steps or en masse.

Unfortunately, right now I am mired in getting Slate's self-hosted runtime
worked out in detail (and keeping the bills paid), so this will take some
time to demonstrate (funding, anyone? :).

PS: BTW, I am intrigued by a generative programming idea that was
prototyped by Henrik Gedenryd in Squeak a couple of years ago, called
Universal Composition, and am still looking for contact info for him.

-- 
Brian T. Rice
LOGOS Research and Development
http://tunes.org/~water/