[Newcompiler] Properties for AST

[Andrew Tween]

Hi Marcus,
----- Original Message ----- 
From: "Marcus Denker" <denker at iam.unibe.ch>
To: "The New Compiler, AST, IRBuilder,Closures..."
<newcompiler at lists.squeakfoundation.org>
Sent: Monday, April 09, 2007 9:23 AM
Subject: Re: [Newcompiler] Properties for AST



> In case of the Tokens, I think we don't need them later. The whole
> idea of providing a high-level
> representation is to be able to throw away the low level one. What we
> need from the Tokens
> is meta data: comments and possitions in source. The position data we
> then can flush if it is identical
> to the one the pretty printer would re-create. This way, we don't
> need any position/whitespace data
> on the AST of  any methods of the system other then newly created
> ones by people not using the
> pretty printer.

Doesn't this assume that the pretty printer will always produce the same output?
i.e. no-one will improve the pretty-printer later, or fix a bug in it, resulting
in different source.

You could have a pretty-printer just for the AST -> source transformation, and
never change it.
Or, have multiple pretty-printer versions, and store the pretty-printer version
number in the AST.
Or, always retain the position info.
Or always retain the source.

On the question of tokens...
One reason for using tokens rather than an AST is to deal with incomplete or
invalid source.

For example, Shout uses tokens so that it can style source as it typed. During
typing, the source is usually invalid, but a token stream is still obtainable,
and so the source can still be styled. There is little benefit to dealing with
saved methods as an AST and incomplete methods as a token stream. It is simpler
to treat both as a token stream.

I can imagine that someone, someday, will create a pretty printer that can
format incomplete, half-written methods.
And possibly a refactoring engine that can perform transformations on
incomplete, half-written methods.

The essential requirement is that the original source is always reproducable, or
otherwise obtainable. Then tools can use the AST, or get the source and parse it
and do their own thing.

Today, the source is gettable from .sources and .changes. So there is no need to
store it in an AST, or to use an AST to produce it.
Can I infer from what you have said that you are intending to get rid of method
source from .changes and .sources?

Cheers,
Andy