[Newcompiler] Properties for AST

[Marcus Denker]

On 09.04.2007, at 10:23, Marcus Denker wrote:

>
> On 09.04.2007, at 09:00, Stéphane Ducasse wrote:
>
>> Hi
>>
>> I have a question is the information retained useful for other tools.
>>
>
> In case of the Tokens, I think we don't need them later. The whole  
> idea of providing a high-level
> representation is to be able to throw away the low level one

Another reason why retaining Tokens is a bad idea: They are purely  
implementation level artefacts.
The idea of the Scanner is  to tokenize the  input string so that the  
grammar can be formulated in
terms of scanner tokens. This simplifies the  grammar by a huge  
extend, and thus makes the Parser
faster, as the automatton generated is  far smaller.

But all this parser stuff is fairly old:  it was done in the  
seventies, when computers where *slow*
and had *extremely* small memories. It's purely implementation  
driven. And the cool thing is
that today's machines are so amazingly fast that we can start to  
questions very basic assumptions
defined 30 years ago that are kind of are regarded as "laws of  
nature". Scannerless parsing is one of them.

The extremely nice feature of scannerless parsers is that they are  
compositional: you can merge
any two grammers completely automatically. C with embedded ASM? no  
problem. HTML with embedded
strange language? Done.

This is *very* cool for DSLs (Domain Specific Languages) and in  
general systems where the
language should be extensible (growable) even on the level on the  
Grammar.

	Marcus



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3947 bytes
Desc: not available
Url : http://lists.squeakfoundation.org/pipermail/newcompiler/attachments/20070409/43a70975/smime.bin