On Mon, 13 Aug 2012, Nicolas Cellier wrote:

> That would answer:
>
> #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7
> '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2'
> #number) #(15 14 '' #doIt))
>
> Is it OK to have a final empty #doit ?

I guess it is. If #doIt is there (which used to be), then it should be 
empty.


Levente

>
> Nicolas
>
> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>> It should more or less use the same logic than Parser>>advance
>>
>>         hereEnd := source position - (aheadChar == DoItCharacter
>>                 ifTrue: [hereChar == DoItCharacter
>>                         ifTrue: [0]
>>                         ifFalse: [1]]
>>                 ifFalse: [2]).
>>
>> Nicolas
>>
>> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>>> Yep, this code is highly questionnable:
>>>
>>> "Compensate for the fact that the parser uses two character lookahead.
>>>  Normally we must
>>>                   remove the extra two characters.  But this mustn't happen for the
>>> last token at the end of stream."
>>>                  aBlock
>>>                         value: mark
>>>                         value: (source atEnd
>>>                                         ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position]
>>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO
>>> CHARACTER AHEAD"
>>>                                         ifFalse: [source position - 2])].
>>>
>>> Nicolas
>>>
>>> 2012/8/13 Levente Uzonyi <leves at elte.hu>:
>>>> On Mon, 13 Aug 2012, Colin Putney wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'd like to have the preference to allow underscore selectors working
>>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the
>>>>> preference, ideally without degrading the performance of fileIn and
>>>>> package loading. I don't remember all the issues that came up when the
>>>>> preference was introduced, though. Is there anything else that needs
>>>>> to be done?
>>>>
>>>>
>>>> I started to collect the todos here:
>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html
>>>>
>>>> One more thing that needs to be done is to implement a small tool, which can
>>>> convert code with underscore assignments to ansi assignments. I have a
>>>> prototype, but it's not fully working due to a bug in Scanner >>
>>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue:
>>>>
>>>> Array streamContents: [ :stream |
>>>>         | source scanner |
>>>>         source := 'x := 1. x := 2'.
>>>>         scanner := Scanner new.
>>>>         scanner scanTokenPositionsIn: source into: [ :start :end |
>>>>                 stream nextPut: { start. end. source copyFrom: start to:
>>>> end. scanner instVarNamed: #tokenType } ] ]
>>>>
>>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow and
>>>> #number, but the second one is not. The last 2 tokens are combined into a
>>>> #doIt, so the code cannot find fix last assignment in a method.
>>>>
>>>>
>>>> Levente
>>>>
>>>>>
>>>>> Colin
>>>>>
>>>>>
>>>>
>
>