OK, I just commited, you can check if it helps...

Nicolas

2012/8/13 Levente Uzonyi <leves at elte.hu>:
> On Mon, 13 Aug 2012, Nicolas Cellier wrote:
>
>> That would answer:
>>
>> #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7
>> '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2'
>> #number) #(15 14 '' #doIt))
>>
>> Is it OK to have a final empty #doit ?
>
>
> I guess it is. If #doIt is there (which used to be), then it should be
> empty.
>
>
> Levente
>
>
>>
>> Nicolas
>>
>> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>>>
>>> It should more or less use the same logic than Parser>>advance
>>>
>>>         hereEnd := source position - (aheadChar == DoItCharacter
>>>                 ifTrue: [hereChar == DoItCharacter
>>>                         ifTrue: [0]
>>>                         ifFalse: [1]]
>>>                 ifFalse: [2]).
>>>
>>> Nicolas
>>>
>>> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>>>>
>>>> Yep, this code is highly questionnable:
>>>>
>>>> "Compensate for the fact that the parser uses two character lookahead.
>>>>  Normally we must
>>>>                   remove the extra two characters.  But this mustn't
>>>> happen for the
>>>> last token at the end of stream."
>>>>                  aBlock
>>>>                         value: mark
>>>>                         value: (source atEnd
>>>>                                         ifTrue: [tokenType := #doIt. "to
>>>> cause an immediate ^self" source position]
>>>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO
>>>> CHARACTER AHEAD"
>>>>                                         ifFalse: [source position -
>>>> 2])].
>>>>
>>>> Nicolas
>>>>
>>>> 2012/8/13 Levente Uzonyi <leves at elte.hu>:
>>>>>
>>>>> On Mon, 13 Aug 2012, Colin Putney wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'd like to have the preference to allow underscore selectors working
>>>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the
>>>>>> preference, ideally without degrading the performance of fileIn and
>>>>>> package loading. I don't remember all the issues that came up when the
>>>>>> preference was introduced, though. Is there anything else that needs
>>>>>> to be done?
>>>>>
>>>>>
>>>>>
>>>>> I started to collect the todos here:
>>>>>
>>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html
>>>>>
>>>>> One more thing that needs to be done is to implement a small tool,
>>>>> which can
>>>>> convert code with underscore assignments to ansi assignments. I have a
>>>>> prototype, but it's not fully working due to a bug in Scanner >>
>>>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the
>>>>> issue:
>>>>>
>>>>> Array streamContents: [ :stream |
>>>>>         | source scanner |
>>>>>         source := 'x := 1. x := 2'.
>>>>>         scanner := Scanner new.
>>>>>         scanner scanTokenPositionsIn: source into: [ :start :end |
>>>>>                 stream nextPut: { start. end. source copyFrom: start
>>>>> to:
>>>>> end. scanner instVarNamed: #tokenType } ] ]
>>>>>
>>>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow
>>>>> and
>>>>> #number, but the second one is not. The last 2 tokens are combined into
>>>>> a
>>>>> #doIt, so the code cannot find fix last assignment in a method.
>>>>>
>>>>>
>>>>> Levente
>>>>>
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>
>>
>>
>