On Mon, 13 Aug 2012, Nicolas Cellier wrote:

> OK, I just commited, you can check if it helps...

Great, thanks.


Levente

>
> Nicolas
>
> 2012/8/13 Levente Uzonyi <leves at elte.hu>:
>> On Mon, 13 Aug 2012, Nicolas Cellier wrote:
>>
>>> That would answer:
>>>
>>> #(#(1 1 'x' #word) #(3 4 ':=' #leftArrow) #(6 6 '1' #number) #(7 7
>>> '.' #period) #(9 9 'x' #word) #(11 12 ':=' #leftArrow) #(14 14 '2'
>>> #number) #(15 14 '' #doIt))
>>>
>>> Is it OK to have a final empty #doit ?
>>
>>
>> I guess it is. If #doIt is there (which used to be), then it should be
>> empty.
>>
>>
>> Levente
>>
>>
>>>
>>> Nicolas
>>>
>>> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>>>>
>>>> It should more or less use the same logic than Parser>>advance
>>>>
>>>>         hereEnd := source position - (aheadChar == DoItCharacter
>>>>                 ifTrue: [hereChar == DoItCharacter
>>>>                         ifTrue: [0]
>>>>                         ifFalse: [1]]
>>>>                 ifFalse: [2]).
>>>>
>>>> Nicolas
>>>>
>>>> 2012/8/13 Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:
>>>>>
>>>>> Yep, this code is highly questionnable:
>>>>>
>>>>> "Compensate for the fact that the parser uses two character lookahead.
>>>>>  Normally we must
>>>>>                   remove the extra two characters.  But this mustn't
>>>>> happen for the
>>>>> last token at the end of stream."
>>>>>                  aBlock
>>>>>                         value: mark
>>>>>                         value: (source atEnd
>>>>>                                         ifTrue: [tokenType := #doIt. "to
>>>>> cause an immediate ^self" source position]
>>>>> "ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO
>>>>> CHARACTER AHEAD"
>>>>>                                         ifFalse: [source position -
>>>>> 2])].
>>>>>
>>>>> Nicolas
>>>>>
>>>>> 2012/8/13 Levente Uzonyi <leves at elte.hu>:
>>>>>>
>>>>>> On Mon, 13 Aug 2012, Colin Putney wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'd like to have the preference to allow underscore selectors working
>>>>>>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the
>>>>>>> preference, ideally without degrading the performance of fileIn and
>>>>>>> package loading. I don't remember all the issues that came up when the
>>>>>>> preference was introduced, though. Is there anything else that needs
>>>>>>> to be done?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I started to collect the todos here:
>>>>>>
>>>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html
>>>>>>
>>>>>> One more thing that needs to be done is to implement a small tool,
>>>>>> which can
>>>>>> convert code with underscore assignments to ansi assignments. I have a
>>>>>> prototype, but it's not fully working due to a bug in Scanner >>
>>>>>> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the
>>>>>> issue:
>>>>>>
>>>>>> Array streamContents: [ :stream |
>>>>>>         | source scanner |
>>>>>>         source := 'x := 1. x := 2'.
>>>>>>         scanner := Scanner new.
>>>>>>         scanner scanTokenPositionsIn: source into: [ :start :end |
>>>>>>                 stream nextPut: { start. end. source copyFrom: start
>>>>>> to:
>>>>>> end. scanner instVarNamed: #tokenType } ] ]
>>>>>>
>>>>>> The first assignment x := 1 is tokenized properly to #word, #leftArrow
>>>>>> and
>>>>>> #number, but the second one is not. The last 2 tokens are combined into
>>>>>> a
>>>>>> #doIt, so the code cannot find fix last assignment in a method.
>>>>>>
>>>>>>
>>>>>> Levente
>>>>>>
>>>>>>>
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>
>>>
>>>
>>
>
>