Yep, this code is highly questionnable:

"Compensate for the fact that the parser uses two character lookahead.
 Normally we must
		  remove the extra two characters.  But this mustn't happen for the
last token at the end of stream."
		 aBlock
			value: mark
			value: (source atEnd
					ifTrue: [tokenType := #doIt. "to cause an immediate ^self" source position]
"ABOVE LINE WILL CAUSE A PREMATURE ^self SINCE THE SCANNER SCANS TWO
CHARACTER AHEAD"
					ifFalse: [source position - 2])].

Nicolas

2012/8/13 Levente Uzonyi <leves at elte.hu>:
> On Mon, 13 Aug 2012, Colin Putney wrote:
>
>> Hi all,
>>
>> I'd like to have the preference to allow underscore selectors working
>> in 4.4. To do that, we'll need to fix Character>>tokenish to honor the
>> preference, ideally without degrading the performance of fileIn and
>> package loading. I don't remember all the issues that came up when the
>> preference was introduced, though. Is there anything else that needs
>> to be done?
>
>
> I started to collect the todos here:
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-July/164980.html
>
> One more thing that needs to be done is to implement a small tool, which can
> convert code with underscore assignments to ansi assignments. I have a
> prototype, but it's not fully working due to a bug in Scanner >>
> #scanTokenPositionsIn:into:. Here's a snippet that demonstrates the issue:
>
> Array streamContents: [ :stream |
>         | source scanner |
>         source := 'x := 1. x := 2'.
>         scanner := Scanner new.
>         scanner scanTokenPositionsIn: source into: [ :start :end |
>                 stream nextPut: { start. end. source copyFrom: start to:
> end. scanner instVarNamed: #tokenType } ] ]
>
> The first assignment x := 1 is tokenized properly to #word, #leftArrow and
> #number, but the second one is not. The last 2 tokens are combined into a
> #doIt, so the code cannot find fix last assignment in a method.
>
>
> Levente
>
>>
>> Colin
>>
>>
>