[squeak-dev] Re: [Pharo-dev] String >> #=

Eliot Miranda eliot.miranda at gmail.com
Wed May 28 01:53:58 UTC 2014


Hi Andres,


On Tue, May 27, 2014 at 6:10 PM, Andres Valloud <
avalloud at smalltalk.comcastbiz.net> wrote:

> What is going to happen when one compares two general Unicode series of
> characters that represent the same string but differ in normalization?
> Wouldn't the size test would result in false negatives?
>
> http://unicode.org/reports/tr15/
>
> I'm asking because I haven't seen any discussion on the subject, and the
> decision to change the code as proposed could have side effects.


The issue is whether String supports variable-sized encodings such as UTF-8
where there is no fixed relationship between the number of bytes i a string
and the number of characters in a string.  Right now we have ByteString and
WideString.  ByteString has 1 byte per character.  WideString has 4 bytes
per character.  So 'hello' asByteString contains 5 bytes and has size 5,
but 'hello' asWideString contains 20 bytes and also has size 5.  Hence the
size check is fine, since size answers the number of characters, not the
number of bytes.  If we were to add a UTF8String we'd have to delete the
size check.  But I think for now we're not going to do that.

A ByteString can contain some characters that comprise a UTF-8 string (see
UTF8TextCoverter) but that's a convention of usage.  if you print some
ByteString containing the UTF-8 encoding of a string containing characters
that take more than one byte to encode, that string won't print as the
input, it'll print treating each byte as a character, and so will scramble
the string.  It is up to the user to handle these ByteStrings that happen
by convention to contain UTF-8 correctly.

Note that there is nothing to stop us adding a UTF8String *provided* that
class implements size to answer the number of characters, not the number of
bytes.  My understanding is that VW takes this approach also. File streams
expose the encoding, sicne position is a byte position, not a character
position, and so it is up to the file stream client to cope with the
positioning complexities that this introduces, not the stream.


OK?


>
> On 5/27/14 11:59 , Eliot Miranda wrote:
>
>>
>>
>>
>> On Tue, May 27, 2014 at 6:54 AM, J. Vuletich (mail lists)
>> <juanlists at jvuletich.org <mailto:juanlists at jvuletich.org>> wrote:
>>
>>     __
>>
>>     Quoting Eliot Miranda <eliot.miranda at gmail.com
>>     <mailto:eliot.miranda at gmail.com>>:
>>
>>      Hi Phillipe,
>>>
>>>
>>>     On Mon, May 26, 2014 at 12:51 AM, Philippe Marschall
>>>     <philippe.marschall at netcetera.ch
>>>     <mailto:philippe.marschall at netcetera.ch>> wrote:
>>>
>>>         Hi
>>>
>>>         I have been investigating why Dictionary look up performance
>>>         with String keys is not as good as I would expected. Something
>>>         I noted is that String >> #= is implemented in terms of
>>>         #compare:with:collated:. There is no short circuit if Strings
>>>         are not the same size. In my case some Strings have the same
>>>         prefix but a different length eg 'Content-Type' and
>>>         'Content-Length'. In that case a #compare:with:collated: is
>>>         performed even though we know in advance the answer will be
>>>         false because they have different sizes.
>>>
>>>     Why not rewrite
>>>     String>>= aString
>>>     "Answer whether the receiver sorts equally as aString.
>>>     The collation order is simple ascii (with case differences)."
>>>     aString isString ifFalse: [ ^ false ].
>>>     ^ (self compare: self with: aString collated: AsciiOrder) = 2
>>>     as
>>>     String>>= aString
>>>     "Answer whether the receiver sorts equally as aString.
>>>     The collation order is simple ascii (with case differences)."
>>>     (aString isString
>>>     and: [self size = aString size]) ifFalse: [^false].
>>>     ^ (self compare: self withSize: with: aString collated:
>>>     AsciiOrder) = 2
>>>     ?
>>>
>>
>>
>> This makes a huge difference, over 3 times faster:
>>
>> | bs t1 t2 |
>> bs := ByteString allInstances first: 10000.
>> t1 := [bs do: [:a| bs do: [:b| a = b]]] timeToRun.
>> (FileStream fileNamed: '/Users/eliot/Squeak/Squeak4.5/String-=.st')
>> fileIn.
>> t2 := [bs do: [:a| bs do: [:b| a = b]]] timeToRun.
>> { t1. t2 } #(13726 4467)
>> 4467 - 13726 / 137.26 -67.46%
>>
>>      One /could/ add a replacement compare:with:collated:
>>>     primitive primitiveCompareString which took the sizes as arguments
>>>     to avoid asking twice.  But it wouldn't be safe.  One could abuse
>>>     the primitive and lie about the size.  So I suspect it is best to
>>>     add the size check to String>>#= and accept the duplication of
>>>     the primitive finding the sizes of the two strings.  The cost in
>>>     the primitive is minimal.  A WideString version of the primitive
>>>     might pay its way, but if Spur and Sista arrive soon the primitive
>>>     shouldn't be faster than the optimised Smalltalk code.
>>>     --
>>>     best,
>>>     Eliot
>>>
>>
>>     BTW, any good reason for not prefixing all the implementors of #=
>>     with this?
>>
>>     "Any object is equal to itself"
>>     self == argument ifTrue: [ ^ true ].
>>
>>
>> It doesn't make much difference:
>>
>> | bs t1 t2 |
>> bs := ByteString allInstances first: 10000.
>> t1 := [bs do: [:a| bs do: [:b| a = b]]] timeToRun.
>> (FileStream fileNamed: '/Users/eliot/Squeak/Squeak4.5/String-=.st')
>> fileIn.
>> t2 := [bs do: [:a| bs do: [:b| a = b]]] timeToRun.
>> { t1. t2 } #(4628 4560)
>>
>> 4560 - 4628 / 46.28 -1.47%
>>
>> So is it worth it?  If you feel it is I've no objection other than it
>> feels a little kludgey for such little benefit.  And there are the
>> Symbols if one needs quick comparison and can bear the cost of slow
>> interning.
>> --
>> best,
>> Eliot
>>
>
>


-- 
best,
Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20140527/acdbb4df/attachment.htm


More information about the Squeak-dev mailing list