UTF8 Squeak
Janko Mivšek
janko.mivsek at eranova.si
Mon Jun 11 20:51:32 UTC 2007
Hi Yoshiki,
Yoshiki Ohshima wrote:
> Janko,
>
>> It seems that this was already a Yoshiki idea with WideString, so I'm
>> just extending that idea with a TwoByteString to cover 16 bits too.
>>
>> Yoshiki, am I right?
>
> For storing the bare Unicode code points, I think so. I'm not
> convinced that adding 16-bit variation solves any real problems. But
> there may be something.
For Slovenian language with Latin 2 script we need to have
TwoByteStrings, same goes for all East Europe, Greek, and Cyrillic. And
because I'm using an image as a database, I just cannot afford 4 byte
strings... And for shorter Slovenian strings even ByteStrings suffice.
> - While vast majority of strings for, say, Japanese can be
> represented with in the characters in BMP, you would use
> FourByteString for Chinese/Japanese/Korean and some others. Does
> this mean that you would *always* use FourByteString for these
> "languages" (and not scripts?)
My proposal allows strings to "scale" to support wider characters, by
widen themselves, from Byte to TwoByte and then FourByteString.
Determination of width of a string is automatic (as is already for
WideString): you start with ByteString and when you put a first
character with code point above 256, a ByteString is automatically
converted to TwoByteString or even FourByteString. Same goes for
TwoByteString when you add a character > 2**16.
Strings therefore don't need to be aware at all about languages they
support.
> - Suppose you would like to use different line wrapping algorithms
> for different languages, how would you keep that information?
Line ends internally should be Character cr only (how is that in Squeak
anyway?). Different line-ends are again a responsibility of streams to
the external world.
What about a separate Locale object for all that language specific
information?
Best regards
JAnko
--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
More information about the Squeak-dev
mailing list
|