[squeak-dev] The Trunk: Multilingual-ul.238.mcz
commits at source.squeak.org
commits at source.squeak.org
Wed Jun 13 15:09:01 UTC 2018
Levente Uzonyi uploaded a new version of Multilingual to project The Trunk:
http://source.squeak.org/trunk/Multilingual-ul.238.mcz
==================== Summary ====================
Name: Multilingual-ul.238
Author: ul
Time: 13 June 2018, 5:07:40.804457 pm
UUID: 8f89eae0-1eca-48c6-a3f7-ebe13afa2dbe
Ancestors: Multilingual-nice.237
Changes to various TextConverters:
#nextPut:toStream: can not be used to initialize latin1Map and latin1Encodings anymore (because most implementations reference them), so
- moved the current implementation of #initializeLatin1MapAndEncodings to ByteTextConverter, because only those converters understand #encode:.
- TextConverter >> #initializeLatin1MapAndEncodings has become subclass responsibility
- implemented #initializeLatin1MapAndEncodings in UTF* text converters
- removed class-side #initialize implementations. Will add a new one to TextConverter once all subclasses can reinitialize their tables.
- added class side #initializeTables which is expected to initialize the various class side tables in the correct order
- postscript will reinitialize class side tables where possible (all converters except for EUCTextConverter, CNGBTextConverter, EUCJPTextConverter, EUCKRTextConverter, ShiftJISTextConverter, CompoundTextConverter, KOI8RTextConverter)
- latin1Map and latin1Encodings are not initialized lazily anymore
UTF16TextConverter will take endianness into account when writing the BOM. Same thing is probably missing from UTF32TextConverter too.
=============== Diff against Multilingual-nice.237 ===============
Item was removed:
- ----- Method: ByteTextConverter class>>initialize (in category 'class initialization') -----
- initialize
-
- self == ByteTextConverter
- ifTrue: [self allSubclassesDo: [:c | c initialize]]
- ifFalse: [self
- initializeDecodeTable;
- initializeEncodeTable;
- initializeLatin1MapAndEncodings]
- !
Item was added:
+ ----- Method: ByteTextConverter class>>initializeLatin1MapAndEncodings (in category 'class initialization') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast.
+ This implementation assumes that encodeTable is initialized."
+
+ latin1Map := ByteArray new: 256.
+ latin1Encodings := Array new: 256.
+ 0 to: 255 do:[:i |
+ | latin1 encoded |
+ latin1 := String with: (Character value: i).
+ [encoded := String with: (Character value: (self new encode: latin1 first charCode))]
+ ifError: [].
+ latin1 = encoded ifTrue:[
+ latin1Map at: i+1 put: 0. "no translation needed"
+ ] ifFalse: [
+ latin1Map at: i+1 put: 1. "translation needed"
+ latin1Encodings at: i+1 put: encoded.
+ ]].!
Item was added:
+ ----- Method: ByteTextConverter class>>initializeTables (in category 'class initialization') -----
+ initializeTables
+
+ self == ByteTextConverter ifTrue: [ ^self ].
+ self
+ initializeDecodeTable;
+ initializeEncodeTable;
+ initializeLatin1MapAndEncodings
+ !
Item was changed:
+ ----- Method: TextConverter class>>initializeLatin1MapAndEncodings (in category 'class initialization') -----
- ----- Method: TextConverter class>>initializeLatin1MapAndEncodings (in category 'accessing') -----
initializeLatin1MapAndEncodings
"Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
- These variables ensure that conversions from latin1 ByteString is reasonably fast"
+ self subclassResponsibility
+
+ !
- latin1Map := ByteArray new: 256.
- latin1Encodings := Array new: 256.
- 0 to: 255 do:[:i |
- | latin1 encoded |
- latin1 := String with: (Character value: i).
- [encoded := String with: (Character value: (self new encode: latin1 first charCode))]
- ifError: [].
- latin1 = encoded ifTrue:[
- latin1Map at: i+1 put: 0. "no translation needed"
- ] ifFalse: [
- latin1Map at: i+1 put: 1. "translation needed"
- latin1Encodings at: i+1 put: encoded.
- ]].!
Item was added:
+ ----- Method: TextConverter class>>initializeTables (in category 'class initialization') -----
+ initializeTables
+
+ self initializeLatin1MapAndEncodings!
Item was changed:
----- Method: TextConverter class>>latin1Encodings (in category 'accessing') -----
latin1Encodings
"Answer an Array mapping latin1 characters to conversion string"
+ ^latin1Encodings!
- ^latin1Encodings ifNil:
- [self initializeLatin1MapAndEncodings.
- latin1Encodings]!
Item was changed:
----- Method: TextConverter class>>latin1Map (in category 'accessing') -----
latin1Map
"Answer a ByteArray map telling if latin1 characters needs conversion or not"
+ ^latin1Map!
- ^latin1Map ifNil:
- [self initializeLatin1MapAndEncodings.
- latin1Map]!
Item was added:
+ ----- Method: UTF16TextConverter class>>initializeLatin1MapAndEncodings (in category 'accessing') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
+
+ latin1Map := ByteArray new: 256 withAll: 1.
+ latin1Encodings := (0 to: 255) collect: [ :i | { 0. i } asByteArray asString ]!
Item was changed:
----- Method: UTF16TextConverter>>nextPut:toStream: (in category 'conversion') -----
nextPut: aCharacter toStream: aStream
| charCode |
aStream isBinary ifTrue: [ ^aCharacter storeBinaryOn: aStream ].
(useByteOrderMark and: [ byteOrderMarkDone not ]) ifTrue: [
+ self next16BitValue: (useLittleEndian ifTrue: [ 16rFFFE ] ifFalse: [ 16rFEFF ]) toStream: aStream.
- self next16BitValue: 16rFEFF toStream: aStream.
byteOrderMarkDone := true ].
(charCode := aCharacter charCode) < 256
ifTrue: [
(latin1Encodings at: charCode + 1)
ifNil: [ self next16BitValue: charCode toStream: aStream ]
ifNotNil: [ :encodedString | aStream basicNextPutAll: encodedString ] ]
ifFalse: [
charCode <= 16rFFFF
ifTrue: [ self next16BitValue: charCode toStream: aStream ]
ifFalse: [
| low high |
charCode := charCode - 16r10000.
low := charCode \\ 16r400 + 16rDC00.
high := charCode // 16r400 + 16rD800.
self
next16BitValue: high toStream: aStream;
next16BitValue: low toStream: aStream ] ].
^aCharacter!
Item was changed:
----- Method: UTF32TextConverter class>>initializeLatin1MapAndEncodings (in category 'utilities') -----
initializeLatin1MapAndEncodings
"Initialize the latin1Map and latin1Encodings.
+ These variables ensure that conversions from latin1 ByteString is reasonably fast."
- These variables ensure that conversions from latin1 ByteString is reasonably fast"
+ latin1Map := ByteArray new: 256 withAll: 1.
+ latin1Encodings := (0 to: 255) collect: [ :i | { 0. 0. 0. i } asByteArray asString ]
+ !
- latin1Map := (ByteArray new: 256) atAllPut: 1.
- latin1Encodings := (0 to: 255) collect: [:i | (ByteArray newFrom: {0 . 0 . 0 . i}) asString]!
Item was added:
+ ----- Method: UTF8TextConverter class>>initializeLatin1MapAndEncodings (in category 'as yet unclassified') -----
+ initializeLatin1MapAndEncodings
+ "Initialize the latin1Map and latin1Encodings. These variables ensure that conversions from latin1 ByteString is reasonably fast."
+
+ latin1Map := (0 to: 255) collect: [ :each | each bitShift: -7 ] as: ByteArray.
+ latin1Encodings := (0 to: 255) collect: [ :each |
+ each <= 127
+ ifTrue: [ nil ]
+ ifFalse: [
+ { 192 bitOr: (each bitShift: -6). (each bitAnd: 63) bitOr: 128 } asByteArray asString ] ]!
Item was changed:
+ (PackageInfo named: 'Multilingual') postscript: '"Initialize the tables in all TextConverters that support it."
+ TextConverter allSubclassesDo: [ :each | [ each initializeTables ] ifError: [] ]'!
- (PackageInfo named: 'Multilingual') postscript: 'Unicode
- initializeCaseFolding;
- initializeCompositionMappings'!
More information about the Squeak-dev
mailing list
|