Squeak-dev December 2015

squeak-dev@lists.squeakfoundation.org

60 participants
161 discussions

The Trunk: Morphic-cwp.1057.mcz
by commits＠source.squeak.org 08 Dec '15

08 Dec '15

Colin Putney uploaded a new version of Morphic to project The Trunk: http://source.squeak.org/trunk/Morphic-cwp.1057.mcz ==================== Summary ==================== Name: Morphic-cwp.1057 Author: cwp Time: 7 December 2015, 9:30:19.787 pm UUID: c8dfb49a-6bd4-486f-8a25-592c57330da7 Ancestors: Morphic-dtl.1056 Don't clobber the existing reference pool in PasteUpMorph>>referencePool =============== Diff against Morphic-dtl.1056 =============== Item was changed: ----- Method: PasteUpMorph>>referencePool (in category 'objects from disk') ----- referencePool + ^ self + valueOfProperty: #References + ifAbsentPut: [OrderedCollection new] + + ! - | pool | - pool := OrderedCollection new. - self setProperty: #References toValue: pool. - ^ pool!

1 0

HandMorph position gridding
by tim Rowledge 08 Dec '15

08 Dec '15

After much fun, I have a plausible mechanism that introduces a way to have a temporary cursor move on a grid rather than in single pixel steps. This is useful for painting programs that want to work on scaled up bitmaps, or perhaps for drawing programs wanting to place morphs on a grid (say, electrical component icons on a virtual breadboard?). It requires 4 new HandMorph methods, 1 small change to HandMorph>position: and either the use of a couple of properties in HandMorph or a couple of additional instvars. Does anyone have objections to this being added? Speak soon so I know if I have to try yet another tactic. tim -- tim Rowledge; tim(a)rowledge.org; http://www.rowledge.org/tim Useful random insult:- Doesn't just know nothing; doesn't even suspect much.

2 3

Squeak 4.6: System-cwp.755.mcz
by commits＠source.squeak.org 08 Dec '15

08 Dec '15

Colin Putney uploaded a new version of System to project Squeak 4.6: http://source.squeak.org/squeak46/System-cwp.755.mcz ==================== Summary ==================== Name: System-cwp.755 Author: cwp Time: 7 December 2015, 9:16:07.606 pm UUID: 8240fbad-2095-45dc-97af-65ab0d381a52 Ancestors: System-cwp.754, System-dtl.754 Merge =============== Diff against System-cwp.754 =============== Item was changed: ----- Method: SmalltalkImage>>recreateSpecialObjectsArray (in category 'special objects') ----- recreateSpecialObjectsArray "Smalltalk recreateSpecialObjectsArray" "To external package developers: **** DO NOT OVERRIDE THIS METHOD. ***** If you are writing a plugin and need additional special object(s) for your own use, use addGCRoot() function and use own, separate special objects registry " "The Special Objects Array is an array of objects used by the Squeak virtual machine. Its contents are critical and accesses to it by the VM are unchecked, so don't even think of playing here unless you know what you are doing." | newArray | newArray := Array new: 58. "Nil false and true get used throughout the interpreter" newArray at: 1 put: nil. newArray at: 2 put: false. newArray at: 3 put: true. "This association holds the active process (a ProcessScheduler)" newArray at: 4 put: (self specialObjectsArray at: 4) "(self bindingOf: #Processor) but it answers an Alias". "Numerous classes below used for type checking and instantiation" newArray at: 5 put: Bitmap. newArray at: 6 put: SmallInteger. newArray at: 7 put: ByteString. newArray at: 8 put: Array. newArray at: 9 put: Smalltalk. newArray at: 10 put: Float. newArray at: 11 put: MethodContext. newArray at: 12 put: BlockContext. newArray at: 13 put: Point. newArray at: 14 put: LargePositiveInteger. newArray at: 15 put: Display. newArray at: 16 put: Message. newArray at: 17 put: CompiledMethod. newArray at: 18 put: (self specialObjectsArray at: 18). "(low space Semaphore)" newArray at: 19 put: Semaphore. newArray at: 20 put: Character. newArray at: 21 put: #doesNotUnderstand:. newArray at: 22 put: #cannotReturn:. newArray at: 23 put: nil. "This is the process signalling low space." "An array of the 32 selectors that are compiled as special bytecodes, paired alternately with the number of arguments each takes." newArray at: 24 put: #( #+ 1 #- 1 #< 1 #> 1 #<= 1 #>= 1 #= 1 #~= 1 #* 1 #/ 1 #\\ 1 #@ 1 #bitShift: 1 #// 1 #bitAnd: 1 #bitOr: 1 #at: 1 #at:put: 2 #size 0 #next 0 #nextPut: 1 #atEnd 0 #== 1 #class 0 #blockCopy: 1 #value 0 #value: 1 #do: 1 #new 0 #new: 1 #x 0 #y 0 ). "An array of the 255 Characters in ascii order. Cog inlines table into machine code at: prim so do not regenerate it." newArray at: 25 put: (self specialObjectsArray at: 25). newArray at: 26 put: #mustBeBoolean. newArray at: 27 put: ByteArray. newArray at: 28 put: Process. "An array of up to 31 classes whose instances will have compact headers" newArray at: 29 put: self compactClassesArray. newArray at: 30 put: (self specialObjectsArray at: 30). "(delay Semaphore)" newArray at: 31 put: (self specialObjectsArray at: 31). "(user interrupt Semaphore)" "Entries 32 - 34 unreferenced. Previously these contained prototype instances to be copied for fast initialization" newArray at: 32 put: nil. "was (Float new: 2)" newArray at: 33 put: nil. "was (LargePositiveInteger new: 4)" newArray at: 34 put: nil. "was Point new" newArray at: 35 put: #cannotInterpret:. "Note: This must be fixed once we start using context prototypes (yeah, right)" "(MethodContext new: CompiledMethod fullFrameSize)." newArray at: 36 put: (self specialObjectsArray at: 36). "Is the prototype MethodContext (unused by the VM)" newArray at: 37 put: BlockClosure. "(BlockContext new: CompiledMethod fullFrameSize)." newArray at: 38 put: (self specialObjectsArray at: 38). "Is the prototype BlockContext (unused by the VM)" "array of objects referred to by external code" newArray at: 39 put: (self specialObjectsArray at: 39). "preserve external semaphores" newArray at: 40 put: nil. "Reserved for Mutex in Cog VMs" newArray at: 41 put: nil. "Reserved for a LinkedList instance for overlapped calls in CogMT" "finalization Semaphore" newArray at: 42 put: ((self specialObjectsArray at: 42) ifNil: [Semaphore new]). newArray at: 43 put: LargeNegativeInteger. "External objects for callout. Note: Written so that one can actually completely remove the FFI." newArray at: 44 put: (self at: #ExternalAddress ifAbsent: []). newArray at: 45 put: (self at: #ExternalStructure ifAbsent: []). newArray at: 46 put: (self at: #ExternalData ifAbsent: []). newArray at: 47 put: (self at: #ExternalFunction ifAbsent: []). newArray at: 48 put: (self at: #ExternalLibrary ifAbsent: []). newArray at: 49 put: #aboutToReturn:through:. newArray at: 50 put: #run:with:in:. "51 reserved for immutability message" "newArray at: 51 put: #attemptToAssign:withIndex:." newArray at: 51 put: (self specialObjectsArray at: 51 ifAbsent: []). newArray at: 52 put: #(nil "nil => generic error" #'bad receiver' #'bad argument' #'bad index' #'bad number of arguments' #'inappropriate operation' #'unsupported operation' #'no modification' #'insufficient object memory' #'insufficient C memory' #'not found' #'bad method' #'internal error in named primitive machinery' #'object may move' #'resource limit exceeded' #'object is pinned'). "53 to 55 are for Alien" newArray at: 53 put: (self at: #Alien ifAbsent: []). + newArray at: 54 put: #invokeCallbackContext:. "use invokeCallback:stack:registers:jmpbuf: for old Alien callbacks." - newArray at: 54 put: #invokeCallbackContext::. "use invokeCallback:stack:registers:jmpbuf: for old Alien callbacks." newArray at: 55 put: (self at: #UnsafeAlien ifAbsent: []). "Weak reference finalization" newArray at: 56 put: (self at: #WeakFinalizationList ifAbsent: []). "reserved for foreign callback process" newArray at: 57 put: (self specialObjectsArray at: 57 ifAbsent: []). newArray at: 58 put: #unusedBytecode. "Now replace the interpreter's reference in one atomic operation" self specialObjectsArray becomeForward: newArray !

1 0

Squeak 4.6: System-cwp.754.mcz
by commits＠source.squeak.org 08 Dec '15

08 Dec '15

Colin Putney uploaded a new version of System to project Squeak 4.6: http://source.squeak.org/squeak46/System-cwp.754.mcz ==================== Summary ==================== Name: System-cwp.754 Author: cwp Time: 7 December 2015, 9:14:14.615 pm UUID: 68390bf4-02ab-46d4-b3bc-96bf2723e7d5 Ancestors: System-topa.753 Backported From: System-cwp.782 Backport image segment serialization fixes from trunk =============== Diff against System-topa.753 =============== Item was removed: - ----- Method: Association>>objectForDataStream: (in category '*System-Object Storage-objects from disk') ----- - objectForDataStream: refStrm - | dp | - "I am about to be written on an object file. If I am a known global, write a proxy that will hook up with the same resource in the destination system." - self flag: #environments. - - ^ (Smalltalk globals associationAt: key ifAbsent: [nil]) == self - ifTrue: [dp := DiskProxy global: #Smalltalk selector: #associationOrUndeclaredAt: - args: (Array with: key). - refStrm replace: self with: dp. - dp] - ifFalse: [self]! Item was changed: ----- Method: DiskProxy>>comeFullyUpOnReload: (in category 'i/o') ----- comeFullyUpOnReload: smartRefStream "Internalize myself into a fully alive object after raw loading from a DataStream. (See my class comment.) DataStream will substitute the object from this eval for the DiskProxy." + | globalObj symbol pr nn arrayIndex env | - | globalObj symbol pr nn arrayIndex | - self flag: #environments. symbol := globalObjectName. "See if class is mapped to another name" (smartRefStream respondsTo: #renamed) ifTrue: [ "If in outPointers in an ImageSegment, remember original class name. See mapClass:installIn:. Would be lost otherwise." ((thisContext sender sender sender sender sender sender sender sender receiver class == ImageSegment) and: [ thisContext sender sender sender sender method == (DataStream compiledMethodAt: #readArray)]) ifTrue: [ arrayIndex := (thisContext sender sender sender sender) tempAt: 4. "index var in readArray. Later safer to find i on stack of context." smartRefStream renamedConv at: arrayIndex put: symbol]. "save original name" symbol := smartRefStream renamed at: symbol ifAbsent: [symbol]]. "map" + env := Environment current. + globalObj := env valueOf: symbol ifAbsent: [ - globalObj := Smalltalk at: symbol ifAbsent: [ preSelector == nil & (constructorSelector = #yourself) ifTrue: [ Transcript cr; show: symbol, ' is undeclared.'. + env undeclare: symbol. - (Undeclared includesKey: symbol) ifTrue: [^ Undeclared at: symbol]. - Undeclared at: symbol put: nil. ^ nil]. ^ self error: 'Global "', symbol, '" not found']. ((symbol == #World) and: [Smalltalk isMorphic not]) ifTrue: [ self inform: 'These objects will work better if opened in a Morphic World. Dismiss and reopen all menus.']. preSelector ifNotNil: [ Symbol hasInterned: preSelector ifTrue: [:selector | [globalObj := globalObj perform: selector] on: Error do: [:ex | ex messageText = 'key not found' ifTrue: [^ nil]. ^ ex signal]] ]. symbol == #Project ifTrue: [ (constructorSelector = #fromUrl:) ifTrue: [ nn := (constructorArgs first findTokens: '/') last. nn := (nn findTokens: '.|') first. pr := Project named: nn. ^ pr ifNil: [self] ifNotNil: [pr]]. pr := globalObj perform: constructorSelector withArguments: constructorArgs. ^ pr ifNil: [self] ifNotNil: [pr]]. "keep the Proxy if Project does not exist" constructorSelector ifNil: [^ globalObj]. Symbol hasInterned: constructorSelector ifTrue: [:selector | [^ globalObj perform: selector withArguments: constructorArgs] on: Error do: [:ex | ex messageText = 'key not found' ifTrue: [^ nil]. ^ ex signal] ]. "args not checked against Renamed" ^ nil "was not in proper form"!

1 0

Squeak 4.6: Environments-cwp.58.mcz
by commits＠source.squeak.org 08 Dec '15

08 Dec '15

Colin Putney uploaded a new version of Environments to project Squeak 4.6: http://source.squeak.org/squeak46/Environments-cwp.58.mcz ==================== Summary ==================== Name: Environments-cwp.58 Author: cwp Time: 7 December 2015, 9:00:03.154 pm UUID: d851b52d-ed87-40d7-88d4-d196ea0e09da Ancestors: Environments-cmm.57 Backported From: Environments-cwp.60 Backport image segment serializaton fix from trunk =============== Diff against Environments-cmm.57 =============== Item was changed: ----- Method: Binding>>objectForDataStream: (in category 'as yet unclassified') ----- + objectForDataStream: refStrm + "I am about to be written on an object file. I am a global, so write a proxy that + will hook up with the same resource in the destination system." + + | dp | + dp := DiskProxy + global: #Smalltalk + selector: #associationOrUndeclaredAt: + args: (Array with: key). + refStrm replace: self with: dp. + ^ dp! - objectForDataStream: refStream - "It's not yet clear how serialization should work in the presence of environments" - - self shouldBeImplemented.!

1 0

The Inbox: Morphic-tpr.1057.mcz
by commits＠source.squeak.org 08 Dec '15

08 Dec '15

tim Rowledge uploaded a new version of Morphic to project The Inbox: http://source.squeak.org/inbox/Morphic-tpr.1057.mcz ==================== Summary ==================== Name: Morphic-tpr.1057 Author: tpr Time: 7 December 2015, 8:24:53.359 pm UUID: c685257b-5ee7-49b3-a6b7-5255f28cd47f Ancestors: Morphic-dtl.1056 Add the STWCPW version of cursor gridding to HandMorph. It may very well be better to add two instance variables rather than use a couple of properties to hold the state. There may well be other methods needing changes. To use: start by sending gridTo: grid origin: origin to the hand any subsequent temporaryCursor... will be forced to screeen co-ordinates that are on grid sized steps with an offset of the origin. gridPointRaw provides the frid limited position of the current hand. Older events are *not* gridded - this is one of the places where there may be more to do. =============== Diff against Morphic-dtl.1056 =============== Item was added: + ----- Method: HandMorph>>gridPointRaw (in category 'gridded cursor') ----- + gridPointRaw + "return my latest position gridded" + ^self griddedPoint: self currentEvent position! Item was added: + ----- Method: HandMorph>>gridTo:origin: (in category 'gridded cursor') ----- + gridTo: grid origin: offset + "set a couple of properties to specify gridding for the temporaryCursor; instvars would be nicer" + self setProperty: #gridStep toValue: grid; + setProperty: #gridOffset toValue: offset! Item was added: + ----- Method: HandMorph>>griddedPoint: (in category 'gridded cursor') ----- + griddedPoint: aPoint + "return the equivalent point snapped to the grid, if indeed any gridding is set" + self valueOfProperty: #gridStep ifPresentDo: [:grid| |offset| + offset := self valueOfProperty: #gridOffset ifAbsent: [0@0]. + ^ offset + (aPoint + (grid //2) - offset truncateTo: grid)]. + ^aPoint! Item was changed: ----- Method: HandMorph>>position: (in category 'geometry') ----- position: aPoint "Overridden to align submorph origins to the grid if gridding is on." | adjustedPosition delta box | adjustedPosition := aPoint. + temporaryCursor ifNotNil: [adjustedPosition := (self griddedPoint: adjustedPosition) + temporaryCursorOffset]. - temporaryCursor ifNotNil: [adjustedPosition := adjustedPosition + temporaryCursorOffset]. "Copied from Morph to avoid owner layoutChanged" "Change the position of this morph and and all of its submorphs." delta := adjustedPosition - bounds topLeft. delta isZero ifTrue: [^ self]. "Null change" box := self fullBounds. (delta dotProduct: delta) > 100 ifTrue:[ "e.g., more than 10 pixels moved" self invalidRect: box. self invalidRect: (box translateBy: delta). ] ifFalse:[ self invalidRect: (box merge: (box translateBy: delta)). ]. self privateFullMoveBy: delta. ! Item was added: + ----- Method: HandMorph>>turnOffGridding (in category 'gridded cursor') ----- + turnOffGridding + "remove the gridding properties to stop gridding" + self removeProperty: #gridStep; removeProperty: #gridOffset!

1 0

Sorting Unicode strings
by H. Hirzel 08 Dec '15

08 Dec '15

Hello According to http://www.unicode.org/cldr/charts/27/collation/de.html the German phonebook sort order is a A ä Ä ą̈ Ą̈ ǟ Ǟ ạ̈ Ạ̈ ḁ̈ Ḁ̈ b B c C d D e E f F g G h H i I j J k K l L m M n N o O ö Ö ǫ̈ Ǫ̈ ȫ Ȫ ơ̈ Ơ̈ ợ̈ Ợ̈ ọ̈ Ọ̈ p P q Q r R s S ss ß t T u U ü Ü ǘ Ǘ ǜ Ǜ ǚ Ǚ ų̈ Ų̈ ǖ Ǖ ư̈ Ư̈ ự̈ Ự̈ ụ̈ Ụ̈ ṳ̈ Ṳ̈ ṷ̈ Ṷ̈ ṵ̈ Ṵ̈ v V w W x X y Y z Z I wonder why it looks like this. A lot of characters which never appear in a German text. For Spanish there is 'traditional' and 'standard' http://www.unicode.org/cldr/charts/27/collation/es.html standard a A á Á b B c C d D e E é É f F g G h H i I í Í j J k K l L m M n N ñ Ñ ņ̃ Ņ̃ ṇ̃ Ṇ̃ ṋ̃ Ṋ̃ ṉ̃ Ṉ̃ o O ó Ó p P q Q r R s S t T u U ú Ú ü Ü v V w W x X y Y z Z traditional a A á Á b B c C ch Ch CH cĥ Cĥ CĤ cȟ Cȟ CȞ cḧ Cḧ CḦ cḣ Cḣ CḢ cḩ Cḩ CḨ cḥ Cḥ CḤ cḫ Cḫ CḪ cẖ Cẖ d D e E é É f F g G h H i I í Í j J k K l L ll Ll LL lĺ Lĺ LĹ lľ Lľ LĽ lļ Lļ LĻ lḷ Lḷ LḶ lḹ Lḹ LḸ lḽ Lḽ LḼ lḻ Lḻ LḺ m M n N ñ Ñ ņ̃ Ņ̃ ṇ̃ Ṇ̃ ṋ̃ Ṋ̃ ṉ̃ Ṉ̃ o O ó Ó p P q Q r R s S t T u U ú Ú ü Ü v V w W x X y Y z Z And French is not easily found http://www.unicode.org/cldr/charts/27/collation/index.html or seems to be defined elsewhere http://unicode.org/repos/cldr/tags/release-27/common/collation/fr.xml Suggestions and hints are welcome --Hannes

2 1

Re: [Pharo-dev] Unicode Support
by EuanM 07 Dec '15

07 Dec '15

Hi Henry, To be honest, at some point I'm going to long for the for the much more succinct semantics of healthcare systems and sports scoring and administration systems again. :-) codepoints are any of *either* - the representation of a component of an abstract character, *or* eg. "A" #(0041) as a component of - the sole representation of the whole of an abstract character *or* of - a representation of an abstract character provided for backwards compatibility which is more properly represented by a series of codepoints representing a composed character e.g. The "A" #(0041) as a codepoint can be: the sole representation of the whole of an abstract character "A" #(0041) The representation of a component of the composed (i.e. preferred) version of the abstract character Å #(0041 030a) Å (#00C5) represents one valid compatibility form of the abstract character Å which is most properly represented by #(0041 030a). Å (#212b) also represents one valid compatibility form of the abstract character Å which is most properly represented by #(0041 030a). With any luck, this satisfies both our semantic understandings of the concept of "codepoint" Would you agree with that? In Unicode, codepoints are *NOT* an abstract numerical representation of a text character. At least not as we generally understand the term "text character" from our experience of non-Unicode character mappings. codepoints represent "*encoded characters*" and "a *text element* ... is represented by a sequence of one or more codepoints". (And the term "text element" is deliberately left undefined in the Unicode standard) Individual codepoints are very often *not* the encoded form of an abstract character that we are interested in. Unless we are communicating to or from another system (Which in some cases is the Smalltalk ByteString class) i.e. in other words *Some* individual codepoints *may* be a representation of a specific *abstract character*, but only in special cases. The general case in Unicode is that Unicode defines (a) representation(s) of a Unicode *abstract character*. The Unicode standard representation of an abstract character is a composed sequence of codepoints, where in some cases that sequence is as short as 1 codepoint. In other cases, Unicode has a compatibility alias of a single codepoint which is *also* a representation of an abstract character There are some cases where an abstract character can be represented by more than one single-codepoint compatibility codepoint. Cheers, Euan On 7 December 2015 at 11:11, Henrik Johansen <henrik.s.johansen(a)veloxit.no> wrote: > >> On 07 Dec 2015, at 11:51 , EuanM <euanmee(a)gmail.com> wrote: >> >> And indeed, in principle. >> >> On 7 December 2015 at 10:51, EuanM <euanmee(a)gmail.com> wrote: >>> Verifying assumptions is the key reason why you should documents like >>> this out for review. >>> >>> Sven - >>> >>> I'm confident I understand the use of UTF-8 in principal. > > I can only second Sven's sentiment that you need to better differentiate code points (an abstract numerical representation of a character, where a set of such mappings > define a charset, such as Unicode), and character encoding forms. (which are how code points are represented in bytes by a defined process such as UTF-8, UTF-16 etc). > > I know you'll probably think I'm arguing semantics again, but these are *important* semantics ;) > > Cheers, > Henry

3 3

Unicode Support
by EuanM 07 Dec '15

07 Dec '15

I'm currently groping my way to seeing how feature-complete our Unicode support is. I am doing this to establish what still needs to be done to provide full Unicode support. This seems to me to be an area where it would be best to write it once, and then have the same codebase incorporated into the Smalltalks that most share a common ancestry. I am keen to get: equality-testing for strings; sortability for strings which have ligatures and diacritic characters; and correct round-tripping of data. Call to action: ========== If you have comments on these proposals - such as "but we already have that facility" or "the reason we do not have these facilities is because they are dog-slow" - please let me know them. If you would like to help out, please let me know. If you have Unicode experience and expertise, and would like to be, or would be willing to be, in the 'council of experts' for this project, please let me know. If you have comments or ideas on anything mentioned in this email In the first instance, the initiative's website will be: http://smalltalk.uk.to/unicode.html I have created a SqueakSource.com project called UnicodeSupport I want to avoid re-inventing any facilities which already exist. Except where they prevent us reaching the goals of: - sortable UTF8 strings - sortable UTF16 strings - equivalence testing of 2 UTF8 strings - equivalence testing of 2 UTF16 strings - round-tripping UTF8 strings through Smalltalk - roundtripping UTF16 strings through Smalltalk. As I understand it, we have limited Unicode support atm. Current state of play =============== ByteString gets converted to WideString when need is automagically detected. Is there anything else that currently exists? Definition of Terms ============== A quick definition of terms before I go any further: Standard terms from the Unicode standard =============================== a compatibility character : an additional encoding of a *normal* character, for compatibility and round-trip conversion purposes. For instance, a 1-byte encoding of a Latin character with a diacritic. Made-up terms ============ a convenience codepoint : a single codepoint which represents an item that is also encoded as a string of codepoints. (I tend to use the terms compatibility character and compatibility codepoint interchangably. The standard only refers to them as compatibility characters. However, the standard is determined to emphasise that characters are abstract and that codepoints are concrete. So I think it is often more useful and productive to think of compatibility or convenience codepoints). a composed character : a character made up of several codepoints Unicode encoding explained ===================== A convenience codepoint can therefore be thought of as a code point used for a character which also has a composed form. The way Unicode works is that sometimes you can encode a character in one byte, sometimes not. Sometimes you can encode it in two bytes, sometimes not. You can therefore have a long stream of ASCII which is single-byte Unicode. If there is an occasional Cyrillic or Greek character in the stream, it would be represented either by a compatibility character or by a multi-byte combination. Using compatibility characters can prevent proper sorting and equivalence testing. Using "pure" Unicode, ie. "normal encodings", can cause compatibility and round-tripping probelms. Although avoiding them can *also* cause compatibility issues and round-tripping problems. Currently my thinking is: a Utf8String class an Ordered collection, with 1 byte characters as the modal element, but short arrays of wider strings where necessary a Utf16String class an Ordered collection, with 2 byte characters as the modal element, but short arrays of wider strings beginning with a 2-byte endianness indicator. Utf8Strings sometimes need to be sortable, and sometimes need to be compatible. So my thinking is that Utf8String will contain convenience codepoints, for round-tripping. And where there are multiple convenience codepoints for a character, that it standardises on one. And that there is a Utf8SortableString which uses *only* normal characters. We then need methods to convert between the two. aUtf8String asUtf8SortableString and aUtf8SortableString asUtf8String Sort orders are culture and context dependent - Sweden and Germany have different sort orders for the same diacritic-ed characters. Some countries have one order in general usage, and another for specific usages, such as phone directories (e.g. UK and France) Similarly for Utf16 : Utf16String and Utf16SortableString and conversion methods A list of sorted words would be a SortedCollection, and there could be pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, seOrder, ukOrder, etc along the lines of aListOfWords := SortedCollection sortBlock: deOrder If a word is either a Utf8SortableString, or a well-formed Utf8String, then we can perform equivalence testing on them trivially. To make sure a Utf8String is well formed, we would need to have a way of cleaning up any convenience codepoints which were valid, but which were for a character which has multiple equally-valid alternative convenience codepoints, and for which the string currently had the "wrong" convenience codepoint. (i.e for any character with valid alternative convenience codepoints, we would choose one to be in the well-formed Utf8String, and we would need a method for cleaning the alternative convenience codepoints out of the string, and replacing them with the chosen approved convenience codepoint. aUtf8String cleanUtf8String With WideString, a lot of the issues disappear - except round-tripping(although I'm sure I have seen something recently about 4-byte strings that also have an additional bit. Which would make some Unicode characters 5-bytes long.) (I'm starting to zone out now - if I've overlooked anything - obvious, subtle, or somewhere in between, please let me know) Cheers, Euan

8 12

Re: [Unicode] collation sequences (Re: [squeak-dev] Unicode Support)
by EuanM 07 Dec '15

07 Dec '15

Hannes, The Unicode standard provide compatibility codepoints for compatibility purposes and prefers all characters to be represented composed form - as that way they are comparable and sortable. (Some composed characters have *more than one* compatibility codepoint. The canonical example is the composed character #(0041 030a) which can be represented by EITHER the compatibility codepoint #(00c5) "Latin Capital Letter A with Ring" above OR by #(212b) "Angstrom sign" ) On 7 December 2015 at 08:17, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: > On 12/7/15, EuanM <euanmee(a)gmail.com> wrote: >> My current thinking for collation sequences: >> >> All strings being collated have had all compatibility codepoints >> expanded into composed sequences. > > What does the Unicode manual suggest? (www.unicode.org reference?) > > > >> >> Strings containing composed sequences and UTF >> -8 strings containing multi-byte characters have these represented by >> a very short ordered collection in place of the single Byte of a >> ByteString. >> >> When we sort characters, words or phrases of strings that contain >> zero compatibility codepoints, we simply pull a pre-defined sortBlock >> out of a Dictionary of pre-defined sortBlocks >> >> >> aDictionaryOfSorts at: ukPhoneBook put: aSortBlock >> or >> aDictionaryOfSorts at: ukPhoneBook put: '[aString representing the >> code of a sortBlock]' . >> >> ASortedCollectionOfUtf8Strings sortBlock: aDictionaryOfSorts at: >> ukPhoneBook >> >> - or some actual working code! :-) >> > > Yes, focusing on this is a real need. > > >> >> On 6 December 2015 at 15:14, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>> P.S. The 30-bit value for each character in Squeak/Pharo (if necessary >>> together with an additional language tag) is a potentially very >>> capable infrastructure. Not really used much at the moment. >>> >>> The challenge is to to make _existing_ Unicode-know-how defined >>> elsewhere (e.g. www.unicode.org) available in Squeak/Pharo/Cuis. >>> >>> Most simple cases would be to start with collation sequences in >>> Italian, French, German, Spanish, Portugese. Later on more complex >>> cases like Arabic. >>> >>> --HH >>> >>> On 12/6/15, H. Hirzel <hannes.hirzel(a)gmail.com> wrote: >>>> Hi Euan, >>>> >>>> On 12/4/15, EuanM <euanmee(a)gmail.com> wrote: >>>>> I'm currently groping my way to seeing how feature-complete our >>>>> Unicode support is. I am doing this to establish what still needs to >>>>> be done to provide full Unicode support. >>>>> >>>>> This seems to me to be an area where it would be best to write it >>>>> once, and then have the same codebase incorporated into the Smalltalks >>>>> that most share a common ancestry. >>>>> >>>>> I am keen to get: equality-testing for strings; sortability for >>>>> strings which have ligatures and diacritic characters; and correct >>>>> round-tripping of data. >>>> >>>> These goals call for a package with SUnit tests which you then can >>>> run on all platforms. This will be a tool to evalutate platforms for >>>> the level of Unicode support. >>>> As mentioned in the thread I would focus on UTF8 only as far as >>>> external files are concerned. >>>> I.E. the test package writes a sample UFT8 file and then reads it to >>>> do the various tests. >>>> I have started doing this for Squeak and Cuis some time ago with a few >>>> tests. >>>> >>>> I am interested in sortability. Round-tripping is fine if you go for >>>> UTF8. >>>> Important of course is which languages you think the package should >>>> work. Some of them are easy, some not. >>>> >>>> This afternoon I did some updates on the Squeak wiki >>>> http://wiki.squeak.org/squeak/recent >>>> >>>> --Hannes >>>> >>>>> >>>>> Call to action: >>>>> ========== >>>>> >>>>> If you have comments on these proposals - such as "but we already have >>>>> that facility" or "the reason we do not have these facilities is >>>>> because they are dog-slow" - please let me know them. >>>>> >>>>> If you would like to help out, please let me know. >>>>> >>>>> If you have Unicode experience and expertise, and would like to be, or >>>>> would be willing to be, in the 'council of experts' for this project, >>>>> please let me know. >>>>> >>>>> If you have comments or ideas on anything mentioned in this email >>>>> >>>>> In the first instance, the initiative's website will be: >>>>> http://smalltalk.uk.to/unicode.html >>>>> >>>>> I have created a SqueakSource.com project called UnicodeSupport >>>>> >>>>> I want to avoid re-inventing any facilities which already exist. >>>>> Except where they prevent us reaching the goals of: >>>>> - sortable UTF8 strings >>>>> - sortable UTF16 strings >>>>> - equivalence testing of 2 UTF8 strings >>>>> - equivalence testing of 2 UTF16 strings >>>>> - round-tripping UTF8 strings through Smalltalk >>>>> - roundtripping UTF16 strings through Smalltalk. >>>>> As I understand it, we have limited Unicode support atm. >>>>> >>>>> Current state of play >>>>> =============== >>>>> ByteString gets converted to WideString when need is automagically >>>>> detected. >>>>> >>>>> Is there anything else that currently exists? >>>>> >>>>> Definition of Terms >>>>> ============== >>>>> A quick definition of terms before I go any further: >>>>> >>>>> Standard terms from the Unicode standard >>>>> =============================== >>>>> a compatibility character : an additional encoding of a *normal* >>>>> character, for compatibility and round-trip conversion purposes. For >>>>> instance, a 1-byte encoding of a Latin character with a diacritic. >>>>> >>>>> Made-up terms >>>>> ============ >>>>> a convenience codepoint : a single codepoint which represents an item >>>>> that is also encoded as a string of codepoints. >>>>> >>>>> (I tend to use the terms compatibility character and compatibility >>>>> codepoint interchangably. The standard only refers to them as >>>>> compatibility characters. However, the standard is determined to >>>>> emphasise that characters are abstract and that codepoints are >>>>> concrete. So I think it is often more useful and productive to think >>>>> of compatibility or convenience codepoints). >>>>> >>>>> a composed character : a character made up of several codepoints >>>>> >>>>> Unicode encoding explained >>>>> ===================== >>>>> A convenience codepoint can therefore be thought of as a code point >>>>> used for a character which also has a composed form. >>>>> >>>>> The way Unicode works is that sometimes you can encode a character in >>>>> one byte, sometimes not. Sometimes you can encode it in two bytes, >>>>> sometimes not. >>>>> >>>>> You can therefore have a long stream of ASCII which is single-byte >>>>> Unicode. If there is an occasional Cyrillic or Greek character in the >>>>> stream, it would be represented either by a compatibility character or >>>>> by a multi-byte combination. >>>>> >>>>> Using compatibility characters can prevent proper sorting and >>>>> equivalence testing. >>>>> >>>>> Using "pure" Unicode, ie. "normal encodings", can cause compatibility >>>>> and round-tripping probelms. Although avoiding them can *also* cause >>>>> compatibility issues and round-tripping problems. >>>>> >>>>> Currently my thinking is: >>>>> >>>>> a Utf8String class >>>>> an Ordered collection, with 1 byte characters as the modal element, >>>>> but short arrays of wider strings where necessary >>>>> a Utf16String class >>>>> an Ordered collection, with 2 byte characters as the modal element, >>>>> but short arrays of wider strings >>>>> beginning with a 2-byte endianness indicator. >>>>> >>>>> Utf8Strings sometimes need to be sortable, and sometimes need to be >>>>> compatible. >>>>> >>>>> So my thinking is that Utf8String will contain convenience codepoints, >>>>> for round-tripping. And where there are multiple convenience >>>>> codepoints for a character, that it standardises on one. >>>>> >>>>> And that there is a Utf8SortableString which uses *only* normal >>>>> characters. >>>>> >>>>> We then need methods to convert between the two. >>>>> >>>>> aUtf8String asUtf8SortableString >>>>> >>>>> and >>>>> >>>>> aUtf8SortableString asUtf8String >>>>> >>>>> >>>>> Sort orders are culture and context dependent - Sweden and Germany >>>>> have different sort orders for the same diacritic-ed characters. Some >>>>> countries have one order in general usage, and another for specific >>>>> usages, such as phone directories (e.g. UK and France) >>>>> >>>>> Similarly for Utf16 : Utf16String and Utf16SortableString and >>>>> conversion methods >>>>> >>>>> A list of sorted words would be a SortedCollection, and there could be >>>>> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder, >>>>> seOrder, ukOrder, etc >>>>> >>>>> along the lines of >>>>> aListOfWords := SortedCollection sortBlock: deOrder >>>>> >>>>> If a word is either a Utf8SortableString, or a well-formed Utf8String, >>>>> then we can perform equivalence testing on them trivially. >>>>> >>>>> To make sure a Utf8String is well formed, we would need to have a way >>>>> of cleaning up any convenience codepoints which were valid, but which >>>>> were for a character which has multiple equally-valid alternative >>>>> convenience codepoints, and for which the string currently had the >>>>> "wrong" convenience codepoint. (i.e for any character with valid >>>>> alternative convenience codepoints, we would choose one to be in the >>>>> well-formed Utf8String, and we would need a method for cleaning the >>>>> alternative convenience codepoints out of the string, and replacing >>>>> them with the chosen approved convenience codepoint. >>>>> >>>>> aUtf8String cleanUtf8String >>>>> >>>>> With WideString, a lot of the issues disappear - except >>>>> round-tripping(although I'm sure I have seen something recently about >>>>> 4-byte strings that also have an additional bit. Which would make >>>>> some Unicode characters 5-bytes long.) >>>>> >>>>> >>>>> (I'm starting to zone out now - if I've overlooked anything - obvious, >>>>> subtle, or somewhere in between, please let me know) >>>>> >>>>> Cheers, >>>>> Euan >>>>> >>>>> >>>> >>

2 1

← Newer
1
...
10
11
12
13
14
15
16
17
Older →

Jump to page: