So, this (before the change) is one of the failing tests today in trunk. With this change (which follows the lead of another test in this class), we can see that reading chunks with lineEndConvention set is completely ignored - in all cases, it returns the raw data unchanged.
Should the text be changed when reading in chunk data? The method #nextChunk calls UTF8TextConverter #nextChunkFromStream:, which calls MultiByteFileStream #basicUpTo:, which does a raw upTo: on the stream. In other words, it explicitly ignores any conversion.
Which path is correct, please?
Also, could $! existing inside of an encoded UTF8 character? It is looking for raw bytes int eh stream that match change character; I'm not familiar enough with UTF8 to know if that it is possible that a second (or third) byte in a UTF8 character could match it, and hence doing a #basicUpTo: would most definitely be wrong.
-cbc
On Sat, Aug 16, 2014 at 6:00 PM, commits@source.squeak.org wrote:
A new version of MultilingualTests was added to project The Inbox: http://source.squeak.org/inbox/MultilingualTests-cbc.19.mcz
==================== Summary ====================
Name: MultilingualTests-cbc.19 Author: cbc Time: 16 August 2014, 6:00:20.609 pm UUID: b3ccb0dc-5adc-3a4f-b722-98deab3ae9be Ancestors: MultilingualTests-fbs.18
MultiByteFileStream>>testLineEndingChunk is failing on Windows platforms (maybe others). Tweaked test to show which of the 4 is failing.
=============== Diff against MultilingualTests-fbs.18 ===============
Item was changed: ----- Method: MultiByteFileStreamTest>>testLineEndingChunk (in category 'testing') ----- testLineEndingChunk
| failures |
fileName := 'foolinend.txt'.
failures := OrderedCollection new. MultiByteFileStream forceNewFileNamed: fileName do: [ :file | file wantsLineEndConversion: false; nextPutAll: 'line 1'; cr; nextPutAll: 'line 2'; crlf; nextPutAll: 'line 3'; lf; nextPutAll: 'line 4'; nextPut: $!! ]. { {#cr. 'line 1' , String cr , 'line 2' , String cr , 'line3' , String cr , 'line 4'}. {#lf. 'line 1' , String cr , 'line 2' , String cr , 'line 3' , String cr , 'line 4'}. {#crlf. 'line 1' , String cr , 'line 2' , String cr , 'line 3' , String cr , 'line 4'}. {nil. 'line 1' , String cr , 'line 2' , String crlf , 'line 3' , String lf , 'line 4'} } do: [:lineEndingResult |
MultiByteFileStream oldFileNamed: fileName do: [ :file | |actual |
MultiByteFileStream oldFileNamed: fileName do: [ :file | file lineEndConvention: lineEndingResult first.
lineEndingResult last = (actual := file nextChunk)ifFalse: [
failures add: (lineEndingResult copyWith:actual).
].] ].self assert: failures isEmpty!
self assert: lineEndingResult last equals: filenextChunk ] ]!