On 02.09.2009, at 07:28, Andreas Raab wrote:
Hi Bert -
I figured it out, but you won't like it The problem comes from a combination of things going wrong. First, you are right, there are non-Latin characters in the source. This causes the MCWriter to silently go WideString when it writes source.st. The resulting WideString gets passed into ZipArchive which compresses it in chunks of 4k. The funny thing is that when you pull 4k chunks out of a WideString it reduces the result to ByteString again if it can fit into Latin1. Meaning that only those definitions that happen to fall into the same 4k chunk that containing a non-Latin character get screwed up (excuse me for a second while I walk out and shoot myself).
Ah, feeling better now. This is why nobody ever noticed it, because it won't affect all of the stuff and since MC is reasonably smart and doesn't need the source too often, screw-ups of the source do not get noticed.
I think there is a solution though, namely having the writer check whether whether the source is wide and if so use utf-8 instead. The big issue is backwards compatibility though. I can see three approaches:
- Write a BOM marker in front of any UTF8 encoded source.st file.
This will work for any Monticello version which is aware of the BOM; for the others YMMV (it depends on whether you're on 3.8 or later - it *should* be okay for those but I haven't tested).
- Assume all source as UTF8 all the time and allow conversion
errors to pass through assuming Latin-1. This will work both ways (older Monticello's would would get multiple characters in some situations but be otherwise unaffected) at the cost of not detecting possibly incorrect encodings in the file (which isn't a terrible choice since the zip file has a CRC).
- Write two versions of the source, one in snapshot/source one in
snapshot.utf8/source. Works both ways, too at the cost of doubling disk space requirements.
One thing to keep in mind here is that MCDs may only work with #2 unless the servers get updated. I think we should also consult with other MC users to ensure future compatibility. FWIW, my vote is with option #2.
Cheers,
- Andreas
Yes, go UTF-8. This is precisely one of the backwards compatibility problems UTF-8 was designed to work around. In fact I had thought we did this already, must be an omission in our MC version.
- Bert -