On Sun, 22 Dec 2013, Bob Arning wrote:
David,
Here is an alternative that tries to print the item rather than delete it. You can revert the previous fix, if you like.
The problem with this solution is that it will print a string using UTF-8 encoding, but Seaside will return a XHTML page with ISO-8859-1 endocing. There are two ways to solve it: 1: use NCRs[1] to encode the names. This might be a problem, because one has to make sure that seaside won't encode the & characters. 2: change the default character encoding to UTF-8, which might lead to other problems.
Levente
[1]https://en.wikipedia.org/wiki/Numeric_character_reference
Cheers, Bob
'From Squeak3.11alpha of 6 January 2010 [latest update: #8824] on 22 December 2013 at 10:32:55 am'!
!WideString methodsFor: 'converting' stamp: 'raa 12/22/2013 10:31'! printHtmlOn: aStream
self squeakToUtf8 printHtmlOn: aStream! !
On 12/22/13 12:03 AM, David T. Lewis wrote:
Bob,
This is absolutely great, thank you so much for the help. I just loaded your patch into the running image, and the http://squeaksource.com web interface is back to normal again.
Dave
On Sat, Dec 21, 2013 at 11:25:59PM -0500, Bob Arning wrote:
David,
The short answer is that two users (St?ane Munioz St?ane Munioz) have wide characters in their name and this does the damage. The method below is a quick hack to filter these 2 out of the list and all seems good.
Cheers, Bob
'From Squeak3.11alpha of 6 January 2010 [latest update: #8824] on 21 December 2013 at 11:21:03 pm'!
!SSHome methodsFor: 'statistics'! membersNew | t1 | t1 := self statistics membersNew. ^ t1 reject: [:t2 | t2 asString anySatisfy: [:t3 | t3 asciiValue > 255]]! !
On 12/21/13 9:25 PM, David T. Lewis wrote:
On Sat, Dec 21, 2013 at 07:41:08PM -0500, Bob Arning wrote:
If you click one of the links on the first page, the next page looks good. If you then go back, the css from page 2 makes page1 look a little better. So, the breakage is limited. Any way to share the image?
Thanks Bob,
A copy of the image is at http://box2.squeak.org/ss_image_for_bob.zip
The zip is encripted, I will send the password to you in a separate email.
Dave
I think that we probably have a few issues related to wide strings in the SqueakSource code, and these issues certainly will effect source.squeak.org in the same way that we have seen on squeaksource.com. The only difference being that we do not happen to have any author names with multibyte characters registered on source.squeak.org at the moment.
When I was originally loading the old SqueakSource onto our squeak.org servers, I found some problems with the image updating its repository from disk, and at the time I chose to work around them manually in order to get the system up and running. But there seem to be places where the identity of an author is saved in the repository (in the image, not on disk), and stored with possibly different encoding in the MCZ file names on disk, and may be stored with yet another possibly different encoding internally within the MCZ file.
The good news is that the squeaksource.com files and image give us enough real life data that we should be able to locate the problem cases and think about how to handle them properly. For example, the ss.log file shows evidence of continuing problems related to six specific files:
2013-12-21T18:39:10.089+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.53.mcz 2013-12-21T18:39:11.353+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.52.mcz 2013-12-21T18:39:11.602+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.55.mcz 2013-12-21T18:39:12.884+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.66.mcz 2013-12-21T18:39:14.193+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.54.mcz 2013-12-21T18:39:15.45+00:00 RECOVERING FelTimetable/Seaside2.8a1-M·Sa.49.mcz
So some follow up is needed. But maybe not today, for now I'm just happy to have the site running again :-)
Dave
On Sun, Dec 22, 2013 at 05:31:13PM +0100, Levente Uzonyi wrote:
On Sun, 22 Dec 2013, Bob Arning wrote:
David,
Here is an alternative that tries to print the item rather than delete it. You can revert the previous fix, if you like.
The problem with this solution is that it will print a string using UTF-8 encoding, but Seaside will return a XHTML page with ISO-8859-1 endocing. There are two ways to solve it: 1: use NCRs[1] to encode the names. This might be a problem, because one has to make sure that seaside won't encode the & characters. 2: change the default character encoding to UTF-8, which might lead to other problems.
Levente
[1]https://en.wikipedia.org/wiki/Numeric_character_reference
Cheers, Bob
'From Squeak3.11alpha of 6 January 2010 [latest update: #8824] on 22 December 2013 at 10:32:55 am'!
!WideString methodsFor: 'converting' stamp: 'raa 12/22/2013 10:31'! printHtmlOn: aStream
??? self squeakToUtf8 printHtmlOn: aStream! !
On 12/22/13 12:03 AM, David T. Lewis wrote:
Bob,
This is absolutely great, thank you so much for the help. I just loaded your patch into the running image, and the http://squeaksource.com web interface is back to normal again.
Dave
On Sat, Dec 21, 2013 at 11:25:59PM -0500, Bob Arning wrote:
David,
The short answer is that two users (St?ane Munioz St?ane Munioz) have wide characters in their name and this does the damage. The method below is a quick hack to filter these 2 out of the list and all seems good.
Cheers, Bob
'From Squeak3.11alpha of 6 January 2010 [latest update: #8824] on 21 December 2013 at 11:21:03 pm'!
!SSHome methodsFor: 'statistics'! membersNew | t1 | t1 := self statistics membersNew. ^ t1 reject: [:t2 | t2 asString anySatisfy: [:t3 | t3 asciiValue > 255]]! !
On 12/21/13 9:25 PM, David T. Lewis wrote:
On Sat, Dec 21, 2013 at 07:41:08PM -0500, Bob Arning wrote:
If you click one of the links on the first page, the next page looks good. If you then go back, the css from page 2 makes page1 look a little better. So, the breakage is limited. Any way to share the image?
Thanks Bob,
A copy of the image is at http://box2.squeak.org/ss_image_for_bob.zip
The zip is encripted, I will send the password to you in a separate email.
Dave
On Sun, Dec 22, 2013 at 5:50 PM, David T. Lewis lewis@mail.msen.com wrote:
I think that we probably have a few issues related to wide strings in the SqueakSource code, and these issues certainly will effect source.squeak.org in the same way that we have seen on squeaksource.com. The only difference being that we do not happen to have any author names with multibyte characters registered on source.squeak.org at the moment.
When I was originally loading the old SqueakSource onto our squeak.org servers, I found some problems with the image updating its repository from disk, and at the time I chose to work around them manually in order to get the system up and running. But there seem to be places where the identity of an author is saved in the repository (in the image, not on disk), and stored with possibly different encoding in the MCZ file names on disk, and may be stored with yet another possibly different encoding internally within the MCZ file.
The good news is that the squeaksource.com files and image give us enough real life data that we should be able to locate the problem cases and think about how to handle them properly. For example, the ss.log file shows evidence of continuing problems related to six specific files:
2013-12-21T18:39:10.089+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.53.mcz 2013-12-21T18:39:11.353+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.52.mcz 2013-12-21T18:39:11.602+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.55.mcz 2013-12-21T18:39:12.884+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.66.mcz 2013-12-21T18:39:14.193+00:00 RECOVERING FelTimetable/FelTimetable-M·Sa.54.mcz 2013-12-21T18:39:15.45+00:00 RECOVERING FelTimetable/Seaside2.8a1-M·Sa.49.mcz
So some follow up is needed. But maybe not today, for now I'm just happy to have the site running again :-)
Long story short it's messy and your options are kinda limited. The problems stem from the fact that the Seaside version of SqueakSource is very old (probably a decade by now). It's unmaintained and missing all the Unicode fixes that went in over the past years. There are newer versions of SqueakSource available [1] [2] [3] that work with newer versions of Seaside. The trouble however is migrating (you'll likely have to migrate to a newer version of Squeak as well). But I'm sure all the advocates of images and objects will be eager lend you a helping hand.
Now regarding encoding there are two things you need decide. What should be the internal encoding in the image and what should be the external encoding on the web page. If they are different some transcoding has to happen for both input and output. In Seaside 3.x this is quite easy to do, in Seaside 2.6 not so much. The webpage currently seems to use iso-8859-1 as indicated in the XML preamble (there is no HTTP header). I assume (without being sure) that the internal encoding is Squeak/MacRoman. Which brings us to the question how St鰨ane Munioz ended up in the image. Can you confirm that his name is a WideString and 鰨 is a single instance of Character? The obvious choice at this point would be to go for utf-8 external and Squeak internal. The easiest way to do this would be to use WAKomEncoded but I don't think this is even present in this version of Seaside. Remember you'll have to encode all the output in decode all the input. For example when I search for "Munioz" under "Members" still only half the page renders. There is one downside to this approach though and that is that you'll end up having WideStrings in the image. WideString has a bad reputation of being slow and buggy. Seaside 3.x helps a bit because the response would be encoded on the fly and therefore avoid a huge WideString response buffer. To avoid this you could use utf-8 internally but that breaks all length related methods and you'll have to pay attention when interacting with external systems (eg. file system).
General itmes: One of the optimizations we never had time to implement was installing mod_xsendfile [4]. Serving all the MCZ files through the image is very inefficient and puts unnecessary pressure on the image. We can't do it directly with Apache because we have to do an authentication check first. mod_xsendfile would allow the image to tell Apache which file to serve.
From time to time the image would lock up completely. We applied
several patches that were supposed to make Semaphore thread-safe but the issue never fully went away. Some people said this was because SqueakSource was never designed to handle this load. I don't understand this argument, even if this is the case that should just make the image slow, not lock it up.
[1] http://www.squeaksource.com/ss2.html [2] http://www.squeaksource.com/squeaksource3.html [3] http://ss3.gemstone.com/ss/ss3.html [4] https://tn123.org/mod_xsendfile/
Cheers Philippe
box-admins@lists.squeakfoundation.org
-
David T. Lewis -
Levente Uzonyi -
Philippe Marschall