Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
source.squeak.org looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi leves@elte.hu wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
That's probably because someone has registered with a name containing wide characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi leves@elte.hu wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing wide characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi leves@elte.hu wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing
wide
characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi leves@elte.hu wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so
hopefully
a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
It’s just a bug. SS is just not converting wide strings to UTF-8.
If you have time to move SS to a new image, sure. But otherwise, fixing this bug is probably less work.
- Bert -
On 01.12.2015, at 16:40, Chris Muller ma.chris.m@gmail.com wrote:
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis <lewis@mail.msen.com mailto:lewis@mail.msen.com> wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing wide characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org http://source.squeak.org/ home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com http://squeaksource.com/ from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org http://source.squeak.org/ image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org http://source.squeak.org/ :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org http://source.squeak.org/ looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi <leves@elte.hu mailto:leves@elte.hu> wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
IIRC (based on information from Tobias???) it's a Seaside related issue that has been fixed in some newer version of something or other. We should probably take this discussion to squeak-dev.
Dave
Itâs just a bug. SS is just not converting wide strings to UTF-8.
If you have time to move SS to a new image, sure. But otherwise, fixing this bug is probably less work.
- Bert -
On 01.12.2015, at 16:40, Chris Muller ma.chris.m@gmail.com wrote:
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis <lewis@mail.msen.com mailto:lewis@mail.msen.com> wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing
wide
characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org http://source.squeak.org/ home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com http://squeaksource.com/ from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org http://source.squeak.org/ image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org http://source.squeak.org/ :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org http://source.squeak.org/ looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi <leves@elte.hu
mailto:leves@elte.hu> wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so
hopefully
a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only
they
can request a restart. So please ask them to do so.
Thanks, Levente
On 01.12.2015, at 20:05, David T. Lewis lewis@mail.msen.com wrote:
IIRC (based on information from Tobias???) it's a Seaside related issue that has been fixed in some newer version of something or other. We should probably take this discussion to squeak-dev.
I'm innocent. Or at least I don't know off head the problem at hand, save the solution…
Best regards -Tobias
Dave
It’s just a bug. SS is just not converting wide strings to UTF-8.
If you have time to move SS to a new image, sure. But otherwise, fixing this bug is probably less work.
- Bert -
On 01.12.2015, at 16:40, Chris Muller ma.chris.m@gmail.com wrote:
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis <lewis@mail.msen.com mailto:lewis@mail.msen.com> wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing
wide
characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org http://source.squeak.org/ home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com http://squeaksource.com/ from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org http://source.squeak.org/ image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org http://source.squeak.org/ :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org http://source.squeak.org/ looks weird.
- Bert -
mailto:leves@elte.hu> wrote: hopefully they
You were the one who set up a newer image with Magma backend on box4, though that image hasn't synchronized the list of users, so it's web interface still works. I assume, it would break, too.
Levente
On Tue, 1 Dec 2015, Chris Muller wrote:
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis lewis@mail.msen.com wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote: > That's probably because someone has registered with a name containing wide > characters, so the whole page is served as a WideString. This bug has > happened a few times before, and I don't remember what the > temporary solution was, but I think Dave knows. > > Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed. I see the same thing on squeaksource.com from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page). If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending WideString and patching it to be displayable. The good news is that we are adding a more diverse range of developers to source.squeak.org :-) Dave > > On Tue, 1 Dec 2015, Bert Freudenberg wrote: > > >source.squeak.org looks weird. > > > >- Bert - > > > > > > > >>On 01.12.2015, at 00:12, Levente Uzonyi <leves@elte.hu> wrote: > >> > >>Thanks whoever requested the restart. The machine seems to have four > >>times as much RAM as before. I've hopefully restarted all services. > >>Should anything be missing, let me know! > >> > >>Levente > >> > >>On Mon, 30 Nov 2015, Levente Uzonyi wrote: > >> > >>>Hi All, > >>> > >>>box4 seems to have been down since yesterday. Box3 is fine, so hopefully > >>>a restart will bring box4 back up. > >>>The problem is that only the SFC has access to the box, and only they > >>>can request a restart. > >>>So please ask them to do so. > >>> > >>>Thanks, > >>>Levente > >>> > > > >
It is running a much newer image (4.5 instead of 3.11), I don't know if it can support the special chars or not, but as I went to test it, I found teh cause of the box4 crash, and a number of other problems:
- my own "repcopy" script which is running on an hourly cron is getting some error, but leaving the process (squeak iamge) running. My bet is box4 ran out of memory and crashed. I just killed about 50 of those processes. The script redirects stdout and stderr to "repcopy.log" and "repcopy.err", respectively. However, upon the next scheduled run, I found the process hung again but those two files were empty. However, there was a SqueakDebug.log file...
- The SqueakDebug.log file shows *Squeak-Version-kfr.4712* is triggering the Warning, "About to serialize empty mcz". This version has a comment of, "Messed up last version, try again", and so is a piece of litter in our ancestry and trunk. I again ask y'all to support me on this littering issue and help me address it with community. Please!
- The weird thing is, the script is wrapped in a Smalltalk>>#run:, which handles all Warnings by resuming them. So, I don't know why it is halting..
As for production with the special-char:
- the object model for production source.squeak.org is stored in file
/srv/box2/home/squeaksource/ss/data.obj
but it is currently timestamped April 20th, 2015. So any changes to that model since that time, including the addition of Stephan and Patrick are probably only in the production image (/srv/box2/home/squeaksource/Squeak3.11-8824-SS.image).
- I have just downloaded that image /srv/box2/home/squeaksource/Squeak3.11-8824-SS.image and exported a new "data.obj" file.
Next I will make a new method in ss-deploy.image to refresh the Magma db with the model from data.obj without having to reload all the versions. I will then know whether it can handle the special characters in people's names..
On Tue, Dec 1, 2015 at 2:35 PM, Levente Uzonyi leves@caesar.elte.hu wrote:
You were the one who set up a newer image with Magma backend on box4, though that image hasn't synchronized the list of users, so it's web interface still works. I assume, it would break, too.
Levente
On Tue, 1 Dec 2015, Chris Muller wrote:
Would a newer image and VM solve this?
Why do we continue to run on this old image and VM?
On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis lewis@mail.msen.com wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote: > That's probably because someone has registered with a name containing wide > characters, so the whole page is served as a WideString. This bug has > happened a few times before, and I don't remember what the > temporary solution was, but I think Dave knows. > > Levente
That is probably it. Note that St??phane Rollandin (spfa) has just
registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com from time to time. If we
do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending
WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of
developers to source.squeak.org :-)
Dave > > On Tue, 1 Dec 2015, Bert Freudenberg wrote: > > >source.squeak.org looks weird. > > > >- Bert - > > > > > > > >>On 01.12.2015, at 00:12, Levente Uzonyi <leves@elte.hu> wrote: > >> > >>Thanks whoever requested the restart. The machine seems to have
four > >>times as much RAM as before. I've hopefully restarted all services. > >>Should anything be missing, let me know! > >> > >>Levente > >> > >>On Mon, 30 Nov 2015, Levente Uzonyi wrote: > >> > >>>Hi All, > >>> > >>>box4 seems to have been down since yesterday. Box3 is fine, so hopefully > >>>a restart will bring box4 back up. > >>>The problem is that only the SFC has access to the box, and only they > >>>can request a restart. > >>>So please ask them to do so. > >>> > >>>Thanks, > >>>Levente > >>> > > > >
On Tue, 1 Dec 2015, Chris Muller wrote:
It is running a much newer image (4.5 instead of 3.11), I don't know if it can support the special chars or not, but as I went to test it, I found teh cause of the box4 crash, and a number of other problems:
- my own "repcopy" script which is running on an hourly cron is getting some error, but leaving the process (squeak iamge) running. My bet is box4 ran out of memory and crashed. I just killed about 50 of those processes. The script redirects stdout and stderr to "repcopy.log" and "repcopy.err", respectively. However, upon the next scheduled run, I found the process hung again but those two files were empty. However, there was a SqueakDebug.log file...
IIRC without a forkbomb, one can't kill linux like that, because linux will just kill the process which would require too much memory to run. Since syslog contains no information about why the server went down, we can't really tell what happened.
Levente
- The SqueakDebug.log file shows Squeak-Version-kfr.4712 is triggering the Warning, "About to serialize empty mcz". This version has a comment of, "Messed up last version, try again", and so is a piece of litter in our ancestry and trunk. I again ask y'all to support me on this littering issue and help me address it with community. Please! - The weird thing is, the script is wrapped in a Smalltalk>>#run:, which handles all Warnings by resuming them. So, I don't know why it is halting..
As for production with the special-char:
- the object model for production source.squeak.org is stored in file
/srv/box2/home/squeaksource/ss/data.obj
but it is currently timestamped April 20th, 2015. So any changes to that model since that time, including the addition of Stephan and Patrick are probably only in the production image (/srv/box2/home/squeaksource/Squeak3.11-8824-SS.image).
- I have just downloaded that image /srv/box2/home/squeaksource/Squeak3.11-8824-SS.image and exported a new "data.obj" file.
Next I will make a new method in ss-deploy.image to refresh the Magma db with the model from data.obj without having to reload all the versions. I will then know whether it can handle the special characters in people's names..
On Tue, Dec 1, 2015 at 2:35 PM, Levente Uzonyi leves@caesar.elte.hu wrote: You were the one who set up a newer image with Magma backend on box4, though that image hasn't synchronized the list of users, so it's web interface still works. I assume, it would break, too.
Levente On Tue, 1 Dec 2015, Chris Muller wrote: Would a newer image and VM solve this? Why do we continue to run on this old image and VM? On Tue, Dec 1, 2015 at 7:56 AM, David T. Lewis <lewis@mail.msen.com> wrote: On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote: > That's probably because someone has registered with a name containing wide > characters, so the whole page is served as a WideString. This bug has > happened a few times before, and I don't remember what the > temporary solution was, but I think Dave knows. > > Levente That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed. I see the same thing on squeaksource.com from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page). If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending WideString and patching it to be displayable. The good news is that we are adding a more diverse range of developers to source.squeak.org :-) Dave > > On Tue, 1 Dec 2015, Bert Freudenberg wrote: > > >source.squeak.org looks weird. > > > >- Bert - > > > > > > > >>On 01.12.2015, at 00:12, Levente Uzonyi <leves@elte.hu> wrote: > >> > >>Thanks whoever requested the restart. The machine seems to have four > >>times as much RAM as before. I've hopefully restarted all services. > >>Should anything be missing, let me know! > >> > >>Levente > >> > >>On Mon, 30 Nov 2015, Levente Uzonyi wrote: > >> > >>>Hi All, > >>> > >>>box4 seems to have been down since yesterday. Box3 is fine, so hopefully > >>>a restart will bring box4 back up. > >>>The problem is that only the SFC has access to the box, and only they > >>>can request a restart. > >>>So please ask them to do so. > >>> > >>>Thanks, > >>>Levente > >>> > > > >
On Tue, Dec 01, 2015 at 05:39:01PM -0600, Chris Muller wrote:
It is running a much newer image (4.5 instead of 3.11), I don't know if it can support the special chars or not, but as I went to test it, I found teh cause of the box4 crash, and a number of other problems:
- my own "repcopy" script which is running on an hourly cron is getting
some error, but leaving the process (squeak iamge) running. My bet is box4 ran out of memory and crashed. I just killed about 50 of those processes. The script redirects stdout and stderr to "repcopy.log" and "repcopy.err", respectively. However, upon the next scheduled run, I found the process hung again but those two files were empty. However, there was a SqueakDebug.log file...
Yep, that would do it. System out of memory, maybe swapping or maybe not, but either way not able to fork processes to serve new ssh connections. The system is still alive and responding to pings, but nothing else works. That's exactly what we were seeing.
Squeak images interacting with MCZ files are not failure proof. If you run one from a cron, you'll want to add some extra protection. You could do some sort of process lock so the cron job first kills off any lingering processes from the last run. Or you can maybe make do by adding a watchdog process (*) in the image itself to exit unconditionally after N minutes if the image is still alive.
- The SqueakDebug.log file shows *Squeak-Version-kfr.4712* is triggering
the Warning, "About to serialize empty mcz". This version has a comment of, "Messed up last version, try again", and so is a piece of litter in our ancestry and trunk. I again ask y'all to support me on this littering issue and help me address it with community. Please!
Well and good, but you cannot rely on that. Bad stuff will happen, people will make mistakes, and images will get stuck.
Dave
(*) As an example, here is the watchdog process that I am using in the VMUnixBuild.st script in the InterpreterVM job on build.squeak.org:
"If exitWhenDone is true, image will exit without saving when this script is complete or if an error is encountered."
exitWhenDone := true. watchdogMinutes := 20. "Kill image if not done in this many minutes"
"Set a watchdog timer to kill this image if it gets stuck while running headless. Applies only if the image is set to exit when complete." exitWhenDone ifTrue: [ [(Delay forSeconds: 60 * watchdogMinutes) wait. log value: 'Watchdog: killing image after ', watchdogMinutes asString, ' minutes'. OSProcess thisOSProcess sigkill: OSProcess thisOSProcess ] forkAt: Processor userInterruptPriority].
Waiting won't help, because of the "Recently Joined" list. Since no one can join now, that list will stay the same.
Levente
On Tue, 1 Dec 2015, David T. Lewis wrote:
On Tue, Dec 01, 2015 at 12:42:33PM +0100, Levente Uzonyi wrote:
That's probably because someone has registered with a name containing wide characters, so the whole page is served as a WideString. This bug has happened a few times before, and I don't remember what the temporary solution was, but I think Dave knows.
Levente
That is probably it. Note that St??phane Rollandin (spfa) has just registered a new account. That is probably showing up on the source.squeak.org home page, and blowing up when the wide string is displayed.
I see the same thing on squeaksource.com from time to time. If we do nothing, it will probably "fix" itself in a little while (after Stef's activity becomes old news and scrolls off the home page).
If the problem does not go away by itself, may need to fix it in the source.squeak.org image, which involves finding the offending WideString and patching it to be displayable.
The good news is that we are adding a more diverse range of developers to source.squeak.org :-)
Dave
On Tue, 1 Dec 2015, Bert Freudenberg wrote:
source.squeak.org looks weird.
- Bert -
On 01.12.2015, at 00:12, Levente Uzonyi leves@elte.hu wrote:
Thanks whoever requested the restart. The machine seems to have four times as much RAM as before. I've hopefully restarted all services. Should anything be missing, let me know!
Levente
On Mon, 30 Nov 2015, Levente Uzonyi wrote:
Hi All,
box4 seems to have been down since yesterday. Box3 is fine, so hopefully a restart will bring box4 back up. The problem is that only the SFC has access to the box, and only they can request a restart. So please ask them to do so.
Thanks, Levente
box-admins@lists.squeakfoundation.org