[Box-Admins] For information - possible squeaksource.com issue

[David T. Lewis]

I have repaired the broken squeaksource.com image and reloaded it on box3.
As far as I can tell, no data was lost and nothing bad happened aside from
a couple of very brief outages as I was discovering and then fixing the problem.

Good news: We now have a snapshot of an image that captures the problem
and that can be used for debugging. I am looking at it now, and anyone
else who might be interested can download it from box3.squeak.org. The
backup is in /home/ssdotcom/SqueakSource/BACKUPS/ss-image-save-20131111.tgz.

This problem has definitely happened before, both on the old squeaksource.com
image from SCG, and also on the new image that is based on source.squeak.org.
I think it is quite likely that source.squeak.com was in this condition at
the time of our recent failure, and that is my working assumption at this
point.

I also corrected the problem in which VNC access was not available when
an error occurred in startUp processing.

Here is what I did to repair the image:

- First, open a process browser and note the runaway session handlers.
	ProcessBrowser open.

- Then kill runaway processes:

	Process allInstances do: [ :each |
		each priority = 30 ifTrue: [ 
			each terminate ] ].

Get a clean image restart by temporarily commenting out the line
"SSKom startOn: self port" in the SSRepository class>>startUp: method.

Then close all the debug notifiers, save the image, and restart it.

Restore SSRepository class>>startUp: to its previous version.

Move the RFBServer up in the system startup list so that it will be
activated prior to starting Seaside and SqueakSource. This will make it
more likely that VNC access will be maintained the next time a problem
like this happens.

	Smalltalk addToStartUpList: RFBServer after: ModDoc. 

Dave


On Sun, Nov 10, 2013 at 11:23:27PM -0500, David T. Lewis wrote:
> For information - no action required:
> 
> Our squeaksource.com service is running with no apparent problems, but
> there is a possibility of problems arising in the next day, so I am
> sending this for information.
> 
> I checked our squeaksource.com image on box3 via VNC about an hour ago,
> and noticed an error notifier related to startup processing following
> an image save. The error was related to port 8888 already in use. I
> closed the notifier, then found that http://squeaksource.com was no
> longer active.  I made a backup copy of the most recently saved disk
> image (in a BACKUPS subdirectory), then saved the image and restarted
> it.
> 
> Squeaksource.com is running normally, but VNC access is no longer
> working. This means that something failed in the startup processing,
> and this may lead to problems later on. It also means that it is
> hard to debug, because I no longer have VNC access.
> 
> I cannot do anything further tonight, but will follow up tomorrow
> to correct the problem.
> 
> Dave