[Box-Admins] David!: SqueakSource.com problem

[David T. Lewis]

On Sat, Dec 21, 2013 at 12:37:26PM -0500, David T. Lewis wrote:
> On Sat, Dec 21, 2013 at 10:30:14AM -0600, Ken Causey wrote:
> > I don't always think of it but occasionally I visit www.squeaksource.com 
> > just to make sure that the front page at least looks right and that it 
> > is responding.  Today I'm not getting the complete HTML source much less 
> > images or styling.
> > 
> > Under such circumstances can one of us just kill the existing process 
> > and let daemontools restart it at least as a first attempt to fix the 
> > problem?  I'll leave this to David this time.  But guidance and comments 
> > on how others can help maintain this service, at least at a first 
> > approximation, would be appreciated.
> 
> Thanks Ken!
> 
> Here is what I see prior to taking any action:
> 
> The squeaksource.com repository is working normally from the point of
> view of Monticello clients, but the web interface is not working. Connecting
> to the image with VNC and looking at a process browser, I see one extra
> Seaside process. Normally there is one processes at priority 30 when no
> activity is going on, but I see two of them, one of which shows no context
> in the right pane, so I'm sure this is related to the problem.
> 
> I am going to resist the temptation to fix it in the image, and instead
> will just do the daemontools command "svc -t ssdotcom" and see what happens.
> That's pretty much the only available action to be taken if squeaksource.com
> gets stuck and I am not around to look at it, so I'm curious to see if it
> works.
> 
> Results of the experiment to follow ...
> 

Results: It is still broken. I actually did "sudo svc -i ssdotcom" (not -t)
to kill it gently. The image restarted normally, and the extra process is
now gone, but the web interface is still broken. I then reconnected to the
image through VNC, and stopped and started SSKom. No joy.

At this point, the repository is functional but the web interface is not.

A copy of the image is at box3-squeak:/home/ssdotcom/SqueakSource/BACKUPS/ss-image-save-20131221.tgz

I will try downloading that image and debugging it off line, although I cannot
spend much time on it for the next few hours at least.

> 
> > 
> > Apologies if I have overlooked something already stated.  The team could 
> > use a better documentation strategy.  Suggestions welcome, at least for 
> > discussion purposes.  As usual the trick is finding something that a 
> > quorum of the ever changing group will actually use.
> > 
> > Ken

I'm happy with daemontools as the management interface, with a presumption
that any box admin should be able to do a "sudo svc -i <something>" at any
time.

I put a README file in the home directory for the service (~ssdotcom), and
have tried to document as much as I am able in that README. This seems like
a reasonable convention for services managed by daemontools.

In this particular instance, I have no clue what the actual problem is, but
hopefully I will have something more useful to say later.

Dave