[Box-Admins] David!: SqueakSource.com problem

[Ken Causey]

On 12/21/2013 12:01 PM, David T. Lewis wrote:
> On Sat, Dec 21, 2013 at 12:37:26PM -0500, David T. Lewis wrote:
>> On Sat, Dec 21, 2013 at 10:30:14AM -0600, Ken Causey wrote:
>>> I don't always think of it but occasionally I visit www.squeaksource.com
>>> just to make sure that the front page at least looks right and that it
>>> is responding.  Today I'm not getting the complete HTML source much less
>>> images or styling.
>>>
>>> Under such circumstances can one of us just kill the existing process
>>> and let daemontools restart it at least as a first attempt to fix the
>>> problem?  I'll leave this to David this time.  But guidance and comments
>>> on how others can help maintain this service, at least at a first
>>> approximation, would be appreciated.
>>
>> Thanks Ken!
>>
>> Here is what I see prior to taking any action:
>>
>> The squeaksource.com repository is working normally from the point of
>> view of Monticello clients, but the web interface is not working. Connecting
>> to the image with VNC and looking at a process browser, I see one extra
>> Seaside process. Normally there is one processes at priority 30 when no
>> activity is going on, but I see two of them, one of which shows no context
>> in the right pane, so I'm sure this is related to the problem.
>>
>> I am going to resist the temptation to fix it in the image, and instead
>> will just do the daemontools command "svc -t ssdotcom" and see what happens.
>> That's pretty much the only available action to be taken if squeaksource.com
>> gets stuck and I am not around to look at it, so I'm curious to see if it
>> works.
>>
>> Results of the experiment to follow ...
>>
>
> Results: It is still broken. I actually did "sudo svc -i ssdotcom" (not -t)
> to kill it gently. The image restarted normally, and the extra process is
> now gone, but the web interface is still broken. I then reconnected to the
> image through VNC, and stopped and started SSKom. No joy.
>
> At this point, the repository is functional but the web interface is not.
>
> A copy of the image is at box3-squeak:/home/ssdotcom/SqueakSource/BACKUPS/ss-image-save-20131221.tgz
>
> I will try downloading that image and debugging it off line, although I cannot
> spend much time on it for the next few hours at least.
>
>>
>>>
>>> Apologies if I have overlooked something already stated.  The team could
>>> use a better documentation strategy.  Suggestions welcome, at least for
>>> discussion purposes.  As usual the trick is finding something that a
>>> quorum of the ever changing group will actually use.
>>>
>>> Ken
>
> I'm happy with daemontools as the management interface, with a presumption
> that any box admin should be able to do a "sudo svc -i <something>" at any
> time.
>
> I put a README file in the home directory for the service (~ssdotcom), and
> have tried to document as much as I am able in that README. This seems like
> a reasonable convention for services managed by daemontools.
>
> In this particular instance, I have no clue what the actual problem is, but
> hopefully I will have something more useful to say later.
>
> Dave

Thanks David.  I agree that README in the service home directory is a 
good idea.  It seems though that it would be good to have something 
easily accessible if the server were not running at all.  I don't know, 
I tend to violate the 'the perfect is the enemy of the good' dictum 
rather often.

Anyway, I'm glad you are on the case at this point, I don't have 
anything further to contribute.  I don't think it is an issue that can't 
wait hours or even a day or more to be resolved, work on it when your 
life allows.

Ken