[Box-Admins] Re: I spoke too soon - squeak.org is down again

[Ken Causey]

On Tue, Apr 24, 2012 at 7:48 PM, Chris Cunnington
<smalltalktelevision at gmail.com> wrote:
> This is not a message I want to send. squeak.org is down again and I can't
> bring it back up. There's a 502 Gateway error. One that I did not create.
> map.squeak.org and wiki.squeak.org and Squeak-Dev are up.
>
> Following Ken's instructions, I did:
>
> vncserver -kill :1
>
> which closes things down and produces this:
>
> website at box2:~$ ps aux | grep website
> root     19315  0.0  0.1  6224 1912 ?        Ss   Apr24   0:00 sshd: website
> [priv]
> website  19317  0.0  0.2  6388 1980 ?        S    Apr24   0:00 sshd:
> website at pts/0
> website  19318  0.0  0.1  2600 1500 pts/0    Rs   Apr24   0:00 -bash
> website  21913  0.0  0.0  2496  848 pts/0    R+   00:22   0:00 ps aux
> website  21914  0.0  0.0  1548  480 pts/0    R+   00:22   0:00 grep website
>
> Then, following his instructions:
>
> vncserver
>
> which produces this:
>
> website at box2:~$ ps aux | grep website
> root     19315  0.0  0.1  6224 1912 ?        Ss   Apr24   0:00 sshd: website
> [priv]
> website  19317  0.0  0.2  6388 1980 ?        S    Apr24   0:00 sshd:
> website at pts/0
> website  19318  0.0  0.1  2600 1500 pts/0    Ss   Apr24   0:00 -bash
> website  22163  0.0  0.4  6544 4760 pts/0    S    00:23   0:00 Xtightvnc :1
> -desktop X -auth /home/website/.Xauthority -geometry 1024x768 -depth 24
> -rfbwait 120000 -rfbauth /home/website/.vnc/passwd -rfbport 5901 -fp
> /usr/X11R6/lib/X11/fonts/Type1/,/usr/X11R6/lib/X11/fonts/Speedo/,/usr/X11R6/lib/X11/fonts/misc/,/usr/X11R6/lib/X11/fonts/75dpi/,/usr/X11R6/lib/X11/fonts/100dpi/
> -co /usr/X11R6/lib/X11/rgb
> website  22168 36.4 11.7 1052208 113860 pts/0 R   00:23   1:03
> /usr/local/lib/squeak/3.11.3-2135/squeakvm -pathenc UTF-8 -encoding UTF-8
> -plugins /usr/local/lib/squeak/3.11.3-2135
> /home/website/website/squeaksite.image
> website  22238  0.0  0.0  2496  848 pts/0    R+   00:25   0:00 ps aux
> website  22239  0.0  0.0  1548  480 pts/0    R+   00:25   0:00 grep website
>
> The image is up and running. The vnc is running. And there is still a 502
> Bad Gateway from www.squeak.org. I took a crack at:
>
> svc -u wwwservice
>
> and was permission denied. But if you, as a root person, do that, then the
> VNC doesn't come up, because the squeak.org process can run either under
> daemontools or under VNC, but not both.

Yes, you have to be root to do this.  But DON'T DO THIS if you are
still using VNC as that will start up yet another process of the
website and is likely to just make it worse as they both try to
contend for the same ports.  There is never a need to do both of
these.

All I could do is simply repeat what you have already done which is to
shut it down with vncserver -kill :1 (this :1 assumes that this is the
assigned display port stated when vncserver was original run, you can
also see it in the command line of the XTightvnc command line) and
then start it back up with vncserver.  After that the problem is
almost certainly with something in the image and so you will have to
fix it from there.

So I just tried this and am not able to get into the image via VNC, so
something is badly messed up with the image.  At that point the only
choice is to recover from backup.  One piece of information you
perhaps did not have is that you can find local backups on the system
under /var/cache/rsnapshot/*/localhost/home/website/ .  It just so
happens that under

/var/cache/rsnapshot/daily.0/localhost/home/website/website/  (today's backup)

there were copies made at about 15:00 UTC on the 24th.  I copied those
over the current copies (~website/website/squeaksite.image and
squeaksite.changes) and started vncserver back up and the site is now
running again.

Ken

>
> I'm out of options. Sorry.
>
> Chris