[Box-Admins] squeaksource.com on box3

[David T. Lewis]

The transition of SqueakSource to our box3 server was completed earlier today,
with the DNS record transition being completed about 11 hours ago. Everything
seems to be working fine, but I want to note that I did some disruptive things
about a half hour ago in case anyone has noticed any problems.

Right before the transition (yesterday), I ran some wget comparisons to look
for MCZ commits that were present on the SCG server but not yet updated on
our copy of SqueakSource. I found two missing commits, one in Chronos, and
the other in Cryptography. After allowing our SqueakSource to run without
interruption today, I copied those two files into our ~ssdotcom/Squeaksource/ss
repository, and killed the SqueakSource image to allow it to restart normally
and pick up the two missing commits. I actually restarted the image several
times, because I noticed high CPU loads that I could not explain. This turned
out to be normal processing of the repository cache thread, but it looked
alarming to me at the time.

So ... if anyone noticed some brief outages on our squeaksource.com service,
it was just me fumbling around trying to load the missing updates.

We will want to keep an eye on overall system performance on box3 for a while.
The squeaksource.com service is competing for resources with our Jenkins jobs,
and I don't yet know if this will lead to any issues.

In the event that squeaksource.com requires a restart, as will be the case
whenever box3 is rebooted, see the instructions in ~ssdotcom/README and
copied below.

Dave


HOW TO START THE SYSTEM
-----------------------

Starting SqueakSource after a system boot:

1) Use sudo to log in to the system with the ssdotcom user:

lewis at box3-squeak:~$ sudo su - ssdotcom
ssdotcom at box3-squeak:~$

2) Verify that runimage.sh is not currently running. When it is running,
you will see the following, which is NOT what you want (so make sure
the line with "/bin/sh ./runimage.sh" is NOT present):

ssdotcom at box3-squeak:~$ ps -aef | grep runimage.sh
ssdotcom  6005  5999  0 20:35 pts/1    00:00:00 grep runimage.sh
ssdotcom 20344     1  0 Sep15 ?        00:00:12 /bin/sh ./runimage.sh

You can also do a similar check to satisfy yourself that the
SqueakSource image is not running (so you should NOT see the line
with "squeaksource.2.image" in the example below):

ssdotcom at box3-squeak:~$ ps -aef | grep squeaksource
ssdotcom  3255 20344  8 Sep22 ?        03:40:47 /usr/local/lib/squeak/4.10.5-2619/squeakvm -vm-display-null squeaksource.2.image
ssdotcom  6008  5999  0 20:35 pts/1    00:00:00 grep squeaksource

3) Change to administrative directory

ssdotcom at box3-squeak:~$ cd SqueakSource
ssdotcom at box3-squeak:~/SqueakSource$ 

4) Run the shell script using nohup.

ssdotcom at box3-squeak:~/SqueakSource$ nohup ./runimage.sh&

Note the trailing ampersand, which puts the script into background mode.
The nohup command will prevent the script from being terminated, and also
directs all of the script output (stdout and stderr) to a file called
nohup.out.

Once started, the shell script will keep track of the running SqueakSource
image, and will restart the image within 60 seconds of any unexpected exit.

It is generally safe to kill the image (i.e. kill the squeakvm process) at
any time, and allow the runimage.sh script to restart it automatically.
This may be necessary, for example, if the VNC server process within the
image becomes inaccessible for some reason.