[Box-Admins] Re: squeaksource.com image update (was: Does source.squeak.org have the socket leak problem?)

[David T. Lewis]

On Sat, Oct 12, 2013 at 05:58:06PM -0400, David T. Lewis wrote:
> On Wed, Oct 02, 2013 at 05:33:31PM +0200, Bert Freudenberg wrote:
> > On 2013-10-02, at 17:17, "David T. Lewis" <lewis at mail.msen.com> wrote:
> > 
> > > Would you mind putting a copy of the image and changes files for source.squeak.org
> > > on box3 so I can take a look at it (I don't have access to box2)? Thanks.
> > 
> > Done: ~bertfreudenberg/
> >
> 
> I have replaced the older squeaksource.com image with a newer image based on
> the source.squeak.org image that Bert provided. I activated the new image
> on box3.squeak.org today, and am monitoring it for problems.
> 
> I started the repository export yesterday and completed the import to the new
> image today. During that period there have been no commits to squeaksource.com,
> so I anticipate no loss of data.
> 
> The new image is called ~ssdotcom/SqueakSource/squeaksource.3.image. In the
> event of problems, the rollback plan is to reactivate the older
> ~ssdotcom/SqueakSource/squeaksource.2.image (see ~ssdotcom/README for details).
> 
> Currently the new image is running and appears to work fine (although users
> may have noticed some brief outages, for which I apologize).  However it is
> currently consuming a heavy CPU load, so I am watching to see if this goes
> down (I have seen similar patterns in the past related to cache updates,
> which seem to settle down after a while). If the CPU load does not go back
> down within the next hour, I will revert back to the old image and try this
> again on another day.
>

The new squeaksource image seems to be running well. It is too early to say
for sure, but the runaway socket leaks that I saw in the old image do not
appear to be occurring in the new image. There is however one socket leak
that I have found to be associated with the VNC server. This apparently
leaves one open file descriptor when a client closes its connection (BYW,
this is the first time I've really had occasion to use Ian's RFBServer, it
is really quite amazing). This does not directly impact squeaksource.com
stability so I'm not going to worry about it for now.

Overall performance seems similar to that of the previous image. I made
several commits to the repository today with OSProcess updates, and I
watched the effect on squeaksource (using top). CPU and memory impact was
negligible.

However, there are other things going on in the image that generate a lot
of load. There is a once-daily statistics update that consumes CPU for about
half a minute or so. This is likely a much heavier impact than we would
see with source.squeak.org due to the size of the repository. We also saw
one incident in the old image in which memory utilization went through the
roof, and at this point I have no clue what may have caused it. Aside from
the socket leak problem, this was the most important problem affecting
squeaksource stability (and the rest of the system, e.g. Jenkins, as well),
so I'm watching for evidence of a recurrence.

As per guidance from Ken, squeaksource.com will remain on box3 (good, I
don't really want to move it again). Frank, this does present a risk to the
Jenkins jobs that are the primary purpose of box3. Please let me know if
you see problems that may be related to squeaksource load.

Dave