[Box-Admins] Re: squeaksource.com image update (was: Does source.squeak.org have the socket leak problem?)

Frank Shearar frank.shearar at gmail.com
Mon Oct 14 09:09:47 UTC 2013


On 13 October 2013 22:34, David T. Lewis <lewis at mail.msen.com> wrote:
> On Sat, Oct 12, 2013 at 05:58:06PM -0400, David T. Lewis wrote:
>> On Wed, Oct 02, 2013 at 05:33:31PM +0200, Bert Freudenberg wrote:
>> > On 2013-10-02, at 17:17, "David T. Lewis" <lewis at mail.msen.com> wrote:
>> >
>> > > Would you mind putting a copy of the image and changes files for source.squeak.org
>> > > on box3 so I can take a look at it (I don't have access to box2)? Thanks.
>> >
>> > Done: ~bertfreudenberg/
>> >
>>
>> I have replaced the older squeaksource.com image with a newer image based on
>> the source.squeak.org image that Bert provided. I activated the new image
>> on box3.squeak.org today, and am monitoring it for problems.
>>
>> I started the repository export yesterday and completed the import to the new
>> image today. During that period there have been no commits to squeaksource.com,
>> so I anticipate no loss of data.
>>
>> The new image is called ~ssdotcom/SqueakSource/squeaksource.3.image. In the
>> event of problems, the rollback plan is to reactivate the older
>> ~ssdotcom/SqueakSource/squeaksource.2.image (see ~ssdotcom/README for details).
>>
>> Currently the new image is running and appears to work fine (although users
>> may have noticed some brief outages, for which I apologize).  However it is
>> currently consuming a heavy CPU load, so I am watching to see if this goes
>> down (I have seen similar patterns in the past related to cache updates,
>> which seem to settle down after a while). If the CPU load does not go back
>> down within the next hour, I will revert back to the old image and try this
>> again on another day.
>>
>
> The new squeaksource image seems to be running well. It is too early to say
> for sure, but the runaway socket leaks that I saw in the old image do not
> appear to be occurring in the new image. There is however one socket leak
> that I have found to be associated with the VNC server. This apparently
> leaves one open file descriptor when a client closes its connection (BYW,
> this is the first time I've really had occasion to use Ian's RFBServer, it
> is really quite amazing). This does not directly impact squeaksource.com
> stability so I'm not going to worry about it for now.
>
> Overall performance seems similar to that of the previous image. I made
> several commits to the repository today with OSProcess updates, and I
> watched the effect on squeaksource (using top). CPU and memory impact was
> negligible.
>
> However, there are other things going on in the image that generate a lot
> of load. There is a once-daily statistics update that consumes CPU for about
> half a minute or so. This is likely a much heavier impact than we would
> see with source.squeak.org due to the size of the repository. We also saw
> one incident in the old image in which memory utilization went through the
> roof, and at this point I have no clue what may have caused it. Aside from
> the socket leak problem, this was the most important problem affecting
> squeaksource stability (and the rest of the system, e.g. Jenkins, as well),
> so I'm watching for evidence of a recurrence.
>
> As per guidance from Ken, squeaksource.com will remain on box3 (good, I
> don't really want to move it again). Frank, this does present a risk to the
> Jenkins jobs that are the primary purpose of box3. Please let me know if
> you see problems that may be related to squeaksource load.

I'll keep an eye out. It sounds like the load-causing jobs are
intermittent though, so shouldn't present a massive problem. It's also
a problem that can be mitigated by farming out the work to non-box3
build slaves.

I set up another slave, running off the laptop that's turned into my
sons' Scratch-pad, but haven't yet mastered the unix-fu to make the
build slave start automatically. That will help a bit, as do Tony's
slaves (now that we beat their memory usage into shape).

frank

> Dave
>


More information about the Box-Admins mailing list