[Box-Admins] box3.squeak.org off line - HELP neeeded

[David T. Lewis]

On Tue, Oct 08, 2013 at 12:12:12PM -0500, Ken Causey wrote:
> 
> On Tue, Oct 08, 2013 at 02:08:01PM -0400, David T. Lewis wrote:
> > Aha. That would do it for sure. So something is going on in the squeaksource
> > image that is using a *lot* of object memory for some period of time. The use
> > of 957m resident memory would very likely be enough to cause the symptoms
> > that we saw.
> > 
> > Do you know if we see any similar pattern of memory usage on the source.squeak.org
> > server? I'm already convinced that the squeaksource.com image badly needs
> > to be updated to the same level of Squeak/Seaside/SqueakSource as our
> > source.squeak.org server (due to socket leak problems if nothing else).
> > 
> > I also recall that squeaksource.com on the SCG server had horrible performance
> > problems whenever we tried to commit a large MCZ to the VMMaker repository,
> > and I always assumed that it was memory related in some way on another.
> > 
>
> The answer is yes but of course source.squeak.org is so so much smaller 
> in terms of the data it has to handle and the traffic level that its 
> swings in memory usage are rarely a significant problem.
> 
> I should have said something more when you started discussing this, and 
> I apologize for my level of silence. I'm concerned that given the amount 
> of trouble the original owners had with SqueakSource.com, I don't see 
> how we can expect to do better, particularly with a virtual server with 
> only 1GB of RAM allocated to it. Setting it to read-only, and I'm not 
> sure but perhaps that is how it is set now, is of course going to reduce 
> the load. But how much? It's not an easy question to answer.

I am cautiously optimistic. We used to have the VMMaker repository hosted
on squeaksource.com, and the performance and reliability were so horrible
that we moved it to source.squeak.org. Wonder of wonders, all the problems
went away and the VMMaker repository has been trouble-free ever since.

That tells me that upgrading the squeaksource.com image to the same level of
code that we are already using for source.squeak.org has a high probability
of improving things. I plan to do that as soon as possible, and I'm sure
that Tobias and Bert will help as needed.

Also, based on my personal experience as a user of the VMMaker repo on
squeaksource.com, I always suspected that the performance and reliability
problems were related to memory usage on the host. That's because I could
watch VMMaker commits in the progress bar, and they would proceed slowly
but normally about 2/3 of the way through the progress bar, then would
then grind to a halt and either take a long time to complete, or time
out and fail completely. That always looked to me like a server app running
out of memory and starting to swap to disk. So I am not surprised that we
are seeing a memory related problem with our copy of the squeaksource
image now. Again, I suspect that the problem may magically be cured by
updating the squeaksource.com image to the same level as source.squeak.org.

Anyhow, that's my story and I'm sticking to it until proven wrong ;-)

Dave