[Box-Admins] box3.squeak.org off line - HELP neeeded

[Chris Muller]

I just deployed an updated image for "new trunk" at
box4.squeak.org:8888/trunk.  This is the one that uses the Magma
backend.  I haven't done any benchmarks but the response time seems
pretty snappy, even compared to regular source.squeak.org.  Could be
due to Cog, the Magma backend, or box4 being under less load than
box3.

I need to tell you -- I did experience a 1GB memory situation myself
the other day!  No idea how it could happen especially since I'm
doubtful there was any significant level of activity.

It was strange, but I simply restarted it.  Now I'll really keep my eyes open.

On Tue, Oct 8, 2013 at 6:55 PM, David T. Lewis <lewis at mail.msen.com> wrote:
> On Tue, Oct 08, 2013 at 12:12:12PM -0500, Ken Causey wrote:
>>
>> On Tue, Oct 08, 2013 at 02:08:01PM -0400, David T. Lewis wrote:
>> > Aha. That would do it for sure. So something is going on in the squeaksource
>> > image that is using a *lot* of object memory for some period of time. The use
>> > of 957m resident memory would very likely be enough to cause the symptoms
>> > that we saw.
>> >
>> > Do you know if we see any similar pattern of memory usage on the source.squeak.org
>> > server? I'm already convinced that the squeaksource.com image badly needs
>> > to be updated to the same level of Squeak/Seaside/SqueakSource as our
>> > source.squeak.org server (due to socket leak problems if nothing else).
>> >
>> > I also recall that squeaksource.com on the SCG server had horrible performance
>> > problems whenever we tried to commit a large MCZ to the VMMaker repository,
>> > and I always assumed that it was memory related in some way on another.
>> >
>>
>> The answer is yes but of course source.squeak.org is so so much smaller
>> in terms of the data it has to handle and the traffic level that its
>> swings in memory usage are rarely a significant problem.
>>
>> I should have said something more when you started discussing this, and
>> I apologize for my level of silence. I'm concerned that given the amount
>> of trouble the original owners had with SqueakSource.com, I don't see
>> how we can expect to do better, particularly with a virtual server with
>> only 1GB of RAM allocated to it. Setting it to read-only, and I'm not
>> sure but perhaps that is how it is set now, is of course going to reduce
>> the load. But how much? It's not an easy question to answer.
>
> I am cautiously optimistic. We used to have the VMMaker repository hosted
> on squeaksource.com, and the performance and reliability were so horrible
> that we moved it to source.squeak.org. Wonder of wonders, all the problems
> went away and the VMMaker repository has been trouble-free ever since.
>
> That tells me that upgrading the squeaksource.com image to the same level of
> code that we are already using for source.squeak.org has a high probability
> of improving things. I plan to do that as soon as possible, and I'm sure
> that Tobias and Bert will help as needed.
>
> Also, based on my personal experience as a user of the VMMaker repo on
> squeaksource.com, I always suspected that the performance and reliability
> problems were related to memory usage on the host. That's because I could
> watch VMMaker commits in the progress bar, and they would proceed slowly
> but normally about 2/3 of the way through the progress bar, then would
> then grind to a halt and either take a long time to complete, or time
> out and fail completely. That always looked to me like a server app running
> out of memory and starting to swap to disk. So I am not surprised that we
> are seeing a memory related problem with our copy of the squeaksource
> image now. Again, I suspect that the problem may magically be cured by
> updating the squeaksource.com image to the same level as source.squeak.org.
>
> Anyhow, that's my story and I'm sticking to it until proven wrong ;-)
>
> Dave
>