[Box-Admins] squeaksource.com is down (EOM)

[David T. Lewis]

Thanks Tobias,

There is definitely something wrong with the image. It is growing in
memory and currently using a lot of CPU. I will attempt an off-line
fix, but brief outages are possible over the next hour or two.

The image normally saves itself every hour, and it looks like this last
happened successfully three days ago:

squeaksourcecom at dan:~/SqueakSource$ ls -lt squeaksource.6.*
-rw-r--r-- 1 squeaksourcecom www-data  16317608 Feb 16 14:13 squeaksource.6.changes
-rw-r--r-- 1 squeaksourcecom www-data 814463464 Feb 14 08:33 squeaksource.6.image

That means that today's restart would have begun with the Feb 14 copy of
the image, so there may be some data loss related to this (I'll check it
later).

Dave



On Thu, Feb 16, 2017 at 03:16:43PM +0100, Tobias Pape wrote:
> TL;DR: oom at 7am,  Too many open files at 11am.
> 
> I cycled the image, lets wait?
> 
> To quote the log:
> 
> 2017-02-16 07:31:22.318429500 
> 2017-02-16 07:31:22.318431500 out of memory
> 2017-02-16 07:31:22.318431500 
> 2017-02-16 07:31:22.318431500 1066476624 Behavior>new
> 2017-02-16 07:31:22.318432500 1066476532 Exception class>signal
> 2017-02-16 07:31:22.318432500 1066476440 Behavior>basicNew
> 2017-02-16 07:31:22.318432500 1066476348 Behavior>new
> 2017-02-16 07:31:22.318433500 1066476256 Exception class>signal
> 2017-02-16 07:31:22.318433500 1066476164 Behavior>basicNew
> 2017-02-16 07:31:22.318433500 1066476072 Behavior>new
> 2017-02-16 07:31:22.318434500 1066475980 Exception class>signal
> 2017-02-16 07:31:22.318465500 1066475796 Behavior>basicNew
> 2017-02-16 07:31:22.318466500 1066475704 Behavior>new
> 2017-02-16 07:31:22.318466500 1066475612 Exception class>signal
> 2017-02-16 07:31:22.318466500 1066475520 Behavior>basicNew
> 2017-02-16 07:31:22.318467500 1066475428 Behavior>new
> 2017-02-16 07:31:22.318467500 1066475336 Exception class>signal
> 2017-02-16 07:31:22.318467500 1066475244 Behavior>basicNew
> 2017-02-16 07:31:22.318468500 1066475152 Behavior>new
> 2017-02-16 07:31:22.318468500 1066475060 Exception class>signal
> 2017-02-16 07:31:22.318471500 1066474968 Behavior>basicNew
> 2017-02-16 07:31:22.318471500 1066474876 Behavior>new
> 2017-02-16 07:31:22.318471500 1066474784 Exception class>signal
> 2017-02-16 07:31:22.318472500 1066474692 Behavior>basicNew:
> 2017-02-16 07:31:22.318472500 1066474600 Array class>new:
> 2017-02-16 07:31:22.318472500 1066474508 EventSensor>fetchMoreEvents
> 2017-02-16 07:31:22.318473500 1066473864 EventSensor>eventTickler
> 2017-02-16 07:31:22.318473500 1066473716 BlockClosure>on:do:
> 2017-02-16 07:31:22.318475500 817936580 EventSensor>eventTickler
> 2017-02-16 07:31:22.318475500 817936488 EventSensor>installEventTickler
> 2017-02-16 07:31:22.318476500 817936672 BlockClosure>newProcess
> 2017-02-16 07:31:22.388240500 starting ./run in pid 15028
> 2017-02-16 07:31:22.388284500 cd /?/squeaksourcecom/SqueakSource
> 2017-02-16 07:31:22.388379500 exec setuidgid squeaksourcecom /usr/local/bin/squeak -vm-display-null /srv/squeaksourcecom/SqueakSource/squeaksource.6.image
> 2017-02-16 10:41:23.678772500 acceptHandler: Too many open files
> 2017-02-16 10:41:23.680035500 acceptHandler: aborting server 7 pss=0x1a136a0
> 2017-02-16 10:42:13.734814500 socketStatus: freeing invalidated pss=0x1a136a0