Box-Admins November 2013

box-admins@lists.squeakfoundation.org

10 participants
27 discussions

RE: [Box-Admins] Runaway ruby process on box3 (was: [squeak-dev] trunk down again)
by Ken Causey 09 Nov '13

09 Nov '13

And as soon as I wrote this I realized having strace64 was pointless and uninstalled it along with a dependency that was installed with it. I guess I thought I was on box4 for a moment. Ken > -------- Original Message -------- > Subject: RE: [Box-Admins] Runaway ruby process on box3 (was: > [squeak-dev] trunk down again) > From: "Ken Causey" <ken(a)kencausey.com> > Date: Fri, November 08, 2013 6:35 pm > To: "Squeak Hosting Support" <box-admins(a)lists.squeakfoundation.org> > > > strace and strace64 are now installed on box3. Of course anyone with > sudo access could have done the same. > > Ken > > > -------- Original Message -------- > > Subject: Re: [Box-Admins] Runaway ruby process on box3 (was: > > [squeak-dev] trunk down again) > > From: "David T. Lewis" <lewis(a)mail.msen.com> > > Date: Fri, November 08, 2013 6:23 pm > > To: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > > > > > > On Fri, Nov 08, 2013 at 11:16:34PM +0000, Frank Shearar wrote: > > > On 8 November 2013 23:06, David T. Lewis <lewis(a)mail.msen.com> wrote: > > > > On Wed, Nov 06, 2013 at 10:02:38PM -0600, Chris Muller wrote: > > > >> Trunk stopped responding again. box2 might need to be rebooted again. > > > > > > > > We have a ruby process running on box3 that is consuming all available CPU, > > > > that has been reparented to init, and that has been running for a long time. > > > > > > > > jenkins@box3-squeak:~$ ps -aef | grep ruby > > > > jenkins 19054 18955 0 22:48 pts/1 00:00:00 grep ruby > > > > jenkins 28923 1 87 Nov06 ? 1-19:18:12 /var/lib/jenkins/.rvm/rubies/ruby-1.9.3-p392/bin/ruby -S rspec test/image_test.rb > > > > jenkins@box3-squeak:~$ ps -l -p 28923 > > > > F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD > > > > 0 R 103 28923 1 87 80 0 - 5205 - ? 1-19:18:19 ruby > > > > jenkins@box3-squeak:~$ top -p 28923 -b -n 1 > > > > top - 22:48:47 up 199 days, 7:38, 2 users, load average: 7.11, 7.11, 7.16 > > > > Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie > > > > Cpu(s): 2.0%us, 0.2%sy, 2.0%ni, 95.3%id, 0.4%wa, 0.0%hi, 0.0%si, 0.1%st > > > > Mem: 1032140k total, 1009656k used, 22484k free, 74156k buffers > > > > Swap: 524280k total, 10732k used, 513548k free, 209540k cached > > > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > 28923 jenkins 20 0 20820 12m 828 R 87.0 1.3 2598:23 ruby > > > > > > > > This appears to be a process that got disconnected from one of our Jenkins > > > > jobs and has been stuck burning cpu for that last couple of days. That also > > > > happens to be roughly the time frame in which our source.squeak.org service > > > > got hung up. The Jenkins jobs (e.g. SqueakTrunk) are interacting with > > > > source.squeak.org, so it is possible that the two problems are related. > > > > > > > > I noticed this because the InterpreterVM and CogVM jobs are failing after > > > > their watchdog timers expire, but the actual jobs succeed if I run them on > > > > my own local PC. Those jobs run Squeak at low priority (nice) and it is > > > > possible that their failures are due to the runaway ruby job consuming all > > > > available resource. > > > > > > > > I have not killed the runaway process yet, in case anyone wants to have a > > > > look at first. > > > > > > I would be happy if you attached strace to it, collected some data, > > > and then killed it. It's a runaway SqueakTrunk job. (Well, it's clear > > > you know that, but I just had to point it out.) Hopefully the strace > > > would give enough clues to find what looks like a tight loop... > > > > > > > We don't have strace installed on box3, sorry. > > > > Dave

1 0

trunk down again
by Chris Muller 09 Nov '13

09 Nov '13

Trunk stopped responding again. box2 might need to be rebooted again.

3 3

source.squeak.org daemontools service
by Chris Muller 07 Nov '13

07 Nov '13

Trunk was not responding again, so I ssh'd into box2, killed the process owned by squeaksource. Unfortunately, daemontools did not restart it again. So, I started it manually as user squeaksource but it is still not responding. Ken, do we need to reboot box2 again?

1 0

Re: I am missing all emails from lists.squeakfoundation.org for the last few days
by David T. Lewis 05 Nov '13

05 Nov '13

> On Tue, Nov 05, 2013 at 12:22:12PM -0500, David T. Lewis wrote: >> > On Mon, Nov 04, 2013 at 08:30:09PM -0500, David T. Lewis wrote: >> >> Hi, >> >> >> >> I receive all my personal email at lewis(a)mail.msen.com, including >> mail >> >> from mailing lists originating from lists.squeakfoundation.org. The >> >> lists include: >> > >> > Could you please forward a copy of ALL the headers (or the whole >> > message) for one email that came through in the past? Makes tracing >> > things much faster. >> > >> >> Hi Michael, >> >> I put a sample email message in /tmp/lewis-mail/sample on >> shell.msen.com. > > We saw one issue - looks like box2.squeakfoundation.org might have > been spamming. Some of the checks people use are a bit too sensative. > Whitelisted so it should not be an issue. If anything OTHER than box2 > sends mail, you might mention it and we can make sure it's OK. > Thanks Michael, That's interesting. There was in fact a failure in a web app server running on box2.squeak.org that lasted several days and was resolved only yesterday. The web app (visible as http://source.squeak.org) is a repository that holds the source code for an open source project that I am involved with. It is essentially a version control repository, but it also generates email notifications to various subscribers. Something went wrong in that web application, and this may well have been generating some bad email traffic. Did the spamming from box2 look like actual spam (phishing attacks etc) or did it look like a large number of, or repeat copies of, otherwise legitimate messages? When I first noticed that my mailing list updates had gone silent, I assumed that is was related to the web app failure described above. I later noticed that other people were receiving normal updates, and that I could receive mailing list updates if I sent them to an account on gmail, hence my request to you for help yesterday (thanks!!! again for the assistance). But the bottom line is that my loss of email apparently *was* directly related to the problem on box2.squeak.org. I am sending a CC to the box-admins mailing list, which is where the adminstrators (including myself) for box2.squeakfoundation.org and the source.squeak.org web app reside. Folks, it appears that one side effect of the kind of SqueakSource failure that we experienced over the weekend is that we might inadvertently generate a lot of email traffic. Alternatively, we may have had a mail related issue that was somehow involved in the SqueakSource failure. Dave

2 2

[Fwd: Re: I am missing all emails from lists.squeakfoundation.org for the last few days]
by David T. Lewis 05 Nov '13

05 Nov '13

Thanks Michael, Forwarding to box-admins list. Dave ---------------------------- Original Message ---------------------------- Subject: Re: I am missing all emails from lists.squeakfoundation.org for the last few days From: "Michael Wayne" <service(a)msen.com> Date: Tue, November 5, 2013 3:07 pm To: "David T. Lewis" <lewis(a)mail.msen.com> -------------------------------------------------------------------------- On Tue, Nov 05, 2013 at 02:59:50PM -0500, David T. Lewis wrote: > > > Did the spamming from box2 look like actual spam (phishing attacks etc) or > did it look like a large number of, or repeat copies of, otherwise > legitimate messages? It's all automated but I'd guess that it was trying to either 1) send empty email 2) send email to non-existant users. It was not a direct attack. > Folks, it appears that one side effect of the kind of SqueakSource failure > that we experienced over the weekend is that we might inadvertently > generate a lot of email traffic. Likely some sort of "bad" email. Enough to trip alerts.

1 0

RE: [Box-Admins] Slowness of source.squeak.org
by Ken Causey 05 Nov '13

05 Nov '13

> -------- Original Message -------- > Subject: Re: [Box-Admins] Slowness of source.squeak.org > From: Ken Causey <ken(a)kencausey.com> > Date: Mon, November 04, 2013 1:49 pm > To: ma.chris.m(a)gmail.com, Squeak Hosting Support > <box-admins(a)lists.squeakfoundation.org> > > This system is the first one I setup sudo on and just expediently > started listing users, not really considering that there may be other > options. On the other servers I instead enabled it for all users in the > sudo group and instead add users to the group. I've had little reason > to re-evaluate this on box2 and haven't bothered to change it. I finally fixed this. On box2 now (consistently with box3 and box4) to give a user sudo access just add them to the sudo group. Ken

1 0

Slowness of source.squeak.org
by Levente Uzonyi 05 Nov '13

05 Nov '13

Hi, source.squeak.org became unacceptably slow. Both the web interface, and the MC API are very slow. Can someone take a look? Cheers, Levente P.S.: It's suspicious that some packages seem to be uploaded multiple times: http://lists.squeakfoundation.org/pipermail/vm-dev/2013-November/date.html P.P.S: I've lost the private key to the server, that's why I can't check it myself.

5 12

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Box-Admins November 2013