Box-Admins April 2013

box-admins@lists.squeakfoundation.org

5 participants
10 discussions

RE: Problems with Build.squeak.org
by Ken Causey 28 Apr '13

28 Apr '13

> -------- Original Message -------- > Subject: Re: Problems with Build.squeak.org > From: Frank Shearar <frank.shearar(a)gmail.com> > Date: Sun, April 28, 2013 7:39 am > To: Ken Causey <ken(a)kencausey.com> > Cc: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > > So it was just a case of not enough time. The builds take around 40 > minutes to run, which isn't bad considering it's 3k tests run three > times. > > But more importantly, how does it look from an unkilled process > standpoint? I saw some messages in Jenkins talking about leaking file > descriptors, so I reckon we probably still have a bit of a problem? > > frank It looks fine. When I started checking today there were a number of processes running and it took a while for it to clear out but once it did there was nothing left hanging about. ExternalProcesses seems to run a lot, is it triggered by commit to trunk or the like? Builds less than an hour apart seem a bit over the top to me. Is it possible to limit it to no more than once every 6 hours or something like that? I ask not because I think it is that important at this point but because I expect it will become more relevant in the future. Ken

2 1

RE: Problems with Build.squeak.org
by Ken Causey 28 Apr '13

28 Apr '13

> -------- Original Message -------- > Subject: Re: Problems with Build.squeak.org > From: Frank Shearar <frank.shearar(a)gmail.com> > Date: Thu, April 25, 2013 4:18 pm > To: Ken Causey <ken(a)kencausey.com> > Cc: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > > > On 24 April 2013 13:31, Ken Causey <ken(a)kencausey.com> wrote: > > On 04/24/2013 01:28 AM, Frank Shearar wrote: > >> > >> On 23 April 2013 23:56, Ken Causey<ken(a)kencausey.com> wrote: > >>> > >>> I may be jumping the gun here, but I'm afraid the issue with box3 is on > >>> its > >>> way to repeating again. > >>> > >>> It seems to me that the ExternalPackages-* tasks are completely fouled > >>> up. > >>> I'm almost certain they take hours wheres as the listed durations are on > >>> the > >>> order of half and hour. More importantly the site currently claims > >>> Jenkins > >>> is idle yet > >> > >> > >> They are indeed the culprits. I think the root of the problem is twofold: > >> * build.squeak.org is different in some way from my own test machines, > >> where these jobs all complete > >> * I'm shelling out from the Rake build, and these child processes > >> aren't killed by the Rake process. > >> > >> I think for now I should disable them. > >> > >> frank > > > > > > Thanks Frank. If there is anything I can do to help you diagnose this > > please let me know. > > I've re-enabled the ExternalPackages job (leaving ExternalPackages-4.3 > and ExternalPackages-4.4 disabled). Instead of just spawning a process > and failing the test after a timeout, we now spawn the process and get > the pid of that process. Then we schedule a future to kill the job > after the maximum test run time, and wait for the process to exit. > > I've kicked off a build, so fingers crossed we _won't_ see those > hanging processes. > > frank Cool. I'll plan to check it out a few times tonight while you are presumably sleeping. Ken

2 2

Re: Problems with Build.squeak.org
by Ken Causey 25 Apr '13

25 Apr '13

On 04/24/2013 01:28 AM, Frank Shearar wrote: > On 23 April 2013 23:56, Ken Causey<ken(a)kencausey.com> wrote: >> I may be jumping the gun here, but I'm afraid the issue with box3 is on its >> way to repeating again. >> >> It seems to me that the ExternalPackages-* tasks are completely fouled up. >> I'm almost certain they take hours wheres as the listed durations are on the >> order of half and hour. More importantly the site currently claims Jenkins >> is idle yet > > They are indeed the culprits. I think the root of the problem is twofold: > * build.squeak.org is different in some way from my own test machines, > where these jobs all complete > * I'm shelling out from the Rake build, and these child processes > aren't killed by the Rake process. > > I think for now I should disable them. > > frank Thanks Frank. If there is anything I can do to help you diagnose this please let me know. Ken

2 1

Problems with Build.squeak.org
by Ken Causey 23 Apr '13

23 Apr '13

I may be jumping the gun here, but I'm afraid the issue with box3 is on its way to repeating again. It seems to me that the ExternalPackages-* tasks are completely fouled up. I'm almost certain they take hours wheres as the listed durations are on the order of half and hour. More importantly the site currently claims Jenkins is idle yet jenkins 2320 0.0 1.4 19332 14480 ? Sl Apr23 0:01 /var/lib/jenkins/.rvm/gems/ruby-1.9.3-p194/bin/rspec jenkins 5272 0.0 0.0 0 0 ? Z Apr23 0:00 [sh] <defunct> jenkins 5435 0.0 1.3 20320 14428 ? Sl Apr23 0:01 /var/lib/jenkins/.rvm/gems/ruby-1.9.3-p194/bin/rspec jenkins 9120 0.0 0.0 1748 492 ? S 00:08 0:00 sh -c nice /var/lib/jenkins/workspace/ExternalPackages/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/var/lib/jenkins/workspace/ExternalPackages/target/FFI.image" ../packag jenkins 9122 0.8 1.8 1052956 19448 ? RN 00:08 0:23 /var/lib/jenkins/workspace/ExternalPackages/target/Squeak-4.10.2.2614-src-32/bld/squeakvm -vm-sound-null -vm-display-null /var/lib/jenkins/workspace/ExternalPackages/target/FFI.image ../package-load-scripts jenkins 9226 0.0 0.0 1748 496 ? S 00:09 0:00 sh -c nice /var/lib/jenkins/workspace/ExternalPackages-Squeak4.3/target/Squeak-4.10.2.2614-src-32/bld/squeak.sh -vm-sound-null -vm-display-null "/var/lib/jenkins/workspace/ExternalPackages-Squeak4.3/target/ jenkins 9228 0.8 1.9 1052956 19632 ? RN 00:09 0:22 /var/lib/jenkins/workspace/ExternalPackages-Squeak4.3/target/Squeak-4.10.2.2614-src-32/bld/squeakvm -vm-sound-null -vm-display-null /var/lib/jenkins/workspace/ExternalPackages-Squeak4.3/target/Control.image I don't think Jenkins is actually able to monitor these tasks correctly currently. Ken

1 0

Re: Reboot box3.squeak.org please
by Ken Causey 23 Apr '13

23 Apr '13

On 04/23/2013 09:06 AM, Bradley M. Kuhn wrote: > Ken Causey wrote at 22:25 (EDT) on Monday: >> When it is convenient for you could you please reboot that system? > > I did a reboot, just now. Sorry that Gandi's interface doesn't allow me > to delegate that to one of you. > > Hopefully the system came back up. I was out sick yesterday, but I'm > online all day and findable on Freenode and other IRC networks if you > need more help. Yes, I checked for you on Freenode and was a bit worried when I didn't find you. ;) Thanks for the reboot. As long as you don't mind us bugging you when we need this (hopefully rarely) I imagine we can deal with a bit of delay in the process. > >> It seems something has gone haywire with one or more of our Jenkins >> build tasks on build.squeak.org (aka box3.squeak.org, 173.246.101.237) >> resulting in the system being unresponsive now for some hours. > > Another of Conservancy's projects is using Jenkins and they noticed they > needed a LOT of RAM to make it work. Do you perhaps need more RAM? Was > it an OOM error that caused this? At this point I have no idea. OOM is a definite possibility but I doubt Jenkins is to blame. We are still working out issues with our customization of the build system and really still learning how to adapt a Smalltalk view of the world to a Ruby/Java/Unix one. Those who actually deal with the Jenkins side of it (not me) will hopefully have more useful information than the hand-waving I am doing here. Ken

1 0

Reboot box3.squeak.org please
by Ken Causey 23 Apr '13

23 Apr '13

It seems something has gone haywire with one or more of our Jenkins build tasks on build.squeak.org (aka box3.squeak.org, 173.246.101.237) resulting in the system being unresponsive now for some hours. When it is convenient for you could you please reboot that system? Thanks, Ken

1 0

Box3 Unresponsive
by Ken Causey 22 Apr '13

22 Apr '13

box3.squeak.org (aka build.squeak.org) has become unresponsive. I noticed over the weekend that it seemed like some of the build processes were living longer and starting to stack up. I'm considering contacting Brad Kuhn and asking him to reset the server but I'm wondering if anyone has any idea how long it has been unresponsive and if maybe I should just be more patient? Ken

2 1

Either need to be Admin on source.squeak.org or have MethodMassage created.
by Eliot Miranda 09 Apr '13

09 Apr '13

Could I become an Administrator on source.squeak.org? Failing that can you create a project called MethodMassage with me as its owner? TIA. Eliot

2 1

[Build server] Problem with Universes test?
by Ken Causey 02 Apr '13

02 Apr '13

There were 4 stuck build processes that all appear to be related to testing Universes, I think. Here is an example: jenkins 27839 0.0 0.0 1748 492 ? S Mar30 0:00 sh -c nice /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2701/coglinux/bin/squeak -vm-sound-null -vm-display-null "/var/lib/jenkins/workspace/SqueakTrunk/target/Universes.image" ../package-load-tests/Universes.st jenkins 27841 2.2 2.3 1054528 24096 ? RN Mar30 60:37 /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2701/coglinux/bin/../lib/squeak/4.0-2701/squeak -vm-sound-null -vm-display-null /var/lib/jenkins/workspace/SqueakTrunk/target/Universes.image ../package-load-tests/Universes.st I killed them. Ken

2 4

Squeak4.5-12568 available
by Chris Muller 01 Apr '13

01 Apr '13

I noticed we did not have any 4.5 alpha version available for download. A clean update from the 4.4 release to the current 4.5 alpha is now available at http://ftp.squeak.org/4.5alpha. PS -- I accidentally also uploaded .image and .changes but no permissions to delete them. Could someone from box-admins do that or add me to the "updates" group and then I can do it?

4 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Box-Admins April 2013