-------- Original Message -------- Subject: Re: [Box-Admins] How does Jenkins start? From: "David T. Lewis" lewis@mail.msen.com Date: Sun, February 09, 2014 1:30 pm To: Squeak Hosting Support box-admins@lists.squeakfoundation.org
On Sun, Feb 09, 2014 at 11:57:23AM -0700, Ken Causey wrote:
-------- Original Message -------- Subject: [Box-Admins] How does Jenkins start? From: Frank Shearar frank.shearar@gmail.com Date: Sun, February 09, 2014 6:49 am To: Squeak Hosting Support box-admins@lists.squeakfoundation.org
I see all too often that builds fail because the JVM process runs out of PermGen space. Up until today I'd assumed that was a problem on the build agents. However, today I bounced my Mac mini (latest victim of this problem) and... nothing happened. But when I bounced _Jenkins_ the build succeeded.
Jenkins, being a Java process, does have a bunch of knobs that can be used to adjust GC. How does Jenkins get started? What keeps it running?
Or, meta-question, how can I find this out for myself, and reduce the noise on the list? :)
frank
I'm not aware of any easy way to answer this without some prior knowledge. But here is how I'm finding the answer:
So first of all I know we like to use DJB's daemontools sometimes which provides a way to start and maintain services. If jenkins is started via daemontools then there will be a supervise process running with an argument probably called jenkins or the like which names a directory/link to directory under the daemontools service directory which is commonly found under /service/.
kencausey@box3-squeak:~$ ls -l /service/ total 0 lrwxrwxrwx 1 root root 14 Nov 14 03:23 ssdotcom -> /home/ssdotcom lrwxrwxrwx 1 root root 12 Jan 1 2013 tinydns -> /etc/tinydns
Neither of these seem relevant, so daemontools is not used.
The next lower level to check is in /etc/init.d/
kencausey@box3-squeak:~$ ls -l /etc/init.d/ ... -rwxr-xr-x 1 root root 5867 Jun 25 2012 jenkins ...
So the basic answer is that this program/script is run by the init system to start jenkins.
And in turn, init uses the script in /etc/init.d/ to do the work. The file /etc/init.d/README gives an overview. In the case of jenkins, the script /etc/init.d/jenkins does the work. You can stop the jenkins service by doing this (as root, or with sudo):
# /etc/init.d/jenkins stop
The more modern way now is to use the service command.
# service jenkins stop
and FYI
# service jenkins
will, assuming the standard is followed, provide a short help comment listing the applicable commands.
Likewise, start and restart will do the expected.
The jenkins script will be symlinked to directories associated with various system runlevels:
-rwxr-xr-x 1 root root 5867 Jun 25 2012 etc/init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc0.d/K01jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc1.d/K01jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc2.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc3.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc4.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc5.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc6.d/K01jenkins -> ../init.d/jenkins
The init system uses these (based on naming conventions) to figure out what scripts to start and stop, and in what order, when entering or leaving a runlevel.
Don't ever edit any of this stuff directly. But it would OK to run the /etc/init.d/jenkins script directly if you wanted to stop, start, or restart the service manually.
I'm not on the system now to check but if you read the jenkins startup script you may find that configuration variable, including blocking it from starting may be adjustable via a file in /etc/default/. The trend is towards getting away from modifying /etc/init.d/ scripts directly.
Dave
On 9 February 2014 19:56, Ken Causey ken@kencausey.com wrote:
-------- Original Message -------- Subject: Re: [Box-Admins] How does Jenkins start? From: "David T. Lewis" lewis@mail.msen.com Date: Sun, February 09, 2014 1:30 pm To: Squeak Hosting Support box-admins@lists.squeakfoundation.org
On Sun, Feb 09, 2014 at 11:57:23AM -0700, Ken Causey wrote:
-------- Original Message -------- Subject: [Box-Admins] How does Jenkins start? From: Frank Shearar frank.shearar@gmail.com Date: Sun, February 09, 2014 6:49 am To: Squeak Hosting Support box-admins@lists.squeakfoundation.org
I see all too often that builds fail because the JVM process runs out of PermGen space. Up until today I'd assumed that was a problem on the build agents. However, today I bounced my Mac mini (latest victim of this problem) and... nothing happened. But when I bounced _Jenkins_ the build succeeded.
Jenkins, being a Java process, does have a bunch of knobs that can be used to adjust GC. How does Jenkins get started? What keeps it running?
Or, meta-question, how can I find this out for myself, and reduce the noise on the list? :)
frank
I'm not aware of any easy way to answer this without some prior knowledge. But here is how I'm finding the answer:
So first of all I know we like to use DJB's daemontools sometimes which provides a way to start and maintain services. If jenkins is started via daemontools then there will be a supervise process running with an argument probably called jenkins or the like which names a directory/link to directory under the daemontools service directory which is commonly found under /service/.
kencausey@box3-squeak:~$ ls -l /service/ total 0 lrwxrwxrwx 1 root root 14 Nov 14 03:23 ssdotcom -> /home/ssdotcom lrwxrwxrwx 1 root root 12 Jan 1 2013 tinydns -> /etc/tinydns
Neither of these seem relevant, so daemontools is not used.
The next lower level to check is in /etc/init.d/
kencausey@box3-squeak:~$ ls -l /etc/init.d/ ... -rwxr-xr-x 1 root root 5867 Jun 25 2012 jenkins ...
So the basic answer is that this program/script is run by the init system to start jenkins.
And in turn, init uses the script in /etc/init.d/ to do the work. The file /etc/init.d/README gives an overview. In the case of jenkins, the script /etc/init.d/jenkins does the work. You can stop the jenkins service by doing this (as root, or with sudo):
# /etc/init.d/jenkins stop
The more modern way now is to use the service command.
# service jenkins stop
and FYI
# service jenkins
will, assuming the standard is followed, provide a short help comment listing the applicable commands.
Yep :) Thanks!
Likewise, start and restart will do the expected.
The jenkins script will be symlinked to directories associated with various system runlevels:
-rwxr-xr-x 1 root root 5867 Jun 25 2012 etc/init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc0.d/K01jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc1.d/K01jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc2.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc3.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc4.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 15 2012 etc/rc5.d/S03jenkins -> ../init.d/jenkins lrwxrwxrwx 1 root root 17 Oct 9 2012 etc/rc6.d/K01jenkins -> ../init.d/jenkins
The init system uses these (based on naming conventions) to figure out what scripts to start and stop, and in what order, when entering or leaving a runlevel.
Don't ever edit any of this stuff directly. But it would OK to run the /etc/init.d/jenkins script directly if you wanted to stop, start, or restart the service manually.
I'm not on the system now to check but if you read the jenkins startup script you may find that configuration variable, including blocking it from starting may be adjustable via a file in /etc/default/. The trend is towards getting away from modifying /etc/init.d/ scripts directly.
Yep. In particular, it looks like I want to set JENKINS_ARGS somewhere. I highly doubt that fiddling with /etc/default/jenkins is what I want so I'll just scrabble around a bit more remembering all I've forgotten about init.d style services.
frank
Dave
<snip>
Just so people know what I've been hacking on, Tony Garnock-Jones and I have seen many builds fail because PermGen runs out of space. This is a GC generation where the JVM stores class definitions. In vanilla Java classes only get set up at the start of a process, usually, so doesn't cause a problem. Script-y languages (and we use Groovy to trigger builds) generate classes at runtime... and that's a memory leak.
So I've added to the JAVA_ARGS in /etc/defaults/jenkins the arguments "-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled".
These allow classes to be unloaded. (I don't know why, but if you want CMSClassUnloadingEnabled to actually work, you _must_ also use UseConcMarkSweepGC. (An ex colleague of mine told me this _should_ be the default GC: it's simply much better, and he can't understand why it's not.)
Hopefully this will (a) result in a leaner Jenkins process and (b) (more importantly) improve the stability of the build processes.
Thanks to Ken & David for their help, as always.
frank
box-admins@lists.squeakfoundation.org