[Box-Admins] Re: [Board] Disk space is low on box3

[David T. Lewis]

On Sun, Nov 02, 2014 at 09:31:31AM -0600, Chris Muller wrote:
> On Sat, Nov 1, 2014 at 8:07 PM, Levente Uzonyi <leves at elte.hu> wrote:
> > Hi All,
> >
> > There's less than 3GB (out of 60) free disk space at the moment on box3, and
> > it keeps decreasing.
> >
> > What services are running on box3?
> > - squeaksource.com which currently uses about 17GB
> > - build.squeak.org - aka jenkins - which uses about 35 GB at the moment
> > - apache2 as a frontend for the websites, which uses negligible disk space
> > - a yet unused but configured nameserver, which uses negligible disk space
> > - a seemingly unused ftp server, which should probably be removed, but it
> > uses negligible disk space anyway
> >
> > What are the possible solutions?
> > - increase the size of the disk. This can be done by the SFC, if the Board
> > requests it, and the SFC is willing to do it. In theory this can be done on
> > the fly - without stopping the server[1].
> >         How large should the increase be?
> >         - It depends on what the plans about jenkins are, but the more the
> > merrier.
> > - move squeaksource.com to box4. box3 was planned to host nothing but
> > jenkins. There's plenty of space left on box4, so this is something we can
> > do fairly easily.
> > - clean up jenkins. We can keep less build artifacts, delete unused projects
> > (if any), try to remove unused versions of softwares, try to share files
> > using hard links, etc. But this is probably what requires the most effort.
> 
> Why do we need to devote more than HALF of box3 to Jenkins?  35GB
> of....  what?  For goodness sake it runs the tests in a 20MB image and
> that's it!  Why in the world does it need so much space?  And what
> valuable output of Jenkins are we as a community, consuming from it
> that we should expend resources to "keep it growing" instead of
> putting it on a diet?


The ongoing growth is mainly from our Jenkins jobs. I don't see any obvious
way to tell Jenkins to clean up after itself, and google does not offer
any enlightenment. I suspect that most Jenkins sites may just be in
the habit of cleaning up with cron jobs or perhaps even manually. We
might need to do the same.

Just as an example, if we remove the older entries from ~jenkins/jobs/SqueakTrunk,
we would free up 1GB from that job alone. But I think that all of the jobs
require attention.

I think a few guidelines might help. My suggestions:

- Each Jenkin job must have an owner, identified in some recognizable way (such
as in the job description on the build.squeak.org web page).

- The job owner is responsible for making sure the job gets cleaned up once in
a while, either by doing the cleanup or by asking for help to get it done (not
all of us are unix hackers, and that might be what is needed to clean some of
these things up).

- Nothing should be removed without permission of the job owner.

- If the job owner cannot be identified or contacted, then the job may be
archived and/or disabled as needed.


Dave