[Box-Admins] Proposed week-long shutdown of Jenkins

[Frank Shearar]

On 20 February 2014 22:30, Chris Cunnington <websela at yahoo.com> wrote:
> OK, well, nobody thinks that's a good idea, so it probably isn't.
>
> And yet box2 runs a version of Linux from 2006 or something like that.
>
> Sooo, perhaps there's a season for upgrades and I posit that after version of Squeak has been released may be that time.
> When Frank goes on his trip he knows Jenkins maybe a bit odd because it's been upgraded to a new version? No?
>
> In the seasons that are Squeak there probably needs to be a window for this kind of thing.

Maybe run the idea past Chris Muller, because CI impacts 4.5's release
process. But maybe not heavily: I _think_ I remember Chris & I
agreeing that a human would take a ReleaseSqueakTrunk artifact and
give it a final polish anyway, and we do have a release candidate out
that hasn't had any reported problems...

If Chris is OK with it, and releases 4.5, I think that's a fine time
to upgrade bits that need upgrading. We froze Jenkins' auto-upgrading
for a time, but it was a stopgap measure. I just won't be able to help
out with anything until March.

frank

> Chris
>
>
> On Feb 20, 2014, at 3:51 PM, David T. Lewis <lewis at mail.msen.com> wrote:
>
>> Ken, thanks for the explanation. I recognize all of those issues.
>>
>> I do think it is more appropriate to use the Jenkins UI to turn off the
>> problematic jobs until the issues can be addressed, as opposed to shutting
>> down the whole Jenkins system.
>>
>> Yes, some of our Jenkins jobs are wasting a lot of space. Yes, that is a
>> fixable problem. No I don't think that a bigger disk drive will fix it ;-)
>>
>> Dave
>>
>>> Let me just list the issues I'm aware of, not that these can all be
>>> fixed in the same way or require any significant overall downtime.
>>>
>>> 1. Jenkins broke some of our build processes with a release months ago.
>>>  Since that time we have been pinned to a specific release and have not
>>> updated.  Initially the plan was to be agile and keep up to date with
>>> Jenkins releases, but no one has found the time to figure out why the
>>> builds broke or at least the proper way to address the problem.  I know
>>> Frank tried but he has only so much time and other fish to fry.  I
>>> approached Chris C as he was the original instigator for Jenkins to try
>>> to see if he had the interest to help Frank out.
>>>
>>> 2. The issue I have harped on about in the past about filling up the
>>> filesystem on box3.  I'm convinced that Jenkins jobs are wasting space
>>> somewhere or that maybe there are some jobs that can be deleted?  I'm
>>> just speculating, but there are a number of jobs that have not succeeded
>>> in months.  By the way growth has been generally slow of late but we are
>>> at 97%, no immediate fear but 'vigilance!'.  If ultimately
>>> build.squeak.org is as big as it is because it has to be, then we
>>> probably need to approach SFC and see if there is budget to upgrade the
>>> disk space on box3.  That's not my first choice however.
>>>
>>> 3. The issue that Chris has referred to which is that we still get jobs
>>> stuck fairly regularly that have to be killed manually.
>>>
>>> Ken
>>>
>>> On 02/20/2014 11:11 AM, David T. Lewis wrote:
>>>> What problem are we trying to solve here?
>>>>
>>>> If there are Jenkins jobs that cause problems, and if those problems
>>>> cannot be addressed right away, then the appropriate thing to do is
>>>> disable them using the normal Jenkins console. If an explanation is
>>>> needed, just update the job description to say what is going on.
>>>>
>>>> A little bit of updating of the Jenkins job descriptions would do no
>>>> harm
>>>> in any case. Sort of like a class comment: "I am a Jenkins job that
>>>> tests
>>>> the FreebleBaz package. If I stop working, please contact
>>>> bilbo at baggins.org".
>>>>
>>>> :)
>>>>
>>>> Dave
>>>>
>>>>> Ken and I have been thinking of shutting down Jenkins (OK, it was my
>>>>> idea)
>>>>> for a week after 4.5 is released. The aim is to address hanging issues.
>>>>>
>>>>> A week is a long time from a technical point of view, but it allows
>>>>> people
>>>>> using it to take a break. Mainly we're thinking of Frank here.
>>>>> We're thinking of upgrades, disk usage, necessary and un-necessary
>>>>> builds
>>>>> (if there are any). Basically stopping that world for a week.
>>>>>
>>>>> What do you think, Frank? If you are opposed, then we'll chuck this
>>>>> idea.
>>>>>
>>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>