and in danger (was Re: [Box-Admins] Disk space usage on box3)

Frank Shearar frank.shearar at
Sun Dec 8 10:50:05 UTC 2013

On 7 December 2013 22:58, Chris Muller <asqueaker at> wrote:
> Hi guys -- First, Ken, thanks again for setting a reference example of
> how we need to be thinking about our systems from an admin-support
> perspective.  Taking a proactive tact is not just excellent service,
> there are many intangible benefits like image and community
> constitution.
> I totally agree with Dave.  Pruning SqueakSource may be something we
> might want to consider in the future, but not now, and never simply
> because we're low on disk space.  First because pruning empty projects
> won't recover anything significant, secondly because we're in a
> "archival preservation" mode right now with SqueakSource -- not a mode
> to be making significant changes to it.
> How many times have we said, "disk space is cheap"?  We cannot go back
> on that now!  :)
> Frank wrote:
>> As a separate step, we can think about how to both produce versioned
>> artifacts (i.e., zipfiles with versions in their file names) and not
>> eat all the disk space.
> I thought about it 4 years ago and made an object model to take care
> of it.  The problem with zip-files is how wasteful they are.  When
> someone changes one single method of the Morphic package, the other 1K
> definitions (however many there are) are duplicated in new zip (mcz)
> file.  By contrast, the object model refers to the same canonicalized
> MCDefinition instances across Versions, adding only one new
> MCDefinition to the bulk of the model in that example.

That's very cool, but it's not the zips we're talking about. We're
talking about the artifacts produced by Jenkins, which are Squeak
images. I wrote the release process to produce a series of artifacts
with different names. Because nothing gets replaced/overwritten, disk
usage is unbound.

> The result is that the redundant Magma-backed copy of
> consumes less than 1/4th the space of the original
> File based version.  A Magma-backed copy of would
> about 12GB of space.
> For now, though, Frank has the ability and responsibility to trim up
> the jenkins stuff.  Thanks Frank.

I nuked the ReleaseSqueakTrunk target directory, freeing up 5.9 GB of
space. (We should always consider the target/ directory of every job
as being evanescent. Feel free to nuke these at any time. Every job
should be written such that it can reconstitute its working
environment in target/.

> PS -- For interest, I just kicked off a bulk-load of entire
> repository into Magma to see how much space it will
> take..
> On Sat, Dec 7, 2013 at 9:37 AM, David T. Lewis <lewis at> wrote:
>> On Sat, Dec 07, 2013 at 10:17:06AM +0000, Frank Shearar wrote:
>>> I mailed Ken separately, but the time zone happy conjunction must have
>>> just passed. I'd have solved the problem (wipe out the
>>> ReleaseSqueakTrunk/target/ directory (5.9GB)) except that I can't sudo
>>> because I don't know the password for the account (because I don't
>>> think I ever actually set it)! If some kind soul can change it and let
>>> me know the password, I'll (a) change it to something only I know and
>>> (b) wipe out the directory causing the problem.
>>> As a separate step, we can think about how to both produce versioned
>>> artifacts (i.e., zipfiles with versions in their file names) and not
>>> eat all the disk space.
>>> frank
>> Hi Frank,
>> Cool, I think we just came to the identical conclusions :) I was just
>> about to send the following email to you before I read this, so here
>> is what I was going to say:
>> I think we can pretty easily free up a bunch of disk space. It turns out
>> that our SqueakTrunk job is currently using 44% of the entire disk space
>> on box3, and a lot of that can probably be freed without harming anything
>> for the SqueakTrunk job itself.
>> Here are some things I think we can do, but I want to run it by you before
>> actually changing anything:
>> 1) The build artifacts are TrunkImage.changes, TrunkImage.image,
>> TrunkImage.manifest and TrunkImage.version. The image and changes files
>> take most of the space, so we could add a build step to compress them
>> like this:
>> $ zip TrunkImage.changes TrunkImage.image TrunkImage.manifest TrunkImage.version
>>   adding: TrunkImage.changes (deflated 77%)
>>   adding: TrunkImage.image (deflated 54%)
>>   adding: TrunkImage.manifest (deflated 54%)
>>   adding: TrunkImage.version (stored 0%)
>> Then we can specify as the build artifact. This will save
>> a lot of disk space in the future.

I've implemented this for ReleaseSqueakTrunk, which will produce an
artifact named This will contain a properly named image
& changes file of the form Squeak4.5-NNNN.

>> 2) The Jenkins job is saving all of the build artifacts since we began
>> running the job.  I'm not sure if that's what you want it to do, but if
>> the old artifacts are not needed, then we can change the job configuration.
>> In the "Archive the Artifacts" section of the job configuration, there
>> is a setting for "Discard all but the last successful/stable artifact to
>> save disk space". That might more aggressive that we want, but there must
>> be some setting that would let us trim the archives down to what we really
>> need.
>> 3) If we run out of disk space and need to take emergency action, we can
>> just compress the older build artifacts (from the unix command line). It's
>> probably not good to do this outside of Jenkins tools, but at least we would
>> not lose the actual data, and it would free up a lot of disk space right away.
>> 4) If we don't need all of the historical artifacts, and if we can't figure
>> out how to trim them down through Jenkins job configurations, then I can
>> delete the older ones from the unix command line.
>> 5) Not directly related to disk space, but we should probably also enable the
>> "Abort the build if it's stuck" option under "Build Environment". We can set
>> it to time out after 30 minutes or so, and I think that might cure our problem
>> with stuck ruby and squeakvm processes.
>> Dave
>>> On 7 December 2013 01:23, David T. Lewis <lewis at> wrote:
>>> > On Fri, Dec 06, 2013 at 11:40:54AM -0600, Ken Causey wrote:
>>> >> On 12/06/2013 11:19 AM, David T. Lewis wrote:
>>> >> >Thanks Ken,
>>> >> >
>>> >> >I will look at it as soon as I get home, about 8 hours from now.
>>> >> >
>>> >> >Frank,
>>> >> >
>>> >> >There's not much I can do on the side other than move the
>>> >> >repository to another box (which is not easy to do). Short term we need to
>>> >> >tidy up the jobs where can can.
>>> >> >
>>> >> >Dave
>>> >>
>>> >> Thanks Dave,
>>> >>
>>> >> I agree that it is probably easier to find some space to clear up under
>>> >> but I think the community as a whole has to give some
>>> >> serious thought to the future of
>>> >
>>> > I fully agree that the community should give some consideration to how
>>> > should be managed moving forward. But please do not
>>> > portray this as a disk space problem. If that is the problem, then I'll
>>> > pay for the disk space myself, just tell me where to send the check.
>>> >
>>> > The disk utilization problem is due to unnecessary accumulation of build
>>> > artifacts from Jenkins jobs. It looks to me like most of this is accumulating
>>> > by accident rather than by intent, and this can probably be easily fixed
>>> > with some changes to the job configurations, with no loss of useful data
>>> > from the jobs themselves. Clearly this needs to be addressed anyway,
>>> > because if you doubled our available disk space we would be having the
>>> > same discussion 12 months from now. So we need to fix it.
>>> >
>>> > I'll try to get with Frank over the weekend and see if we can clean up
>>> > some easy stuff (Frank, I am "dtlewis290 at" on gmail, so I'll
>>> > try to connect with you there).
>>> >
>>> > Meanwhile I deleted a few unnecessary backup files under ~ssdotcom, which
>>> > gives us another 1% free disk space to keep things going for another day
>>> > or so ;-)
>>> >
>>> > Dave
>>> >

More information about the Box-Admins mailing list