[Webteam] File cleanups, yes... again

Jason Rogers jacaetevha at gmail.com
Thu Dec 20 21:18:31 UTC 2007 

I will be home tonight and able to take care of this.

On Dec 20, 2007 4:16 PM, Ken Causey <ken at kencausey.com> wrote:
> So?  What's the consensus?  I would like to make progress on this issue
> and lighten the webteam's disk usage.
>
> Here is my understand of the system based on an examine of what I can
> see on the server.  There appear to be three levels of backups
> currently.
>
> 1.  I can see that for every site image there is also a
> secondary .backup image.  My assumption is that whenever someone
> requests that the image be saved (changes in the website) that the
> previously saved image is renamed and the current image is saved.
>
> 2.  There is a cron script under the website account that periodically
> builds a tarball from all .image and .changes files in the site
> directories.  This unfortunately includes old temporary and failed
> images that are not in fact used.  If nothing else, I would appreciate
> this being cleaned up.
>
> 3.  The entire server is backed locally on a short term frequent (every
> 4 hours) schedule and remotely on a more long term less frequent (daily)
> schedule.  This of course includes the entire contents of the website
> account.
>
> Again as a first priority I would appreciate a simple cleanup of old
> undesirable files from the website account with a focus on those extra
> image and changes files being repeatedly backed up.  Secondarily I
> certainly support reexamining the webteam's internal backup policy,
> particularly in light of the fact that the entire server is backed up
> regularly.
>
> Ken
>
>
> On Tue, 2007-12-18 at 15:05 -0500, Jason Rogers wrote:
> > Oh... I misunderstood.  I was think to fork a process that would
> > snapshot the image every so often and create/update the back up at
> > that time.  I see what you are getting at with buttons in the Web Page
> > and will do it that way.
> >
> > I don't think we need more than one copy of the backup, though.  Can
> > you tell me why we should have multiple copies?
> >
> > Thanks.
> >
> > On Dec 18, 2007 2:58 PM, Karl <karl.ramberg at comhem.se> wrote:
> > > Jason Rogers wrote:
> > > > I could implement it.  It shouldn't be too hard right?  I don't know
> > > > about how to save the image as another image name though.  Once I find
> > > > how to do that it will be easy:
> > > >
> > > >   1. Capture current image name
> > > >   2. Snapshot as backup image first
> > > >   3. Snapshot as current image
> > > >
> > > > Right?  There aren't any gotchas are there?
> > > >
> > > Keep the backup and snapshot as two different buttons or issues. I think
> > > most bad things happen to the image while editing and adding or deleting
> > > features so it would be good to snapshot, see that everything is working
> > > for a few days, then do a backup. Or do a backup before starting to
> > > edit, and then edit, snapshot and wait a few days and then backup again ?
> > >
> > > Another issue is how many backup images do we need to keep ?  2 or 3 of
> > > the most resent and delete the older ones ?
> > >
> > > Karl
> > >
> > > > On Dec 18, 2007 10:59 AM, Karl <karl.ramberg at comhem.se> wrote:
> > > >
> > > >> Jason Rogers wrote:
> > > >>
> > > >>> I will hop on as soon as I can to take care of this.  I am in New York
> > > >>> right now and unable (company firewall) to access the box.  We really
> > > >>> need a better backup policy in general, but I don't know what to do.
> > > >>> Perhaps we don't use a Unix process at all.  We could schedule a
> > > >>> process in the images that will snapshot the image as a current and a
> > > >>> backup.
> > > >>>
> > > >>> What do you all think?
> > > >>>
> > > >>>
> > > >> Sounds good. We already do manual image save on each change on the
> > > >> Smallwiki process. Maybe a similar backup button would be enough ? Do
> > > >> you want to implement it?
> > > >> Karl
> > > >>
> > > >>
> > > >>> On Dec 17, 2007 6:56 PM, Ken Causey <ken at kencausey.com> wrote:
> > > >>>
> > > >>>
> > > >>>> On Tue, 2007-12-18 at 00:45 +0100, karl wrote:
> > > >>>>
> > > >>>>
> > > >>>>> Ken Causey wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>>> We are climbing up above 90% disk usage on box2 so time for another
> > > >>>>>> audit.  Previously I managed to talk you into a more conservative backup
> > > >>>>>> schedule.  Now I would like to ask you to cleanup what is being backed
> > > >>>>>> up.  A little nosing around indicates that you are backing up a lot of
> > > >>>>>> files that I suspect were just used in setting up the sites/testing and
> > > >>>>>> or just junk at this point:
> > > >>>>>>
> > > >>>>>> # tar ztf backups/foundation/2007-12-17-0005.tgz
> > > >>>>>> SqF-Pier-1.5-maybe-bad.image
> > > >>>>>> SqF-Pier-1.5-safe.image
> > > >>>>>> SqF-Pier-1.5-safe.old.image
> > > >>>>>> SqF-Pier-1.5.image
> > > >>>>>> SqF-Pier-1.5.old.image
> > > >>>>>> SqueakFoundation.image
> > > >>>>>> SqF-Pier-1.5-maybe-bad.changes
> > > >>>>>> SqF-Pier-1.5-safe.changes
> > > >>>>>> SqF-Pier-1.5-safe.old.changes
> > > >>>>>> SqF-Pier-1.5.changes
> > > >>>>>> SqF-Pier-1.5.old.changes
> > > >>>>>> SqueakFoundation.changes
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Squeak foundation images are not used at all. Brad Fuller put a lot of
> > > >>>>> effort into it but the foundation is a a few pages in the squeak.org
> > > >>>>> image. You can delete foundation directory and backups
> > > >>>>>
> > > >>>>>
> > > >>>> I'd prefer if Brad could confirm he has no more interest in any of that
> > > >>>> content and one of you take care of it.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>>> # tar ztf backups/2007-12-17-0005.tgz
> > > >>>>>> smallwikiSnapshot.1.image
> > > >>>>>> smallwikiSnapshot.backup.image
> > > >>>>>> smallwikiSnapshot.image
> > > >>>>>> smallwikiSnapshot.1.changes
> > > >>>>>> smallwikiSnapshot.backup.changes
> > > >>>>>> smallwikiSnapshot.changes
> > > >>>>>>
> > > >>>>>> s# tar ztf backups/testing/2007-12-17-0005.tgz
> > > >>>>>> smallwikiSnapshot.backup.image
> > > >>>>>> smallwikiSnapshot.image
> > > >>>>>> wwwtest.squeak.org.backup.image
> > > >>>>>> wwwtest.squeak.org.image
> > > >>>>>> smallwikiSnapshot.backup.changes
> > > >>>>>> smallwikiSnapshot.changes
> > > >>>>>> wwwtest.squeak.org.backup.changes
> > > >>>>>> wwwtest.squeak.org.changes
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>> I'm not at all sure how backup is run. I screwed up the squeak.org image
> > > >>>>> a few years back and found that the images backed up were useless
> > > >>>>> because they were copied from a unix process on a running image I think.
> > > >>>>> We need backup of squeak.org image.
> > > >>>>>
> > > >>>>>
> > > >>>> That's fine, but even the backup of the main site involves backing up 3
> > > >>>> image and changes file sets.  I can maybe imagine 2 sets (current and
> > > >>>> previous to last modification), but 3?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> The wwwtest.squeak.org image we hardly use anymore, but it is good for
> > > >>>>> testing major changes to style scripts etc. wwwtest.squeak.org does not
> > > >>>>> need backup now. I guess we can turn backup on when someone get the urge
> > > >>>>> to hack at stuff.
> > > >>>>>
> > > >>>>>
> > > >>>> Either that or just scale back the extent to which wwwtest is backed up.
> > > >>>> Again, I'm primarily concerned about the backing up of files which never
> > > >>>> change.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>>> Are all of these files needed at all, much less needing to be backed up
> > > >>>>>> over and over again?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>> I guess not
> > > >>>>>
> > > >>>>>
> > > >>>>>> The home directory for the website team totals 4.3GB.  Since there is an
> > > >>>>>> rsync backup also on the server that is doubled, and then any images
> > > >>>>>> that change are backed up in their entirety again.  So in effect the
> > > >>>>>> website team ends up using perhaps as much as 10GB on the server.
> > > >>>>>> Anything you can do to lower this I would greatly appreciate.
> > > >>>>>>
> > > >>>>>>
> > > >>>>> I think you can delete all the files I mentioned.
> > > >>>>>
> > > >>>>>
> > > >>>> I'd rather not delete anything myself.  However designed the backup
> > > >>>> process of course would need to change that.
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Ken
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> Karl
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>> _______________________________________________
> > > >>>> Webteam mailing list
> > > >>>> Webteam at lists.squeakfoundation.org
> > > >>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
>



-- 
Jason Rogers

"I am crucified with Christ: nevertheless I live;
yet not I, but Christ liveth in me: and the life
which I now live in the flesh I live by the faith of
the Son of God, who loved me, and gave
himself for me."
    Galatians 2:20

More information about the Webteam mailing list