[Webteam] File cleanups, yes... again

Jason Rogers jacaetevha at gmail.com
Fri Dec 21 02:43:47 UTC 2007 

I have trimmed the usage down to 880 MB.  I don't have a solution
fully implemented yet, but I am working on it.

The Squeak Image that the website is running in is older (3.6 #5429)
so it doesn't have all of the classes I am used to anymore. :(

Anyway, I have a skeleton which I think is working.  Could someone
validate this for me?

I have added a method to SystemDictionary>>saveImageAsFileLike:

saveImageAsFileLike: aNameWithImageExtension
    " Smalltalk saveImageAsFileLike: 'backup.image' "
    (SourceFiles at: 2) ifNotNil:
        [self saveChangesInFileNamed: (self fullNameForChangesNamed:
aNameWithImageExtension].
    self saveImageInFileNamed: (self fullNameForImageNamed:
aNameWithImageExtension


My plan is to hook this into the current backup process we have via
the web page.  Does that work for everyone?

*~*~*~*~*~*~*~*

As far as disk clean up: I have removed most backups for working,
testing, and foundation.  I will change the backup scripts (via cron)
to save only the backup images as they are the only ones that will be
able to be restored.

Let me know if any of this is ludicrous or insufficient.

On Dec 20, 2007 4:18 PM, Jason Rogers <jacaetevha at gmail.com> wrote:
> I will be home tonight and able to take care of this.
>
>
> On Dec 20, 2007 4:16 PM, Ken Causey <ken at kencausey.com> wrote:
> > So?  What's the consensus?  I would like to make progress on this issue
> > and lighten the webteam's disk usage.
> >
> > Here is my understand of the system based on an examine of what I can
> > see on the server.  There appear to be three levels of backups
> > currently.
> >
> > 1.  I can see that for every site image there is also a
> > secondary .backup image.  My assumption is that whenever someone
> > requests that the image be saved (changes in the website) that the
> > previously saved image is renamed and the current image is saved.
> >
> > 2.  There is a cron script under the website account that periodically
> > builds a tarball from all .image and .changes files in the site
> > directories.  This unfortunately includes old temporary and failed
> > images that are not in fact used.  If nothing else, I would appreciate
> > this being cleaned up.
> >
> > 3.  The entire server is backed locally on a short term frequent (every
> > 4 hours) schedule and remotely on a more long term less frequent (daily)
> > schedule.  This of course includes the entire contents of the website
> > account.
> >
> > Again as a first priority I would appreciate a simple cleanup of old
> > undesirable files from the website account with a focus on those extra
> > image and changes files being repeatedly backed up.  Secondarily I
> > certainly support reexamining the webteam's internal backup policy,
> > particularly in light of the fact that the entire server is backed up
> > regularly.
> >
> > Ken
> >
> >
> > On Tue, 2007-12-18 at 15:05 -0500, Jason Rogers wrote:
> > > Oh... I misunderstood.  I was think to fork a process that would
> > > snapshot the image every so often and create/update the back up at
> > > that time.  I see what you are getting at with buttons in the Web Page
> > > and will do it that way.
> > >
> > > I don't think we need more than one copy of the backup, though.  Can
> > > you tell me why we should have multiple copies?
> > >
> > > Thanks.
> > >
> > > On Dec 18, 2007 2:58 PM, Karl <karl.ramberg at comhem.se> wrote:
> > > > Jason Rogers wrote:
> > > > > I could implement it.  It shouldn't be too hard right?  I don't know
> > > > > about how to save the image as another image name though.  Once I find
> > > > > how to do that it will be easy:
> > > > >
> > > > >   1. Capture current image name
> > > > >   2. Snapshot as backup image first
> > > > >   3. Snapshot as current image
> > > > >
> > > > > Right?  There aren't any gotchas are there?
> > > > >
> > > > Keep the backup and snapshot as two different buttons or issues. I think
> > > > most bad things happen to the image while editing and adding or deleting
> > > > features so it would be good to snapshot, see that everything is working
> > > > for a few days, then do a backup. Or do a backup before starting to
> > > > edit, and then edit, snapshot and wait a few days and then backup again ?
> > > >
> > > > Another issue is how many backup images do we need to keep ?  2 or 3 of
> > > > the most resent and delete the older ones ?
> > > >
> > > > Karl
> > > >
> > > > > On Dec 18, 2007 10:59 AM, Karl <karl.ramberg at comhem.se> wrote:
> > > > >
> > > > >> Jason Rogers wrote:
> > > > >>
> > > > >>> I will hop on as soon as I can to take care of this.  I am in New York
> > > > >>> right now and unable (company firewall) to access the box.  We really
> > > > >>> need a better backup policy in general, but I don't know what to do.
> > > > >>> Perhaps we don't use a Unix process at all.  We could schedule a
> > > > >>> process in the images that will snapshot the image as a current and a
> > > > >>> backup.
> > > > >>>
> > > > >>> What do you all think?
> > > > >>>
> > > > >>>
> > > > >> Sounds good. We already do manual image save on each change on the
> > > > >> Smallwiki process. Maybe a similar backup button would be enough ? Do
> > > > >> you want to implement it?
> > > > >> Karl
> > > > >>
> > > > >>
> > > > >>> On Dec 17, 2007 6:56 PM, Ken Causey <ken at kencausey.com> wrote:
> > > > >>>
> > > > >>>
> > > > >>>> On Tue, 2007-12-18 at 00:45 +0100, karl wrote:
> > > > >>>>
> > > > >>>>
> > > > >>>>> Ken Causey wrote:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>> We are climbing up above 90% disk usage on box2 so time for another
> > > > >>>>>> audit.  Previously I managed to talk you into a more conservative backup
> > > > >>>>>> schedule.  Now I would like to ask you to cleanup what is being backed
> > > > >>>>>> up.  A little nosing around indicates that you are backing up a lot of
> > > > >>>>>> files that I suspect were just used in setting up the sites/testing and
> > > > >>>>>> or just junk at this point:
> > > > >>>>>>
> > > > >>>>>> # tar ztf backups/foundation/2007-12-17-0005.tgz
> > > > >>>>>> SqF-Pier-1.5-maybe-bad.image
> > > > >>>>>> SqF-Pier-1.5-safe.image
> > > > >>>>>> SqF-Pier-1.5-safe.old.image
> > > > >>>>>> SqF-Pier-1.5.image
> > > > >>>>>> SqF-Pier-1.5.old.image
> > > > >>>>>> SqueakFoundation.image
> > > > >>>>>> SqF-Pier-1.5-maybe-bad.changes
> > > > >>>>>> SqF-Pier-1.5-safe.changes
> > > > >>>>>> SqF-Pier-1.5-safe.old.changes
> > > > >>>>>> SqF-Pier-1.5.changes
> > > > >>>>>> SqF-Pier-1.5.old.changes
> > > > >>>>>> SqueakFoundation.changes
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>> Squeak foundation images are not used at all. Brad Fuller put a lot of
> > > > >>>>> effort into it but the foundation is a a few pages in the squeak.org
> > > > >>>>> image. You can delete foundation directory and backups
> > > > >>>>>
> > > > >>>>>
> > > > >>>> I'd prefer if Brad could confirm he has no more interest in any of that
> > > > >>>> content and one of you take care of it.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>>> # tar ztf backups/2007-12-17-0005.tgz
> > > > >>>>>> smallwikiSnapshot.1.image
> > > > >>>>>> smallwikiSnapshot.backup.image
> > > > >>>>>> smallwikiSnapshot.image
> > > > >>>>>> smallwikiSnapshot.1.changes
> > > > >>>>>> smallwikiSnapshot.backup.changes
> > > > >>>>>> smallwikiSnapshot.changes
> > > > >>>>>>
> > > > >>>>>> s# tar ztf backups/testing/2007-12-17-0005.tgz
> > > > >>>>>> smallwikiSnapshot.backup.image
> > > > >>>>>> smallwikiSnapshot.image
> > > > >>>>>> wwwtest.squeak.org.backup.image
> > > > >>>>>> wwwtest.squeak.org.image
> > > > >>>>>> smallwikiSnapshot.backup.changes
> > > > >>>>>> smallwikiSnapshot.changes
> > > > >>>>>> wwwtest.squeak.org.backup.changes
> > > > >>>>>> wwwtest.squeak.org.changes
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>> I'm not at all sure how backup is run. I screwed up the squeak.org image
> > > > >>>>> a few years back and found that the images backed up were useless
> > > > >>>>> because they were copied from a unix process on a running image I think.
> > > > >>>>> We need backup of squeak.org image.
> > > > >>>>>
> > > > >>>>>
> > > > >>>> That's fine, but even the backup of the main site involves backing up 3
> > > > >>>> image and changes file sets.  I can maybe imagine 2 sets (current and
> > > > >>>> previous to last modification), but 3?
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> The wwwtest.squeak.org image we hardly use anymore, but it is good for
> > > > >>>>> testing major changes to style scripts etc. wwwtest.squeak.org does not
> > > > >>>>> need backup now. I guess we can turn backup on when someone get the urge
> > > > >>>>> to hack at stuff.
> > > > >>>>>
> > > > >>>>>
> > > > >>>> Either that or just scale back the extent to which wwwtest is backed up.
> > > > >>>> Again, I'm primarily concerned about the backing up of files which never
> > > > >>>> change.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>>> Are all of these files needed at all, much less needing to be backed up
> > > > >>>>>> over and over again?
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>> I guess not
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>> The home directory for the website team totals 4.3GB.  Since there is an
> > > > >>>>>> rsync backup also on the server that is doubled, and then any images
> > > > >>>>>> that change are backed up in their entirety again.  So in effect the
> > > > >>>>>> website team ends up using perhaps as much as 10GB on the server.
> > > > >>>>>> Anything you can do to lower this I would greatly appreciate.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>> I think you can delete all the files I mentioned.
> > > > >>>>>
> > > > >>>>>
> > > > >>>> I'd rather not delete anything myself.  However designed the backup
> > > > >>>> process of course would need to change that.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>>
> > > > >>>> Ken
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> Karl
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>> _______________________________________________
> > > > >>>> Webteam mailing list
> > > > >>>> Webteam at lists.squeakfoundation.org
> > > > >>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
>
>
>
> --
> Jason Rogers
>
> "I am crucified with Christ: nevertheless I live;
> yet not I, but Christ liveth in me: and the life
> which I now live in the flesh I live by the faith of
> the Son of God, who loved me, and gave
> himself for me."
>     Galatians 2:20
>



-- 
Jason Rogers

"I am crucified with Christ: nevertheless I live;
yet not I, but Christ liveth in me: and the life
which I now live in the flesh I live by the faith of
the Son of God, who loved me, and gave
himself for me."
    Galatians 2:20

More information about the Webteam mailing list