Box-Admins February 2011

box-admins@lists.squeakfoundation.org

11 participants
26 discussions

RE: Backup solutions (was Re: [Box-Admins] Permissions)
by Ken Causey 07 Feb '11

07 Feb '11

I said: > Your memory is failing. Let me nuance this a bit. I certainly did not use the figure of 500GB recently. It is possible that I've used that figure in the past as there is more than one way to look at this and I could have made a mistake and counted the entire filesystem size (including the backups). Originally our plan was to support the usual default rsnapshot backup plan. IIRC this is 6 hourly backups 6 * 2GB = 12 GB 7 daily backups 7 * 3GB = 21 GB 4 weekly backups 4 * 3GB = 12 GB 6 monthly backups 6 * 4GB = 24 GB So based on an educated guess this 'full' backup scheme would likely come to about 60 (one full, the rest increment)+12+21+12+24 = 129GB. Also because the backup does not know why a file is deleted and those deleted files remain in the backup until manually deleted, the backup continuously grows, this is why the local backup is about 70GB now. Anyway, it's entirely possible that I said 500GB at some point in the past based on quick estimation and pessimism. Ken

2 1

RE: [Box-Admins] My daily rituals (was RE: Welcome Steve and Casey)
by Ken Causey 07 Feb '11

07 Feb '11

I thought about mentioning automation because I don't really think it's a good idea in this case. What's important here is often tiny details and I know from experience that regularly emailed reports that are much the same all the time get very little scrutiny. The fact that you have to do something to get the information, I believe, causes you to look more closely. Also, my summary here is a bit simplistic since I gloss over the variations that occur when I think I have found something out of the norm. This happens regularly but it's rarely anything worthy of note once I look; but I have to look more closely to know. And how to look more closely varies considerably based on what is found. The human element is essential in my opinion, until we achieve hard AI goals anyway. Ken > -------- Original Message -------- > Subject: Re: [Box-Admins] My daily rituals (was RE: Welcome Steve and > Casey) > From: Steven Elkins <sgelkins(a)gmail.com> > Date: Sat, February 05, 2011 4:37 pm > To: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > > > No, I don't take it as directed at me. I think it's important to to say > what I have time to do. If it isn't enough, fine, no worries. I can commit > to 10 minutes every few days. What I can't promise is that if there's a > problem, I'll always have time to investigate and/or fix it. Often I would > be checking just before leaving for work. > > Thanks for the summary of responsibilities. A lot of it sounds like a job > for a script run via cron (or whatever), with saved reports and/or email to > this list. > > Cheers, > Steve

5 7

RE: [Box-Admins] Permissions
by Ken Causey 07 Feb '11

07 Feb '11

So it was not as bad as I had feared. An incremental backup from 7/6/10 to today: approximately 5GB of additional filespace used, backup time over 10mbit cable connection (DE to US) about 2 hrs 40 min. Again, this is not the entire filesystem but it covers nearly all custom content. Ken > -------- Original Message -------- > Subject: RE: [Box-Admins] Permissions > From: "Ken Causey" <ken(a)kencausey.com> > Date: Sun, February 06, 2011 10:33 am > To: "Squeak Hosting Support" <box-admins(a)lists.squeakfoundation.org> > > > Forget the figures below regarding the time it has taken me to do an > incremental. I screwed up and did not do an incremental but a complete > copy (pathing mistake). I'm going to start it again (right I hope) and > I'll let you know. > > Ken > > > -------- Original Message -------- > > Subject: RE: [Box-Admins] Permissions > > From: "Ken Causey" <ken(a)kencausey.com> > > Date: Sun, February 06, 2011 10:27 am > > To: "Squeak Hosting Support" <box-admins(a)lists.squeakfoundation.org> > > > > > > > -------- Original Message -------- > > > Subject: Re: [Box-Admins] Permissions > > > From: Göran Krampe <goran(a)krampe.se> > > > Date: Sun, February 06, 2011 2:15 am > > > To: box-admins(a)lists.squeakfoundation.org > > > > > > > > > Hi all! > > > > > > Yes, we used rsnapshot earlier, but all though it uses rsync (good for > > > partial file changes) it ends up using hardlinks etc on the target and > > > thus consumes quite a bit of space there. > > > > > > For my local needs (laptop onto a USB vfat drive) I ended up picking > > > Duplicity the other day. But either way we want a solution that > > > transfers only modifications. > > > > > > I would be happy with a solution that simply keeps an offsite "mirror". > > > Rsnapshot gave us "history" too, and sure, if we can use it then it is > > > probably a good choice. Ken suspected transfers were large, but I am > > > unsure, I hope they are not and that it uses rsync for the actual transfer. > > > > There is no doubt that it is using rsync, it's just a fancy shell script > > really and it logs the major commands it executes in the log file > > (/var/log/rsnapshot.log). As such my early speculation that whole files > > are being sent is probably wrong, unless some special invocation of > > rsync is required for deltas to be used (I can't claim to be an rsync > > expert.) > > > > However, as I mentioned separately I'm trying to update my local backup > > of the primary server content in much the same way rsnapshot does, but > > manually (using rsync). I have an, in theory, 10mbit Internet > > connection, of which I really get more like 5-7mbit downloads. The > > incremental (since last July) of /home/ took over 6 hours and actually > > it doesn't look right to me, I'm going to have to do it again. I've > > started a sync of the other stuff now and we'll see how that goes and > > then I'll get back to /home/. My connection is asynchronous like most > > home connection (probably 3/4 mbit), so I wonder if there is a lot of > > two way traffic involved which could explain the time it is taking. I > > haven't done the math though to see what is reasonable. > > > > > > > > Currently I don't have a proper broadband, but if we set up something > > > that can run "night time" off to people "home computers", then that > > > should be enough I think. The reason we used a server that my former > > > employer had was that it was also a Hetzner box so the connection was > > > really good. > > > > > > regards, Göran

3 4

Welcome Steve and Casey
by Ken Causey 07 Feb '11

07 Feb '11

Welcome to the box-admins mailing list. Feel free to introduce yourselves (again perhaps) if you like; if you have any particular interest or thoughts related to managing the community services please contribute. Thanks, Ken

4 3

RE: [Box-Admins] Permissions
by Ken Causey 06 Feb '11

06 Feb '11

Forget the figures below regarding the time it has taken me to do an incremental. I screwed up and did not do an incremental but a complete copy (pathing mistake). I'm going to start it again (right I hope) and I'll let you know. Ken > -------- Original Message -------- > Subject: RE: [Box-Admins] Permissions > From: "Ken Causey" <ken(a)kencausey.com> > Date: Sun, February 06, 2011 10:27 am > To: "Squeak Hosting Support" <box-admins(a)lists.squeakfoundation.org> > > > > -------- Original Message -------- > > Subject: Re: [Box-Admins] Permissions > > From: Göran Krampe <goran(a)krampe.se> > > Date: Sun, February 06, 2011 2:15 am > > To: box-admins(a)lists.squeakfoundation.org > > > > > > Hi all! > > > > Yes, we used rsnapshot earlier, but all though it uses rsync (good for > > partial file changes) it ends up using hardlinks etc on the target and > > thus consumes quite a bit of space there. > > > > For my local needs (laptop onto a USB vfat drive) I ended up picking > > Duplicity the other day. But either way we want a solution that > > transfers only modifications. > > > > I would be happy with a solution that simply keeps an offsite "mirror". > > Rsnapshot gave us "history" too, and sure, if we can use it then it is > > probably a good choice. Ken suspected transfers were large, but I am > > unsure, I hope they are not and that it uses rsync for the actual transfer. > > There is no doubt that it is using rsync, it's just a fancy shell script > really and it logs the major commands it executes in the log file > (/var/log/rsnapshot.log). As such my early speculation that whole files > are being sent is probably wrong, unless some special invocation of > rsync is required for deltas to be used (I can't claim to be an rsync > expert.) > > However, as I mentioned separately I'm trying to update my local backup > of the primary server content in much the same way rsnapshot does, but > manually (using rsync). I have an, in theory, 10mbit Internet > connection, of which I really get more like 5-7mbit downloads. The > incremental (since last July) of /home/ took over 6 hours and actually > it doesn't look right to me, I'm going to have to do it again. I've > started a sync of the other stuff now and we'll see how that goes and > then I'll get back to /home/. My connection is asynchronous like most > home connection (probably 3/4 mbit), so I wonder if there is a lot of > two way traffic involved which could explain the time it is taking. I > haven't done the math though to see what is reasonable. > > > > > Currently I don't have a proper broadband, but if we set up something > > that can run "night time" off to people "home computers", then that > > should be enough I think. The reason we used a server that my former > > employer had was that it was also a Hetzner box so the connection was > > really good. > > > > regards, Göran

1 0

RE: [Box-Admins] Permissions
by Ken Causey 06 Feb '11

06 Feb '11

> -------- Original Message -------- > Subject: Re: [Box-Admins] Permissions > From: Göran Krampe <goran(a)krampe.se> > Date: Sun, February 06, 2011 2:15 am > To: box-admins(a)lists.squeakfoundation.org > > > Hi all! > > Yes, we used rsnapshot earlier, but all though it uses rsync (good for > partial file changes) it ends up using hardlinks etc on the target and > thus consumes quite a bit of space there. > > For my local needs (laptop onto a USB vfat drive) I ended up picking > Duplicity the other day. But either way we want a solution that > transfers only modifications. > > I would be happy with a solution that simply keeps an offsite "mirror". > Rsnapshot gave us "history" too, and sure, if we can use it then it is > probably a good choice. Ken suspected transfers were large, but I am > unsure, I hope they are not and that it uses rsync for the actual transfer. There is no doubt that it is using rsync, it's just a fancy shell script really and it logs the major commands it executes in the log file (/var/log/rsnapshot.log). As such my early speculation that whole files are being sent is probably wrong, unless some special invocation of rsync is required for deltas to be used (I can't claim to be an rsync expert.) However, as I mentioned separately I'm trying to update my local backup of the primary server content in much the same way rsnapshot does, but manually (using rsync). I have an, in theory, 10mbit Internet connection, of which I really get more like 5-7mbit downloads. The incremental (since last July) of /home/ took over 6 hours and actually it doesn't look right to me, I'm going to have to do it again. I've started a sync of the other stuff now and we'll see how that goes and then I'll get back to /home/. My connection is asynchronous like most home connection (probably 3/4 mbit), so I wonder if there is a lot of two way traffic involved which could explain the time it is taking. I haven't done the math though to see what is reasonable. > > Currently I don't have a proper broadband, but if we set up something > that can run "night time" off to people "home computers", then that > should be enough I think. The reason we used a server that my former > employer had was that it was also a Hetzner box so the connection was > really good. > > regards, Göran

1 0

RE: [Box-Admins] Permissions
by Ken Causey 06 Feb '11

06 Feb '11

Further clarification: The total local backup currently takes up 75GB (clearly cleaning that up needs to be on the todo list) We also make a backup of squeaksource.com which comes to 9GB currently. That was supposed to be reciprocal but they couldn't get it to work. Ken > -------- Original Message -------- > Subject: RE: [Box-Admins] Permissions > From: "Ken Causey" <ken(a)kencausey.com> > Date: Sat, February 05, 2011 3:21 pm > To: "Squeak Hosting Support" <box-admins(a)lists.squeakfoundation.org> > Cc: Chris Muller <ma.chris.m(a)gmail.com> > > > I said: > > Well, that depends on what you feel needs to be backed up. Some might > > feel the entire system should be backed up and that is currently 122GB. > > I'm satisfied if /home/ /etc/ /usr/local/ /var/www/ /root/ and > > /var/lib/mailman/ are backed up at minimum. > > Oops, 122GB includes the local backup. You can at least take off a > third of that and perhaps half. > > Ken

2 1

My daily rituals (was RE: Welcome Steve and Casey)
by Ken Causey 06 Feb '11

06 Feb '11

Please don't take this as directed at you Steve, because it definitely is not, but I'm going to take the opportunity of your mention of 'on the weekend' to explain something that I think is obvious, and probably really is, but maybe is not at the top of everyone's awareness. For good or bad the box-admins team work is 24/7. In practice it doesn't often add up to that much time but first of all you never know when disaster in its various forms will strike. Furthermore I think it is essential that every single day the server status is checked. Here is a quick summary of what I do every single day, yes 7 days a week every week (OK, every once in a while I miss a day): login to the server via ssh as root uptime Look at the days of uptime, is the number surprisingly small like 1 day or less? Look at the number of logged in users. If it's not 1 (yourself) then see who it is and have some confidence they aren't doing anything nefarious. Look at the load average, is it in the normal range? df Any significant changes in disk usage? Is it time to do some cleanup and make space? free Any significant changes in RAM usage? ps auwx Quickly glance through the process list looking for anything unusual. This task can lead to other inquiries when there are a lot of qmail processes running or anything else seems unusual. Only experience will help judging what is unusual. top (then hit the $c key to see the full command line which is essential in identifying squeak processes) This is another way to look at the process list in particular monitoring the memory and cpu levels and as with the rest of this checking if things are more or less normal. Further every day I visit source.squeak.org If the front page comes up I'm satisfied all is probably normal. And bugs.squeak.org I check the manage page where I can quickly see any recent signups. We have someone who seems to test their anti-captcha technology and signup with semi-random but consistent account names and user names but they never actually do anything with the accounts. I delete them whenever I see them. After that I check the bug list page with no filtering and no project set so I can always see all changes in case anyone is fooling around. And that's usually it. It probably doesn't take even 10 minutes most days. But doing it every day is essential in catching problems before they become more serious and simply to have the knowledge to judge that a change has occurred. This is nothing new to anyone I suspect, basic Linux admin 101, but I thought I should say it aloud to make sure. Ken > -------- Original Message -------- > Subject: Re: Welcome Steve and Casey > From: Steven Elkins <sgelkins(a)gmail.com> > Date: Sat, February 05, 2011 2:31 pm > To: Ken Causey <ken(a)kencausey.com> > Cc: box-admins(a)lists.squeakfoundation.org, casey.obrien.r(a)gmail.com > > > I've been moderating squeak-dev for some time. If there's something that I > can do (1) on the weekend and (2) not too time-consuming, I might be able to > help. > > Cheers, > Steve > > On Sat, Feb 5, 2011 at 2:30 PM, Ken Causey <ken(a)kencausey.com> wrote: > > > Welcome to the box-admins mailing list. Feel free to introduce > > yourselves (again perhaps) if you like; if you have any particular > > interest or thoughts related to managing the community services please > > contribute. > > > > Thanks, > > > > Ken > > > > > > > -- > How wonderful it is that nobody need wait a single moment > before starting to improve the world. -- Anne Frank > Paradise is exactly where you are > right now...only much, much better. -- Laurie Anderson

3 2

RE: [Box-Admins] Permissions
by Ken Causey 05 Feb '11

05 Feb '11

I said: > Well, that depends on what you feel needs to be backed up. Some might > feel the entire system should be backed up and that is currently 122GB. > I'm satisfied if /home/ /etc/ /usr/local/ /var/www/ /root/ and > /var/lib/mailman/ are backed up at minimum. Oops, 122GB includes the local backup. You can at least take off a third of that and perhaps half. Ken

1 0

RE: [Box-Admins] Permissions
by Ken Causey 05 Feb '11

05 Feb '11

> -------- Original Message -------- > Subject: Re: [Box-Admins] Permissions > From: Chris Muller <ma.chris.m(a)gmail.com> > Date: Sat, February 05, 2011 2:55 pm > To: Ken Causey <ken(a)kencausey.com> > Cc: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > > > Ok, so you're saying, we don't have a place to back-up to right now. > How much space are we using? Well, that depends on what you feel needs to be backed up. Some might feel the entire system should be backed up and that is currently 122GB. I'm satisfied if /home/ /etc/ /usr/local/ /var/www/ /root/ and /var/lib/mailman/ are backed up at minimum. > I'm wondering if whether each box-admin > team member could simply rsync to a dedicated directory on home > machines, perhaps on daily cron schedule? Well, it's a good time to admit I lied a bit. I made a backup locally last July, and I'm updating it right now. Perhaps others feel differently but there is no way I could do this daily and even weekly would be an annoyance. I'm backing up only the essential I mentioned above which in total came to 36G in July. Based on the local backup, if you backup daily the changes (in one day) come to about 3-5GB per day. > OR, we could use a pay service, but it might not be a bad idea to just > try to get ONE copy somewhere. We can't afford to lose > source.squeak.org. > > We won't procrastinate this anymore. I remember now that Ken and > Goran researched the backup a few years ago. Did you guys end up > using rsync? rsync/rsnapshot. We use this for the local backup and it is what Goran used and I'm currently using rsync directly. > > I'll try to poke around on the box this weekend. > > BTW, who (company or individual?) and where, physically, are the > boxe(s) located now? Do we only have one box? We have only one box and it is hosted by Hetzner.de. Given your lack of knowledge of that (it's possible I guess that you heard it mentioned but made no notice of it I guess) it sounds like the situation regarding the payment for the server has still not been sorted out and squeak eV is still having to handle it. I hope I'm wrong about that. (I see Bert has forwarded the email we wrote up when I left the Board, IIRC.) Ken > > - Chris > > On Sat, Feb 5, 2011 at 1:34 PM, Ken Causey <ken(a)kencausey.com> wrote: > > > > > >> -------- Original Message -------- > >> Subject: Re: [Box-Admins] Permissions > >> From: Chris Muller <asqueaker(a)gmail.com> > >> Date: Sat, February 05, 2011 11:25 am > >> To: Squeak Hosting Support <box-admins(a)lists.squeakfoundation.org> > >> > >> > >> Göran, can you elaborate on the backup? What are we using now and > >> what's broken about it. > >> > >> HD's fail, backups are very important. Are you saying we're not > >> backed up right now? > >> > >> thanks.. > > > > Yes, right now there is no offsite backup. We have a local backup > > system for much content in case we just screw up but that doesn't help > > in the case of hard drive failure. > > > > Goran provided an offsite backup for a long time but lost the facility > > when he changed jobs. I've asked for help with this in the past with > > little response. It has been a while and perhaps it is time to ask > > again. > > > > Ken > > > >> > >> 2011/2/5 Göran Krampe <goran(a)krampe.se>: > >> > On 02/04/2011 10:19 PM, Ken Causey wrote: > >> >> > >> >> We already have this in some sense. Most individual projects are setup > >> >> under their own home directories and to administer them it is generally > >> >> sufficient to have access to the individual account, not superuser > >> >> access. For example the webteam manages the website without superuser > >> >> access and release teams can update the ftp site generally without > >> >> superuser access. So for example if someone wanted to volunteer to work > >> >> on bugs.squeak.org it would probably suffice to give them access to the > >> >> mantis user account. > >> >> > >> >> That said the use of sudo has been brought up before and it is worth > >> >> looking into. My grand plans to setup new servers included every > >> >> intention of making liberal use of sudo. But I simply haven't gotten > >> >> around to it. I would welcome any insights anyone has on configuring > >> >> sudo for our purposes. > >> >> > >> >> Ken > >> > > >> > IMHO the levels Ken describe are enough. If you are trusted with Linux admin > >> > stuff then root it is IMHO. Let's not complicate things :) > >> > > >> > ...and we really need to fix a running offsite scheduled backup. ASAP. > >> > > >> > Sidenote: Btw, I now use Duplicity with Deja Dup on my laptop - works good > >> > even for large files onto a vfat external USB drive. > >> > > >> > regards, Göran > >> > > > > >

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Box-Admins February 2011