Hi Tony, hi all,
@Tony: If you send me an ssh key, preferrably ed25519, I'll create a user for you on all the servers, so that you can log in with your key.
@all: More than a week ago, I started to create backups of all servers using rdiff-backup[1]. It took a longer for the initial backup to complete than I expected due to the nature of the network setup we have: tcp is tunneled over tcp, which is known to be slow[2] but it surprised me how slow it turned out to be in practice (~1.66MB/s).
The process copies the following directories: /etc, /home, /opt /root, /srv, /var/backups, /var/log, /var/mail, /var/spool, /var/www. If there are files that need to be preserved in any other directory on any server you are aware of, let me know.
Levente
[1] https://rdiff-backup.net/ [2] http://sites.inka.de/~W1011/devel/tcp-tcp.html
On Fri, 18 Mar 2022, Marcel Taeumel wrote:
+1 for giving Tony access to the servers Best, Marcel
Am 17.03.2022 09:22:17 schrieb Tony Garnock-Jones <tonyg@leastfixedpoint.com>: Hi David, Chris, all, Great, that README is a good initial survey of the machine. The backups I run for the various cloud servers under my control are similar. Generally, I use rsnapshot [1] from cron to make incremental backups across ssh (it uses rsync internally). This works great for unprivileged files, those readable by an account you'd trust to log in to grab files to be backed up. For the privileged ones, as you note in your README, a cronjob running as root on the source machines can archive the bits and pieces for later pickup by the backup account. Levente, Tobias, I'd be happy to log into the boxes to do the initial survey and set up some backups. I suppose the box-admins team should decide if I'm to be trusted to do this! It does require root access at least initially. If you all would prefer not to create a new privileged unix account, etc, that's cool -- in that case I'd also be happy to pair-sysadmin via tmux or screen or similar if that would be helpful. And actually even in the case you would be happy letting me work with root, I'd also welcome having a second set of screen/tmux eyes on what I'm doing. Cheers, Tony [1] https://rsnapshot.org/ On 3/17/22 01:47, David T. Lewis wrote: > Hi Chris, > > Funny you should mention it, I was doing the same thing last > weekend to make sure we could move the squeaksource.com service > if needed. I had intended to share this in the board meeting > today but was not able to attend. In any case, I'm attaching > my "disaster recovery plan" notes (crudely formatted but better > than nothing). > > For me it was a PITA dealing with session timeouts while > download 30GB to my tired little old laptop PC, but at least > I was able to do it after a few hours, and would be able to > restore it to some new server as long as someone else could > handle the sys admin things for ssh and port forwarding. > > I think between the two of us we could handle moving the two > squeaksource services, although I don't know how to handle > the rest of the services (mailing lists, etc). For that we > would need some expertise from the box-admins, especially > Levente and Tobias. > > Dave > > > On Wed, Mar 16, 2022 at 03:58:09PM -0500, Chris Muller wrote: >> Hi Tony, Dave, all, >> >> FYI, I had this *exact thought* yesterday when I was working on our >> server. We "expect" cloud services to be redundant and backed up, but >> for something this important it's necessary to have a backup system >> under our control. Our code is an asset that can't be bought with money, >> whereas the bandwidth is. I'm downloading a backup copy of the ./ss >> directory at this moment which I need to test a server upgrade to 5.3 >> and the latest code. This will at least temporarily reduce our risk >> of "total loss" until we can get a permanent backup solution. >> >> - Chris >> >> On Wed, Mar 16, 2022 at 12:52 PM David T. Lewis wrote: >>> >>> Let's make this an agenda item for our meeting (in a few minutes). I >>> can share some work I did over the weekend to prepare a "poor person's" >>> recovery plan for squeaksource.com. It's painful but it works. >>> >>> As for Rackspace backups, Levente and Tobias are most likely to know >>> how and if this can be done. I don't know the answer, but I have a >>> suspicion that the utility for "help me get my stuff off of Rackspace" >>> is not something that Rackspace, Inc. would be in a big hurry to support. >>> >>> Dave >>> >>> >>> On Wed, Mar 16, 2022 at 06:36:32PM +0100, Tony Garnock-Jones wrote: >>>> Hello all, >>>> >>>> I'm writing on behalf of the Squeak Oversight Board - could someone get >>>> in touch with me, please, to help me arrange backups of the important >>>> bits of our servers with some urgency? We have become concerned with >>>> disaster recovery and would like to quickly get something in place to at >>>> least have *something* if the servers were to vanish in a puff of logic... >>>> >>>> Regards, >>>> Tony
box-admins@lists.squeakfoundation.org