[Box-Admins] [Board] Backing up our servers

[Levente Uzonyi]

Hi Tony, hi all,

@Tony:
If you send me an ssh key, preferrably ed25519, I'll create a user for you 
on all the servers, so that you can log in with your key.

@all:
More than a week ago, I started to create backups of all servers using 
rdiff-backup[1]. It took a longer for the initial backup to complete 
than I expected due to the nature of the network setup we have: tcp is 
tunneled over tcp, which is known to be slow[2] but it surprised me how 
slow it turned out to be in practice (~1.66MB/s).

The process copies the following directories: /etc, /home, /opt /root, 
/srv, /var/backups, /var/log, /var/mail, /var/spool, /var/www.
If there are files that need to be preserved in any other directory on 
any server you are aware of, let me know.


Levente

[1] https://rdiff-backup.net/
[2] http://sites.inka.de/~W1011/devel/tcp-tcp.html

On Fri, 18 Mar 2022, Marcel Taeumel wrote:

> +1 for giving Tony access to the servers
> Best,
> Marcel
>
>       Am 17.03.2022 09:22:17 schrieb Tony Garnock-Jones <tonyg at leastfixedpoint.com>:
>
>       Hi David, Chris, all,
>
>       Great, that README is a good initial survey of the machine. The backups
>       I run for the various cloud servers under my control are similar.
>
>       Generally, I use rsnapshot [1] from cron to make incremental backups
>       across ssh (it uses rsync internally).
>
>       This works great for unprivileged files, those readable by an account
>       you'd trust to log in to grab files to be backed up.
>
>       For the privileged ones, as you note in your README, a cronjob running
>       as root on the source machines can archive the bits and pieces for later
>       pickup by the backup account.
>
>       Levente, Tobias, I'd be happy to log into the boxes to do the initial
>       survey and set up some backups. I suppose the box-admins team should
>       decide if I'm to be trusted to do this! It does require root access at
>       least initially. If you all would prefer not to create a new privileged
>       unix account, etc, that's cool -- in that case I'd also be happy to
>       pair-sysadmin via tmux or screen or similar if that would be helpful.
>       And actually even in the case you would be happy letting me work with
>       root, I'd also welcome having a second set of screen/tmux eyes on what
>       I'm doing.
>
>       Cheers,
>       Tony
>
>       [1] https://rsnapshot.org/
> 
>
>       On 3/17/22 01:47, David T. Lewis wrote:
>       > Hi Chris,
>       >
>       > Funny you should mention it, I was doing the same thing last
>       > weekend to make sure we could move the squeaksource.com service
>       > if needed. I had intended to share this in the board meeting
>       > today but was not able to attend. In any case, I'm attaching
>       > my "disaster recovery plan" notes (crudely formatted but better
>       > than nothing).
>       >
>       > For me it was a PITA dealing with session timeouts while
>       > download 30GB to my tired little old laptop PC, but at least
>       > I was able to do it after a few hours, and would be able to
>       > restore it to some new server as long as someone else could
>       > handle the sys admin things for ssh and port forwarding.
>       >
>       > I think between the two of us we could handle moving the two
>       > squeaksource services, although I don't know how to handle
>       > the rest of the services (mailing lists, etc). For that we
>       > would need some expertise from the box-admins, especially
>       > Levente and Tobias.
>       >
>       > Dave
>       >
>       >
>       > On Wed, Mar 16, 2022 at 03:58:09PM -0500, Chris Muller wrote:
>       >> Hi Tony, Dave, all,
>       >>
>       >> FYI, I had this *exact thought* yesterday when I was working on our
>       >> server. We "expect" cloud services to be redundant and backed up, but
>       >> for something this important it's necessary to have a backup system
>       >> under our control. Our code is an asset that can't be bought with money,
>       >> whereas the bandwidth is. I'm downloading a backup copy of the ./ss
>       >> directory at this moment which I need to test a server upgrade to 5.3
>       >> and the latest code. This will at least temporarily reduce our risk
>       >> of "total loss" until we can get a permanent backup solution.
>       >>
>       >> - Chris
>       >>
>       >> On Wed, Mar 16, 2022 at 12:52 PM David T. Lewis wrote:
>       >>>
>       >>> Let's make this an agenda item for our meeting (in a few minutes). I
>       >>> can share some work I did over the weekend to prepare a "poor person's"
>       >>> recovery plan for squeaksource.com. It's painful but it works.
>       >>>
>       >>> As for Rackspace backups, Levente and Tobias are most likely to know
>       >>> how and if this can be done. I don't know the answer, but I have a
>       >>> suspicion that the utility for "help me get my stuff off of Rackspace"
>       >>> is not something that Rackspace, Inc. would be in a big hurry to support.
>       >>>
>       >>> Dave
>       >>>
>       >>>
>       >>> On Wed, Mar 16, 2022 at 06:36:32PM +0100, Tony Garnock-Jones wrote:
>       >>>> Hello all,
>       >>>>
>       >>>> I'm writing on behalf of the Squeak Oversight Board - could someone get
>       >>>> in touch with me, please, to help me arrange backups of the important
>       >>>> bits of our servers with some urgency? We have become concerned with
>       >>>> disaster recovery and would like to quickly get something in place to at
>       >>>> least have *something* if the servers were to vanish in a puff of logic...
>       >>>>
>       >>>> Regards,
>       >>>> Tony
> 
> 
>