Backup solutions (was Re: [Box-Admins] Permissions)

[Casey Ransberger]

I hope I'm not adding too much noise to the conversation, but what do folks think about using something like S3 for backups? We'd have to find funding for it somewhere, but it might be cheaper than buying terabyte drives, etc, and the thing is pretty darned reliable for backup. 

Also makes sure that the bottleneck for data transfer isn't a home connection. 

Just a thought. 

On Feb 6, 2011, at 4:26 PM, Göran Krampe <goran at krampe.se> wrote:

> Hi all!
> 
> Been investigating this part and it seems to me that the three most interesting and mature tools for this is rdiff-backup, rsnapshot and duplicity.
> 
> 
> Rsnapshot
> =========
> Rsnapshot is what we have used so far. It is a bit tricky to configure IMHO and I must say slightly confusing with the "backup levels". It needs a working rsnapshot on the destination, and since it uses hard links to create the illusion of "full snapshots" it will consume much more space if we have common changes to large files, like say Squeak images - or hey, even worse, VirtualBox/VMWare images :) It also has issues with "file selection quirks" based in oddities in rsync - like sensitivity to trailing slashes etc.
> 
> Rdiff-backup
> ============
> Also relies on being on destination like rsnapshot. Saves metadata separately from the mirror, does not use hardlinks but instead maintain the mirror and store the increments from older snapshots on the side. Some report it is rather slow. Does not have "file selection quirks" but does also not have "backup levels". Seems simpler to use though. Cosumes less space on destination since it does not rely on hard links. Can be used with archs (http://code.google.com/p/archfs/) to create a FUSE illusion of full snapshots like rsnapshot has.
> 
> Duplicity
> =========
> Does NOT rely on being installed on destination, thus can use "dumb servers" through scp/ftp etc. Does NOT maintain a mirror on destination, but instead keeps file data in compressed tar files typically taking less space than rsnapshot/rdiff-backup. Can use encryption, which the above tools do not. Very simple to use and understand, although restores are of course not just an "scp yadda". Speed should IMHO be good compared to rsnap/rdiff since it keeps signatures etc on source and should not need to do ANY round tripping - not verified though, we would need to test.
> 
> 
> 
> I picked Duplicity because I wanted to use an external USB drive with vfat on it as destination, and Duplicity splits very large files so can easily backup my VirtualBox 12Gb files onto it, which of course rdiff-backup failed.
> 
> 
> Conclusion
> ==========
> If we want people to give us a bit of their harddrives for nightly run backups it seems to me that Duplicity might fit the bill best:
> 
> - Smaller size on destination (compressed)
> - Does not need duplicity on destination, just a dumb server
> - Good speed (we would need to test and compare)
> - We could use encryption to make it less "scary" to replicate stuff like "/etc" to boxes not under our direct control.
> 
> It does not handle hard links though, but perhaps not a big issue for us?
> 
> regards, Göran