Backup recommendation
in [UK Linux]
|
Prev: Compaq NS5000...
Next: Backup to DVD
From: Gordon Henderson on 19 Dec 2008 10:51 In article <gigcjh$pfg$1(a)localhost.localdomain>, Martin Gregorie <martin(a)see.sig.for.address.invalid> wrote: >On Fri, 19 Dec 2008 13:45:43 +0000, Ian Rawlings wrote: > >> Can you not use rsync instead of your .tgz backup? >> >I could, but I wanted to use compressed backups. The TGZ files are 8.8 GB >vs a tad under 15 GB uncompressed and the USB disk is only 120 GB and I >wanted to automatically fit as many backups as possible onto the disk. >Using compressed archives and a big empty container gives the easiest way >of doing this. Slightly harder to restore though. The multi-day rsync method: Do the first one by hand: mkdir /backupdevice/00 rsync --stats -aHx --delete /path /backupdevice/00 Then daily: cd /backupdevice rm -rf 99 rename 98 to 99 rename 97 to 98 etc. (obviously in a loop) rename 00 to 01 cp -al 01 00 rsync --stats -aHx --delete /path /backupdevice/00 So while not compressed, each day will only store what gets changed, so you should very probably be able to store even more copies using this method than compressed backups. What you end up with is NN copies of the data, each from the day before. Files that don't change are hard linked to the "originals". Restoring is easy because the files are "just there" with no compression and no searching through tar indexes and so on. (And remember that rsync can send to a remote host, given the right ssh credentials, etc.) And if you want an archive, on the 1st of each month, cp -al 99 archive.month.year An example of space used: Filesystem Size Used Avail Use% Mounted on /dev/md5 35G 9.9G 24G 30% /var /dev/md6 38G 29G 7.7G 79% /archive The /archive here has 60-days worth of backups. So if you got a 50% compression on 10GB of data to 5GB, 60 * 5GB is 300GB, yet the rsync method is only using 29GB. If you want the script, have a look at http://unicorn.drogon.net/dailyBackup That's just the "get out of jail free" accidental deletion backup - the partition it backs up to is on the same box - it's part of a series of scripts that do things like mysql database dumps, take the archive to other servers both on-site and off-site... However if doing it to removable media, it can be changed a little. And yes, it's in csh. Works for me. Gordon
From: Ian Rawlings on 19 Dec 2008 10:56 On 2008-12-19, Martin Gregorie <martin(a)see.sig.for.address.invalid> wrote: > I could, but I wanted to use compressed backups. Yes I did wonder about that. Mind you, with your system if you do get an error then you can lose the entire backup rather than just one file, and I'm told (but have never tested it) that with uncorrectable bit error rates, and the amount of bits on a disc these days, you've got about a 50% chance of being able to read an entire disc without finding at least one error. > I met it with my compressed archiving system. As soon as the archive grew > to 4GB it started failing: tar thought it had hit run out of diskspace > and quit. The solution was simple: I merely reformatted the USB drive as > a single ext3 partition. Last time I hit that problem, the FS reported that the write happened fine, but it actually had looped around and was over-writing the start of the file ;-) I think this was accessing a disc across SAMBA though. > The idea in both schemes is to preserve enough stuff so that, if a disk > dies, I can just flop the latest dump back onto a new disk. I do exactly the same with my systems, a script that mounts /boot and an exclude list that includes /proc/, /tmp/ and a few others, although I do include /dev now as one of the restores I did came up with "can't access /dev/console" and died, requiring a bit of furtling! On my laptop I rsync the encrypted filesystem across unencrypted to a server and do an emerge sync in the script that I use to connect it to my home network, the emerge sync (gentoo) generally takes much longer than the backup. -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on 19 Dec 2008 11:05 On 2008-12-19, alexd <troffasky(a)hotmail.com> wrote: > Have you thought of using rsnapshot? It uses rsync, but with hard links > instead of creating new copies of files, so it could save you a lot of > space. That protects you against accidental deletion and some metadata corruption, but if the file contents get corrupted or a file in the backup is edited, then it happens to all the "copies" in the backups so that's no good. Each hard link points to the same data on the disc, so if the data on the disc changes, then provided that the inode remains the same all your backups will be pointing at the now changed data. E.g.; echo "this is textfile1.txt" > textfile1.txt ln -s textfile1.txt textfile2.txt cat textfile2.txt this is textfile1.txt vi textfile1.txt (change the textfile1 to a textfile3) cat textfile2.txt this is textfile3.txt So you change one of the files and all the hard links now point to the changed data, just what you *don't* want if you're making historic backups. A hard link is not a backup, it's just a pointer to the same data. -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on 19 Dec 2008 11:11 On 2008-12-19, Gordon Henderson <gordon+usenet(a)drogon.net> wrote: > cp -al 01 00 > rsync --stats -aHx --delete /path /backupdevice/00 > > So while not compressed, each day will only store what gets changed, Hard links (cp -al) are not copies, they're pointers to the same data, so a corruption of that data knackers every one of your backups of that file, so only of use to protect against accidental deletion. -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/
From: Ian Rawlings on 19 Dec 2008 11:24
On 2008-12-19, Ian Rawlings <news06(a)tarcus.org.uk> wrote: > echo "this is textfile1.txt" > textfile1.txt > ln -s textfile1.txt textfile2.txt Err, ignore the ln -s, it works the same even if you use a hard link (which I did do when I tested it, just not in the post!) -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/ |