Backup recommendation
in [UK Linux]
|
Prev: Compaq NS5000...
Next: Backup to DVD
From: Ian on 19 Dec 2008 07:17 Dear All, I've just acquired a little HP Pocket Media Drive. I'd like to use it to backup /home, and I could do with some software recommendations. I'd rather not have the backups rolled up into a zip or tar file, so that I can easily pluck out files on another PC. I'd also like something which has the sense not to copy thinks which are already there. Would a simple cp -r --preserve=all --update /home /media/HP do the trick sensibly? Or rsync -Rav /home /media/HP? Advice much appreciated. Ian
From: Ian Rawlings on 19 Dec 2008 08:08 On 2008-12-19, Ian <ian.groups(a)btinternet.com> wrote: > Dear All, > > I've just acquired a little HP Pocket Media Drive. I'd like to use it > to backup /home, and I could do with some software recommendations. > I'd rather not have the backups rolled up into a zip or tar file, so > that I can easily pluck out files on another PC. I'd also like > something which has the sense not to copy thinks which are already > there. Well you can use rsync -av --delete /home/ /<media path/home-backup/ to do it, but if the destination is a FAT partition or similar non-unix format, you can lose permissions, owner details and some files with certain characters in their filenames, e.g. files with ":" in their filenames can't be written to FAT partitions and those beginning with "." can also present problems. -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/
From: Martin Gregorie on 19 Dec 2008 08:23 On Fri, 19 Dec 2008 04:17:54 -0800, Ian wrote: > Dear All, > > I've just acquired a little HP Pocket Media Drive. I'd like to use it to > backup /home, and I could do with some software recommendations. I'd > rather not have the backups rolled up into a zip or tar file, so that I > can easily pluck out files on another PC. I'd also like something which > has the sense not to copy thinks which are already there. > > Would a simple cp -r --preserve=all --update /home /media/HP do the > trick sensibly? Or rsync -Rav /home /media/HP? > I use both: - I have a 120 GB 3.5" drive that lives on top of the PC and gets a daily .TGZ compressed backup of the whole system from a daily cron job. This takes 75 minutes and requires a fairly elaborate script that automatically deletes enough of the oldest backups to leave room for the next, does the backup and mails me a run report. These backups are there to provide protection against finger trouble or disk failure. If there's a fire or a major power spike these backups will be toast along with the PC. - I also have a Formac 80GB 2.5" drive. Its probably similar to your HP. I make a weekly manual backup to this and keep it offline and fairly well protected. I use rsync and again backup the whole system. The initial backup took about 1.5 hours, but subsequent weekly backups take between 8 and 12 minutes. This is my disaster recovery backup. Bottom line: use rsync - its reliable, restartable and fast. Since backups are a relatively frequent, routine task, its worth while writing a script to manage the backup. It needs to: - mount the USB disk (use a volume label) - run rsync - run df so you won't be surprised by the disk getting fiull - unmount the USB disk - fsck the USB disk The script needs to handle failure and recovery in a graceful manner and report what its doing and/or why it failed. Making the backup is a critical task so you want to be in no doubt whether the backup is good or bad. -- martin@ | Martin Gregorie gregorie. | Essex, UK org |
From: Ian Rawlings on 19 Dec 2008 08:45 On 2008-12-19, Martin Gregorie <martin(a)see.sig.for.address.invalid> wrote: > - I have a 120 GB 3.5" drive that lives on top of the PC and gets > a daily .TGZ compressed backup of the whole system from a daily > cron job. This takes 75 minutes and requires a fairly elaborate > script that automatically deletes enough of the oldest backups to > leave room for the next, does the backup and mails me a run report. > These backups are there to provide protection against finger trouble > or disk failure. If there's a fire or a major power spike these > backups will be toast along with the PC. On one of my machines I do a backup of an important directory every day using a line of the following form; rsync --delete -ac /source/ /backup/`date "+\%u"`/ That puts each backup into backup dir inside a directory with a single-digit name, with that digit mirroring the day number. After the initial week, it has seven daily backups, then it loops round to copy across what's changed since the backup of the previous week. So I always have a week's worth of immediately-online backups for that directory. Nice and simple. It then ends up on tape. Can you not use rsync instead of your .tgz backup? It seems like a lot of work but I don't have any detailed knowledge of your restrictions. Rsync has some useful exclude features that can read a file of excluded directories and files with wildcards that you can update via a script for example. It would then only copy across data that has changed which usually takes a lot less time. Note that "No, it's complicated" is a perfectly acceptable reply ;-) -- Blast off and strike the evil Bydo empire! http://youtube.com/user/tarcus69 http://www.flickr.com/photos/tarcus/sets/
From: Martin Gregorie on 19 Dec 2008 09:54
On Fri, 19 Dec 2008 13:45:43 +0000, Ian Rawlings wrote: > Can you not use rsync instead of your .tgz backup? > I could, but I wanted to use compressed backups. The TGZ files are 8.8 GB vs a tad under 15 GB uncompressed and the USB disk is only 120 GB and I wanted to automatically fit as many backups as possible onto the disk. Using compressed archives and a big empty container gives the easiest way of doing this. When I first turned this system on at the start of 2006 backups were only 3.4 GB and I got about 33 of them onto the disk. Since then backups grew as I've migrated archived data off an old 'doze box and the number of backups kept has automatically adjusted to fit. For the OP: =========== There's one HUGE snag with using the default FAT filing system on the USB drive, much more terminal than even losing permissions if you rsync to it. There is an absolute maximum file size of 4GB for any FAT filing system regardless of the size of the partition or of the cluster size. I met it with my compressed archiving system. As soon as the archive grew to 4GB it started failing: tar thought it had hit run out of diskspace and quit. The solution was simple: I merely reformatted the USB drive as a single ext3 partition. So, before you start using your HP drive, reformat it as ext3 (use ext2 if you really don't want a journalling fs on USB). This solves both the file permissions problem and the max file size problem. Similarly, the first thing I did with my 80 GB Formac was to reformat it as a single ext3 partition. This time file permissions was the issue since I'd always to use rsync with this drive. BTW, file permissions were never an issue with a FAT partition and the compressed archives, since the file permissions were preserved inside the archive by tar. > of work but I don't have any detailed knowledge of your restrictions. > Rsync has some useful exclude features that can read a file of excluded > directories and files with wildcards that you can update via a script > for example. > I use some restrictions on both my tar and rsync schemes because there is stuff that nobody needs to back up, e.g the /proc and /tmp structures and ther's a lot of stuff on /var, such as logs, that just wastes time if its backed up. The idea in both schemes is to preserve enough stuff so that, if a disk dies, I can just flop the latest dump back onto a new disk. The compressed backup kicks off at 04:00 as part of the set of daily cron jobs, so I don't care how long it takes to run as long as its finished by the time I've got up and had my first coffee. Its average of 75 mins is quite acceptable. OTOH on Friday nights I rsync both my main server and this laptop onto the little Formac before going to the pub, so speed does matter. They take 8-12 minutes each depending on how big the weekly yum upgrade has been which, again, is very acceptable. -- martin@ | Martin Gregorie gregorie. | Essex, UK org | |