rsync and modifcation times
in [Mac Systems]
|
Prev: Safari not working
Next: Begging on Usenet doesn't work
From: Fergus McMenemie on 26 May 2010 08:57 Odd problem syncing files, I was using a script from the internet to sync some files to and from a central location. I noted that the files were sync-ing twice. I wrote the following test script to check things out. It look as though rsync does not update timestamps properly This bug seems a really massive one to me in that every backup type operation based on rsync has messed up dates. =============start============ #! /bin/bash remote_dir="/Volumes/webdav/Common" file_to_mirror="test.txt" if [ ! -e ~/$file_to_mirror ] then touch ~/$file_to_mirror fi if [ ~/$file_to_mirror -nt $remote_dir/$file_to_mirror ] then # our file is newer echo "Copying to remote dir" rsync -aEXvu --rsync-path="/usr/local/bin/rsync" \ ~/$file_to_mirror $remote_dir/$file_to_mirror elif [ ~/$file_to_mirror -ot $remote_dir/$file_to_mirror ] then # mirror fileis newer echo "Updating from remote dir" rsync -aEXvu --rsync-path="/usr/local/bin/rsync" \ $remote_dir/$file_to_mirror ~/$file_to_mirror else echo "Both files same!" fi stat $remote_dir/$file_to_mirror ~/$file_to_mirror =============end============ I started off 1) with the test file the same in both places. I guess my fist issue is why is "stat" printing four times and what are the four times? I then 2) touched the local file to "change" it and reran the script; the remote copy was updated as expected. Good. Now at 3) things begin to go bad. Rerunning the sync again I should see that the local and remote copies are identical... seeing as that is what step two did! However the files are deemed to differ and the local copy is updated from the remote. At stage 4) things seem OK again. But why the double transfer? It appears that the version of rsync supplied with tiger, leopard and snow leopard are all 2.6.9 which contains a bug which affects the modification date on the copy if the -E flag is present. So I upgraded both ends to use 3.0.6 but see the same effect. Followed the instructions from http://www.bombich.com/mactips/rsync.html when upgrading rsync. In the following I wrapped the output from stat to stop my mailer from doing so! Local is a MacBookPro runing snow leapard, remote is MacMini running tiger. Using webdav to access the remote MacMini. 1)======================================= fergus: ./test-ot.bash Both files same! 788529167 56 -rwx------ 1 fergus fergus 0 0 "May 26 10:44:25 2010" "May 26 10:44:25 2010" "May 26 10:44:25 2010" "May 26 10:44:25 2010" 4096 0 0 /Volumes/webdav/Common/test.txt 234881026 6990277 -rwx------ 1 fergus fergus 0 0 "May 26 10:44:26 2010" "May 26 10:44:25 2010" "May 26 10:44:26 2010" "May 26 10:44:25 2010" 4096 0 0 /Users/fergus/test.txt 2)======================================= fergus: touch test.txt fergus: stat /Volumes/webdav/Common/test.txt test.txt 788529167 56 -rwx------ 1 fergus fergus 0 0 "May 26 10:44:25 2010" "May 26 10:44:25 2010" "May 26 10:44:25 2010" "May 26 10:44:25 2010" 4096 0 0 /Volumes/webdav/Common/test.txt 234881026 6990277 -rwx------ 1 fergus fergus 0 0 "May 26 10:44:48 2010" "May 26 10:44:48 2010" "May 26 10:44:48 2010" "May 26 10:44:25 2010" 4096 0 0 test.txt fergus: fergus: fergus: ./test-ot.bash Copying to remote dir sending incremental file list test.txt sent 89 bytes received 31 bytes 240.00 bytes/sec total size is 0 speedup is 0.00 788529167 57 -rwx------ 1 fergus fergus 0 0 "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt 234881026 6990277 -rwx------ 1 fergus fergus 0 0 "May 26 10:44:48 2010" "May 26 10:44:48 2010" "May 26 10:44:48 2010" "May 26 10:44:25 2010" 4096 0 0 /Users/fergus/test.txt 3)======================================= fergus: ./test-ot.bash Updating from remote dir sending incremental file list test.txt sent 89 bytes received 31 bytes 240.00 bytes/sec total size is 0 speedup is 0.00 788529167 57 -rwx------ 1 fergus fergus 0 0 "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt 234881026 6990281 -rwx------ 1 fergus fergus 0 0 "May 26 10:45:14 2010" "May 26 10:45:09 2010" "May 26 10:45:14 2010" "May 26 10:45:09 2010" 4096 0 0 /Users/fergus/test.txt 4)======================================= fergus: ./test-ot.bash Both files same! 788529167 57 -rwx------ 1 fergus fergus 0 0 "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt 234881026 6990281 -rwx------ 1 fergus fergus 0 0 "May 26 10:45:14 2010" "May 26 10:45:09 2010" "May 26 10:45:14 2010" "May 26 10:45:09 2010" 4096 0 0 /Users/fergus/test.txt fergus:
From: Bruce Esquibel on 26 May 2010 15:07 Fergus McMenemie <fergus(a)twig-me-uk.not.here> wrote: > Odd problem syncing files, I was using a script from the internet > to sync some files to and from a central location. I noted that the > files were sync-ing twice. I wrote the following test script to check > things out. It look as though rsync does not update timestamps > properly > This bug seems a really massive one to me in that every backup type > operation based on rsync has messed up dates. I dunno if it's the script but one thing with rsync and macs is, the volume you are writing the files to, need that "ignore ownership on this volume" box checked on the drive info. This of course applies to getting files from a remote host. If left at default, it doesn't really matter what options you send to rsync for preserving ownership/permissions/dates, the OSX will just ignore it. If you are sending files from your mac to an rsync server, one thing to keep in mind is, it's probably not running as root in the rsyncd.conf so again, the host will place it's own ownership and permissions on the files, likely screwing up the time stamps you want to preserve. Other than that, the 3.0.6 works like a charm. -bruce bje(a)ripco.com
From: Wes Groleau on 26 May 2010 17:24 On 05-26-2010 15:07, Bruce Esquibel wrote: > I dunno if it's the script but one thing with rsync and macs is, the > volume you are writing the files to, need that "ignore ownership on this > volume" box checked on the drive info. Not so. I've been using rsync for ages to keep all or part of two or more Macs identical. Both HFS+ and UFS volumes. I have never "ignored ownership." -- Wes Groleau Hillary Insults Virgen de Guadalupe? http://Ideas.Lang-Learn.us/russell?itemid=1531
From: Bob Harris on 26 May 2010 22:15 In article <1jj3lud.1kfkzxft0aauuN%fergus(a)twig-me-uk.not.here>, fergus(a)twig-me-uk.not.here (Fergus McMenemie) wrote: > Odd problem syncing files, I was using a script from the internet > to sync some files to and from a central location. I noted that the > files were sync-ing twice. I wrote the following test script to check > things out. It look as though rsync does not update timestamps > properly > > This bug seems a really massive one to me in that every backup type > operation based on rsync has messed up dates. > > > =============start============ > #! /bin/bash > > remote_dir="/Volumes/webdav/Common" > file_to_mirror="test.txt" > > if [ ! -e ~/$file_to_mirror ] > then > touch ~/$file_to_mirror > fi > > if [ ~/$file_to_mirror -nt $remote_dir/$file_to_mirror ] > then # our file is newer > echo "Copying to remote dir" > rsync -aEXvu --rsync-path="/usr/local/bin/rsync" \ > ~/$file_to_mirror $remote_dir/$file_to_mirror > elif [ ~/$file_to_mirror -ot $remote_dir/$file_to_mirror ] > then # mirror fileis newer > echo "Updating from remote dir" > rsync -aEXvu --rsync-path="/usr/local/bin/rsync" \ > $remote_dir/$file_to_mirror ~/$file_to_mirror > else > echo "Both files same!" > fi > > stat $remote_dir/$file_to_mirror ~/$file_to_mirror > =============end============ > > I started off 1) with the test file the same in both places. I guess > my fist issue is why is "stat" printing four times and what are the > four times? I then 2) touched the local file to "change" it and reran > the script; the remote copy was updated as expected. Good. > > > Now at 3) things begin to go bad. Rerunning the sync again I should > see that the local and remote copies are identical... seeing as that > is what step two did! However the files are deemed to differ and the > local copy is updated from the remote. > > At stage 4) things seem OK again. But why the double transfer? > > It appears that the version of rsync supplied with tiger, leopard and > snow leopard are all 2.6.9 which contains a bug which affects the > modification date on the copy if the -E flag is present. So I upgraded > both ends to use 3.0.6 but see the same effect. Followed the > instructions > from http://www.bombich.com/mactips/rsync.html when upgrading rsync. > > In the following I wrapped the output from stat to stop my mailer from > doing so! Local is a MacBookPro runing snow leapard, remote is MacMini > running tiger. Using webdav to access the remote MacMini. > > > 1)======================================= > fergus: ./test-ot.bash > Both files same! > 788529167 56 -rwx------ 1 fergus fergus 0 0 > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" 4096 0 0 /Volumes/webdav/Common/test.txt > 234881026 6990277 -rwx------ 1 fergus fergus 0 0 > "May 26 10:44:26 2010" > "May 26 10:44:25 2010" > "May 26 10:44:26 2010" > "May 26 10:44:25 2010" 4096 0 0 /Users/fergus/test.txt > > > > 2)======================================= > fergus: touch test.txt > fergus: stat /Volumes/webdav/Common/test.txt test.txt > 788529167 56 -rwx------ 1 fergus fergus 0 0 > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" > "May 26 10:44:25 2010" 4096 0 0 /Volumes/webdav/Common/test.txt > 234881026 6990277 -rwx------ 1 fergus fergus 0 0 > "May 26 10:44:48 2010" > "May 26 10:44:48 2010" > "May 26 10:44:48 2010" > "May 26 10:44:25 2010" 4096 0 0 test.txt > fergus: > fergus: > fergus: ./test-ot.bash > Copying to remote dir > sending incremental file list > test.txt > > sent 89 bytes received 31 bytes 240.00 bytes/sec > total size is 0 speedup is 0.00 > 788529167 57 -rwx------ 1 fergus fergus 0 0 > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt > 234881026 6990277 -rwx------ 1 fergus fergus 0 0 > "May 26 10:44:48 2010" > "May 26 10:44:48 2010" > "May 26 10:44:48 2010" > "May 26 10:44:25 2010" 4096 0 0 /Users/fergus/test.txt > > > > 3)======================================= > fergus: ./test-ot.bash > Updating from remote dir > sending incremental file list > test.txt > > sent 89 bytes received 31 bytes 240.00 bytes/sec > total size is 0 speedup is 0.00 > 788529167 57 -rwx------ 1 fergus fergus 0 0 > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt > 234881026 6990281 -rwx------ 1 fergus fergus 0 0 > "May 26 10:45:14 2010" > "May 26 10:45:09 2010" > "May 26 10:45:14 2010" > "May 26 10:45:09 2010" 4096 0 0 /Users/fergus/test.txt > > > > 4)======================================= > fergus: ./test-ot.bash > Both files same! > 788529167 57 -rwx------ 1 fergus fergus 0 0 > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" > "May 26 10:45:09 2010" 4096 0 0 /Volumes/webdav/Common/test.txt > 234881026 6990281 -rwx------ 1 fergus fergus 0 0 > "May 26 10:45:14 2010" > "May 26 10:45:09 2010" > "May 26 10:45:14 2010" > "May 26 10:45:09 2010" 4096 0 0 /Users/fergus/test.txt > fergus: Is the destination volume FAT32? FAT only maintains resolutions at 2 second intervals. If the destination is a FAT file system, then use the --modify-window=1 option so rsync will be more tolerant of the FAT short comings.
From: Bruce Esquibel on 27 May 2010 08:06
Wes Groleau <Groleau+news(a)freeshell.org> wrote: > Not so. I've been using rsync for ages to keep all or part of > two or more Macs identical. Both HFS+ and UFS volumes. I have > never "ignored ownership." Between two macs, probably. Going to and from something else, like Solaris 10, sparc or x86, it won't work right. The OP didn't say, or at least wasn't clear on this, the kind of remote host. Even if the owner was root, it'll all end up looking like this... -rw-r--r-- 1 _unknown _unknown 2 Feb 21 2009 Desktop DF -rw-r--r-- 1 _unknown _unknown 92022858 Feb 9 2009 databases-020209.tgz -rw-r--r-- 1 _unknown _unknown 96934920 Feb 9 2009 databases-020909.tgz -rw-r--r-- 1 _unknown _unknown 903741440 Feb 27 2009 databases-022709.tgz The owner/group permissions definetly are shot, but the time stamps aren't. Unless he clarifies the setup it's just second guessing. I never had to use anything more complex with rsync than -avzo and it always worked reliable once the rsyncd.conf was configured properly and that ownership thing turned off. -bruce bje(a)ripco.com |