copy a file while it is actively being written
in [General Linux]
|
Prev: nice bandwidth usage
Next: unexplained bianry "diff"
From: jinzishuai on 1 Feb 2010 18:08 Hi there, I am wondering if serious problem could happen if I copy a file that is actively being written to. I guess some file corruption might happen, right? More specifically, I am dealing with the snapshot of a qcow2 file. # qemu-img snapshot -c snap base.qcow2 # cp base.qcow2 snap.qcow2 # qemu-img snapshot -a snap snap.qcow2 Do you think I will get an image snap.qcow2 that is completely consistent with the base.qcow2 image at the time when the snapshot is taken? For my case, it is OK if new information written to the disk after snapshot taken is lost or corrupted since all I want to copy is the backed up content. But would this copy during write process actually mess up the whole file and make it even un-readable? Thanks a lot. Shi
From: Wanna-Be Sys Admin on 1 Feb 2010 18:10 jinzishuai(a)gmail.com wrote: > Hi there, > > I am wondering if serious problem could happen if I copy a file that > is actively being written to. > I guess some file corruption might happen, right? > > More specifically, I am dealing with the snapshot of a qcow2 file. > # qemu-img snapshot -c snap base.qcow2 > # cp base.qcow2 snap.qcow2 > # qemu-img snapshot -a snap snap.qcow2 > Do you think I will get an image snap.qcow2 that is completely > consistent with the base.qcow2 image at the time when the snapshot is > taken? > > For my case, it is OK if new information written to the disk after > snapshot taken is lost or corrupted since all I want to copy is the > backed up content. But would this copy during write process actually > mess up the whole file and make it even un-readable? > > Thanks a lot. > Shi Depending on how you copy the file, you can test and be warned if the file changed while it was being copied. You should use file locking (exclusive and shared, etc.) for opening and writing to, and reading files that could change (or copying files). I've never used the tool you've mentioned though, so I don't know what checking, if any, that it might do. -- Not really a wanna-be, but I don't know everything.
From: unruh on 1 Feb 2010 18:49 On 2010-02-01, jinzishuai(a)gmail.com <jinzishuai(a)gmail.com> wrote: > Hi there, > > I am wondering if serious problem could happen if I copy a file that > is actively being written to. > I guess some file corruption might happen, right? The copied file will not have everything in it. > > More specifically, I am dealing with the snapshot of a qcow2 file. > # qemu-img snapshot -c snap base.qcow2 > # cp base.qcow2 snap.qcow2 > # qemu-img snapshot -a snap snap.qcow2 > Do you think I will get an image snap.qcow2 that is completely > consistent with the base.qcow2 image at the time when the snapshot is > taken? Of course not. If the file does not contain all the data, you cannot copy all the data. Otherwise you could copy it yesterday, and not even have to take the photo, and the copy will have the photo in it. > > For my case, it is OK if new information written to the disk after > snapshot taken is lost or corrupted since all I want to copy is the > backed up content. But would this copy during write process actually > mess up the whole file and make it even un-readable? No. It would be readable-- maybe not consistant with the format that the picture file expects, but the bits would be there. > > Thanks a lot. > Shi
From: Douglas Mayne on 2 Feb 2010 10:43 On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai(a)gmail.com wrote: > Hi there, > > I am wondering if serious problem could happen if I copy a file that is > actively being written to. > I guess some file corruption might happen, right? > My _guess_ is that any "corruption" would show up as a failed backup attempt. The original file is hopefully locked by its owner for exclusive write access; thus, any external read attempt will not interfere with it. However, the backup attempt will be "blurry" at best. > > More specifically, I am dealing with the snapshot of a qcow2 file. # > qemu-img snapshot -c snap base.qcow2 # cp base.qcow2 snap.qcow2 > # qemu-img snapshot -a snap snap.qcow2 Do you think I will get an image > snap.qcow2 that is completely consistent with the base.qcow2 image at > the time when the snapshot is taken? > My _guess_ is that it is not going to be consistent. Additional measures to freeze the processes would be necessary to obtain a consistent, instantaneous snapshot. In GNU/Linux, instantaneous moving snapshots are handled via the device mapper facility. The object you are attempting to backup is probably already be being managed as device mapper object. It may be possible to make a snapshot of a snapshot, but I think some manual tweaking of the underlying objects would be required to do it. One other caveat: AFAIK, device mapper suffers a substantial performance hit whenever snapshots are active. YMMV. In other words, test that your virtual machines' performance remains acceptable when the file system is modified in this way. > > For my case, it is OK if new information written to the disk after > snapshot taken is lost or corrupted since all I want to copy is the > backed up content. But would this copy during write process actually > mess up the whole file and make it even un-readable? > > Thanks a lot. > Shi > Note: comments inline. IMO, you're going to have to run some tests to find out. Please, report back with any information that you find along the way. BTW, AFAIK, the OpenSolaris + zfs appears to offer the best possibilities for creating advanced, automatic snapshots. However, its future looks a bit muddy with the Oracle takeover of Sun. GNU/Linux is preparing a competing filesystem, btrfs, which will offer some of the same features. But AFAIK, it is not quite for production systems yet. -- Douglas Mayne
From: unruh on 2 Feb 2010 17:30
On 2010-02-02, Douglas Mayne <invalid(a)invalid.com> wrote: > On Mon, 01 Feb 2010 15:08:04 -0800, jinzishuai(a)gmail.com wrote: > >> Hi there, >> >> I am wondering if serious problem could happen if I copy a file that is >> actively being written to. >> I guess some file corruption might happen, right? >> > My _guess_ is that any "corruption" would show up as a failed backup > attempt. The original file is hopefully locked by its owner for exclusive > write access; thus, any external read attempt will not interfere with it. > However, the backup attempt will be "blurry" at best. File locking is something that depends totally on the program, and is a voluntary thing-- ie the reading program must check to see if the file is locked. ARAIK, there is no mechanism in Linux to lock a file so that nothing else can read it. >> >> More specifically, I am dealing with the snapshot of a qcow2 file. # >> qemu-img snapshot -c snap base.qcow2 # cp base.qcow2 snap.qcow2 >> # qemu-img snapshot -a snap snap.qcow2 Do you think I will get an image >> snap.qcow2 that is completely consistent with the base.qcow2 image at >> the time when the snapshot is taken? >> > My _guess_ is that it is not going to be consistent. Additional measures > to freeze the processes would be necessary to obtain a consistent, > instantaneous snapshot. In GNU/Linux, instantaneous moving snapshots are > handled via the device mapper facility. The object you are attempting to > backup is probably already be being managed as device mapper object. It > may be possible to make a snapshot of a snapshot, but I think some manual > tweaking of the underlying objects would be required to do it. > > One other caveat: AFAIK, device mapper suffers a substantial performance > hit whenever snapshots are active. YMMV. In other words, test that your > virtual machines' performance remains acceptable when the file system is > modified in this way. > As far as I know, any read file is a consistant file. Ie, it has bytes from the first location to the last location in the current file, and thus the file, as a file, is not inconsistant. If the contents of the file have some format, they it may well be that that the contents of the file are inconsistant with that format. For example, a pgm file in its header says that it has m rows and n columns and thus (for a mono 8 bit image) it has mxn bytes counting from after the end of the header. The file that copy may not have that many bytes in it. It will be inconsistant with the format of a pgm file. If you have a database or a compressed file, or... it also may be inconsistant with the format that the program which reads that file expects. However, it will still be a valid Linux file. >> >> For my case, it is OK if new information written to the disk after >> snapshot taken is lost or corrupted since all I want to copy is the >> backed up content. But would this copy during write process actually >> mess up the whole file and make it even un-readable? No. It would be readable, just incomplete. If your program expects some format for the contents, it might claim the file to be unuseable. > |