Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Ian Collins on 8 Apr 2010 19:29 On 04/ 9/10 11:13 AM, Chris Friesen wrote: > On 04/08/2010 10:58 AM, Peter Olcott wrote: >> Is there a completely certain way that a write to a file >> can be flushed to the disk that encompasses every possible >> memory buffer, including the hard drives onboard cache? I >> want to be able to yank the power cord at any moment and not >> get corrupted data other than the most recent single >> transaction. > > Do you know what OS, OS version, filesystem, disk controller, and hard > disk you're using? > > If not, then the generic answer is that you need to disable the hard > drive cache. > > fsync() is supposed to work even with the cache enabled, but depending > on the OS and version it may not guarantee that the data has hit the > platter. Or the drive (or array controller) firmware. There are plenty of points between application code and the platter that can lie about honouring a sync request. -- Ian Collins
From: Ian Collins on 8 Apr 2010 19:31 On 04/ 9/10 11:10 AM, Peter Olcott wrote: > "Ian Collins"<ian-news(a)hotmail.com> wrote in message > news:82724vFp8jU6(a)mid.individual.net... >> On 04/ 9/10 04:58 AM, Peter Olcott wrote: >>> Is there a completely certain way that a write to a file >>> can be flushed to the disk that encompasses every >>> possible >>> memory buffer, including the hard drives onboard cache? >>> I >>> want to be able to yank the power cord at any moment and >>> not >>> get corrupted data other than the most recent single >>> transaction. >> >> Does the O_SYNC flag to open meet your requirements? >> >> I've removed c.p.t from the cross-post, it isn't >> appropriate. > > I am not sure. > So far the solution involves using open/read/write&& > fsync()&& turning the drive's write buffering off. > There is an alternative use of setvbuf() and/or fflush() if > I would use buffered file access. You haven't really explained what you are attempting to do. It looks like you are trying to second guess the filesystem's data integrity guarantees. -- Ian Collins
From: Peter Olcott on 8 Apr 2010 19:43 "Ian Collins" <ian-news(a)hotmail.com> wrote in message news:827764Fp8jU12(a)mid.individual.net... > On 04/ 9/10 11:13 AM, Chris Friesen wrote: >> On 04/08/2010 10:58 AM, Peter Olcott wrote: >>> Is there a completely certain way that a write to a file >>> can be flushed to the disk that encompasses every >>> possible >>> memory buffer, including the hard drives onboard cache? >>> I >>> want to be able to yank the power cord at any moment and >>> not >>> get corrupted data other than the most recent single >>> transaction. >> >> Do you know what OS, OS version, filesystem, disk >> controller, and hard >> disk you're using? >> >> If not, then the generic answer is that you need to >> disable the hard >> drive cache. >> >> fsync() is supposed to work even with the cache enabled, >> but depending >> on the OS and version it may not guarantee that the data >> has hit the >> platter. > > Or the drive (or array controller) firmware. There are > plenty of points between application code and the platter > that can lie about honouring a sync request. > > -- > Ian Collins This is exactly why I made this post. I want to categorically handle all of these. This is what I have so far: Turn drive write caching off then open with O_SYNC, then write() followed by fsync()
From: Peter Olcott on 8 Apr 2010 19:48 "Ian Collins" <ian-news(a)hotmail.com> wrote in message news:8277a6Fp8jU13(a)mid.individual.net... > On 04/ 9/10 11:10 AM, Peter Olcott wrote: >> "Ian Collins"<ian-news(a)hotmail.com> wrote in message >> news:82724vFp8jU6(a)mid.individual.net... >>> On 04/ 9/10 04:58 AM, Peter Olcott wrote: >>>> Is there a completely certain way that a write to a >>>> file >>>> can be flushed to the disk that encompasses every >>>> possible >>>> memory buffer, including the hard drives onboard cache? >>>> I >>>> want to be able to yank the power cord at any moment >>>> and >>>> not >>>> get corrupted data other than the most recent single >>>> transaction. >>> >>> Does the O_SYNC flag to open meet your requirements? >>> >>> I've removed c.p.t from the cross-post, it isn't >>> appropriate. >> >> I am not sure. >> So far the solution involves using open/read/write&& >> fsync()&& turning the drive's write buffering off. >> There is an alternative use of setvbuf() and/or fflush() >> if >> I would use buffered file access. > > You haven't really explained what you are attempting to > do. It looks like you are trying to second guess the > filesystem's data integrity guarantees. > > -- > Ian Collins I am trying to have completely reliable writes to a transaction log. This transaction log includes financial transactions. Even if someone pulls the plug in the middle of a transaction I want to only lose this single transaction and not have and other missing or corrupted data. One aspect of the solution to this problem is to make sure that all disk write are immediately flushed to the actual platters. It is this aspect of this problem that I am attempting to solve in this thread.
From: Moi on 8 Apr 2010 19:50
On Thu, 08 Apr 2010 18:43:40 -0500, Peter Olcott wrote: > "Ian Collins" <ian-news(a)hotmail.com> wrote in message > news:827764Fp8jU12(a)mid.individual.net... >> On 04/ 9/10 11:13 AM, Chris Friesen wrote: >>> On 04/08/2010 10:58 AM, Peter Olcott wrote: >>>> Is there a completely certain way that a write to a file can be >>>> flushed to the disk that encompasses every possible >>>> memory buffer, including the hard drives onboard cache? I >>>> want to be able to yank the power cord at any moment and not >>>> get corrupted data other than the most recent single transaction. >>> >>> Do you know what OS, OS version, filesystem, disk controller, and hard >>> disk you're using? >>> >>> If not, then the generic answer is that you need to disable the hard >>> drive cache. >>> >>> fsync() is supposed to work even with the cache enabled, but depending >>> on the OS and version it may not guarantee that the data has hit the >>> platter. >> >> Or the drive (or array controller) firmware. There are plenty of >> points between application code and the platter that can lie about >> honouring a sync request. >> >> -- >> Ian Collins > > This is exactly why I made this post. I want to categorically handle all > of these. This is what I have so far: > > Turn drive write caching off then > open with O_SYNC, then > write() followed by fsync() It is equivalent. If you open with O_SYNC or O_DATASYNC, the system acts *as if* every write were followed by a sync(). But that is only at the OS level. The disk driver will commit it to the disk, and the disk firmware may decide to let it wait a few rotations before the block hits the platter. AvK |