Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Peter Olcott on 20 Apr 2010 14:06 "Joe Beanfish" <joe(a)nospam.duh> wrote in message news:hqkp2c$c98(a)news.thunderstone.com... > On 04/20/10 08:38, Peter Olcott wrote: >> "Tony Delroy"<tony_in_da_uk(a)yahoo.co.uk> wrote in >> message >> news:7ed1ecad-c641-4d32-bcef-6bd5b3bc5632(a)c20g2000prb.googlegroups.com... >> On Apr 9, 1:58 am, "Peter Olcott"<NoS...(a)OCR4Screen.com> >> wrote: >>> Is there a completely certain way that a write to a file >>> can be flushed to the disk that encompasses every >>> possible >>> memory buffer, including the hard drives onboard cache? >>> I >>> want to be able to yank the power cord at any moment and >>> not >>> get corrupted data other than the most recent single >>> transaction. >> >> --You'd have to check the hard disk programming >> documents, >> you may be >> --able to do direct I/O to ensure your data is written. >> Even if the >> --drive's onboard cache has not been flushed, it might >> have >> enough >> --capacitance to flush during a power failure, or use >> non-volatile >> --memory that can be flushed when power returns. Above >> that, the answer >> --is highly OS dependent, and you've specified absolutely >> nothing about >> --your hardware, OS, programming language etc.... >> -- >> --Cheers, >> --Tony >> >> It looks like that OS is not the big problem. The OS can >> always be bypassed, by working directly with the >> hardware. >> The big problem is that for example Western Digital SATA >> drives simply do not implement the "Flush Cache" ATA >> command. >> >> Seagate drives do implement this command. It was >> Seagate's >> idea to create this command in 2001. Although it may >> still >> be possible to simply shut off write caching for these >> drives, this will wear the drive out much more quickly, >> and >> drastically reduce performance. > > Have you considered solid state hard disks? Server > quality, not the > cheap desktop quality ones. > > IMHO, with a magnetic HD with a journalling filesystem and > a good UPS > with software to shutdown before battery runs out are all > you need. > Then you won't have to sacrifice speed trying to sync all > the way to > the hard media. I will be renting my system from my service provider, thus no choices are available for hardware. Both UPS and backup generators are provided by my service provider. SSD have a limited life that is generally not compatible with extremely high numbers of transactions. Some drives might not even be smart enough to flush their buffers even when UPS kicks in. I guess that you could force a buffer flush for every drive by simply writing a file larger than the buffer. If you make sure that this file is not fragmented, it might even be fast enough to do this after every transaction. Obviously the best way to do this would be to have a drive that correctly implements some sort of "Flush Cache" command such as the ATA command.
From: Joe Beanfish on 20 Apr 2010 13:42 On 04/20/10 08:38, Peter Olcott wrote: > "Tony Delroy"<tony_in_da_uk(a)yahoo.co.uk> wrote in message > news:7ed1ecad-c641-4d32-bcef-6bd5b3bc5632(a)c20g2000prb.googlegroups.com... > On Apr 9, 1:58 am, "Peter Olcott"<NoS...(a)OCR4Screen.com> > wrote: >> Is there a completely certain way that a write to a file >> can be flushed to the disk that encompasses every possible >> memory buffer, including the hard drives onboard cache? I >> want to be able to yank the power cord at any moment and >> not >> get corrupted data other than the most recent single >> transaction. > > --You'd have to check the hard disk programming documents, > you may be > --able to do direct I/O to ensure your data is written. > Even if the > --drive's onboard cache has not been flushed, it might have > enough > --capacitance to flush during a power failure, or use > non-volatile > --memory that can be flushed when power returns. Above > that, the answer > --is highly OS dependent, and you've specified absolutely > nothing about > --your hardware, OS, programming language etc.... > -- > --Cheers, > --Tony > > It looks like that OS is not the big problem. The OS can > always be bypassed, by working directly with the hardware. > The big problem is that for example Western Digital SATA > drives simply do not implement the "Flush Cache" ATA > command. > > Seagate drives do implement this command. It was Seagate's > idea to create this command in 2001. Although it may still > be possible to simply shut off write caching for these > drives, this will wear the drive out much more quickly, and > drastically reduce performance. Have you considered solid state hard disks? Server quality, not the cheap desktop quality ones. IMHO, with a magnetic HD with a journalling filesystem and a good UPS with software to shutdown before battery runs out are all you need. Then you won't have to sacrifice speed trying to sync all the way to the hard media.
From: Tony Delroy on 20 Apr 2010 03:20 On Apr 9, 1:58 am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > Is there a completely certain way that a write to a file > can be flushed to the disk that encompasses every possible > memory buffer, including the hard drives onboard cache? I > want to be able to yank the power cord at any moment and not > get corrupted data other than the most recent single > transaction. You'd have to check the hard disk programming documents, you may be able to do direct I/O to ensure your data is written. Even if the drive's onboard cache has not been flushed, it might have enough capacitance to flush during a power failure, or use non-volatile memory that can be flushed when power returns. Above that, the answer is highly OS dependent, and you've specified absolutely nothing about your hardware, OS, programming language etc.... Cheers, Tony
From: Casper H.S. Dik on 20 Apr 2010 04:24 Tony Delroy <tony_in_da_uk(a)yahoo.co.uk> writes: >On Apr 9, 1:58=A0am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: >> Is there a completely certain way that a write to a file >> can be flushed to the disk that encompasses every possible >> memory buffer, including the hard drives onboard cache? =A0I >> want to be able to yank the power cord at any moment and not >> get corrupted data other than the most recent single >> transaction. >You'd have to check the hard disk programming documents, you may be >able to do direct I/O to ensure your data is written. Even if the >drive's onboard cache has not been flushed, it might have enough >capacitance to flush during a power failure, or use non-volatile >memory that can be flushed when power returns. Above that, the answer >is highly OS dependent, and you've specified absolutely nothing about >your hardware, OS, programming language etc.... The OS should shield the programmer from the particulars of the hardware. So read the manuals and hope they give you a promise you can live with (and not lie to you) (I'm somewhat disappointed that fsync() in Linux doesn't offer anything if your write cache is enabled) Casper -- Expressed in this posting are my opinions. They are in no way related to opinions held by my employer, Sun Microsystems. Statements on Sun products included here are not gospel and may be fiction rather than truth.
From: Peter Olcott on 20 Apr 2010 08:38
"Tony Delroy" <tony_in_da_uk(a)yahoo.co.uk> wrote in message news:7ed1ecad-c641-4d32-bcef-6bd5b3bc5632(a)c20g2000prb.googlegroups.com... On Apr 9, 1:58 am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > Is there a completely certain way that a write to a file > can be flushed to the disk that encompasses every possible > memory buffer, including the hard drives onboard cache? I > want to be able to yank the power cord at any moment and > not > get corrupted data other than the most recent single > transaction. --You'd have to check the hard disk programming documents, you may be --able to do direct I/O to ensure your data is written. Even if the --drive's onboard cache has not been flushed, it might have enough --capacitance to flush during a power failure, or use non-volatile --memory that can be flushed when power returns. Above that, the answer --is highly OS dependent, and you've specified absolutely nothing about --your hardware, OS, programming language etc.... -- --Cheers, --Tony It looks like that OS is not the big problem. The OS can always be bypassed, by working directly with the hardware. The big problem is that for example Western Digital SATA drives simply do not implement the "Flush Cache" ATA command. Seagate drives do implement this command. It was Seagate's idea to create this command in 2001. Although it may still be possible to simply shut off write caching for these drives, this will wear the drive out much more quickly, and drastically reduce performance. |