Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Scott Lurndal on 20 Apr 2010 19:51 "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: > >"Casper H.S. Dik" <Casper.Dik(a)Sun.COM> wrote in message >news:4bcd64ac$0$22903$e4fe514c(a)news.xs4all.nl... >> Tony Delroy <tony_in_da_uk(a)yahoo.co.uk> writes: >> >>>On Apr 9, 1:58=A0am, "Peter Olcott" >>><NoS...(a)OCR4Screen.com> wrote: >>>> Is there a completely certain way that a write to a file >>>> can be flushed to the disk that encompasses every >>>> possible >>>> memory buffer, including the hard drives onboard cache? >>>> =A0I >>>> want to be able to yank the power cord at any moment and >>>> not >>>> get corrupted data other than the most recent single >>>> transaction. >> >>>You'd have to check the hard disk programming documents, >>>you may be >>>able to do direct I/O to ensure your data is written. >>>Even if the >>>drive's onboard cache has not been flushed, it might have >>>enough >>>capacitance to flush during a power failure, or use >>>non-volatile >>>memory that can be flushed when power returns. Above >>>that, the answer >>>is highly OS dependent, and you've specified absolutely >>>nothing about >>>your hardware, OS, programming language etc.... >> >> The OS should shield the programmer from the particulars >> of the >> hardware. So read the manuals and hope they give you a >> promise >> you can live with (and not lie to you) >> >> (I'm somewhat disappointed that fsync() in Linux doesn't >> offer anything >> if your write cache is enabled) > >There is a "Flush Cache" ATA command on some SATA drives. >From what I was able to find out turning off the write cache >is a bad idea too. It wears out the drive much more quickly >because this maximizes rather this minimizes drive head >movement. This is odd, since most server drives don't enable the write cache. > >I was also able to figure out that groups of transactions >could be batched together to increase performance, if there >is a very high transaction rate. Turning off write cache Such batching is typically done by the operating system. >would prohibit this. This could still be reliable because Write caching on the drive has _nothing_ to do with batching transactions, that's done at a higher level in the operating system and relies on: 1) The batch of transactions living contiguously on the media and 2) The OS and drive supporting scatter-gather lists. >each batch of transactions could be flushed to disk >together. This could provide as much as a 1000-fold increase >in performance without losing any reliability, and depends >upon write cache being turned on. No, it doesn't. scott
From: Scott Lurndal on 20 Apr 2010 19:59 "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: >I will be renting my system from my service provider, thus >no choices are available for hardware. Both UPS and backup >generators are provided by my service provider. > >SSD have a limited life that is generally not compatible >with extremely high numbers of transactions. Where _do_ you get this stuff? I'm running an Oracle database on 64 160-GB Intel SSD's as I write this. The life of any SSD will exceed that of spinning media, even with high write to read ratios, and the performance blows them all away. I've been getting upwards of 10 gigabytes transferred per second from those drives (16 raid controllers each connected to four SSD drives configured as RAID-0, 1 TB of main memory). scott
From: Ian Collins on 20 Apr 2010 20:08 On 04/21/10 11:51 AM, Scott Lurndal wrote: > "Peter Olcott"<NoSpam(a)OCR4Screen.com> writes: >> >> There is a "Flush Cache" ATA command on some SATA drives. >> From what I was able to find out turning off the write cache >> is a bad idea too. It wears out the drive much more quickly >> because this maximizes rather this minimizes drive head >> movement. > > This is odd, since most server drives don't enable the > write cache. Isn't that filesystem dependent? ZFS enabled the drive's cache when it uses whole drives. -- Ian Collins
From: Ian Collins on 20 Apr 2010 20:13 On 04/21/10 06:06 AM, Peter Olcott wrote: > > SSD have a limited life that is generally not compatible > with extremely high numbers of transactions. > Not any more. They are used in the most transaction intensive (cache and logs) roles in many ZFS storage configurations. They are used where a very high number of IOPs are required. -- Ian Collins
From: Peter Olcott on 20 Apr 2010 20:15
"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message news:p6rzn.1$XD2.0(a)news.usenetserver.com... > "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: >> >>There is a "Flush Cache" ATA command on some SATA drives. >>From what I was able to find out turning off the write >>cache >>is a bad idea too. It wears out the drive much more >>quickly >>because this maximizes rather this minimizes drive head >>movement. > > This is odd, since most server drives don't enable the > write cache. Not enabling the write cache is the same thing as maximizing wear and tear because it maximizes head movement on writes. >>I was also able to figure out that groups of transactions >>could be batched together to increase performance, if >>there >>is a very high transaction rate. Turning off write cache > > Such batching is typically done by the operating system. That is no good for a database provider. The database provider must itself know which transactions it can count on. > >>would prohibit this. This could still be reliable because > > Write caching on the drive has _nothing_ to do with > batching > transactions, that's done at a higher level in the > operating > system and relies on: > > 1) The batch of transactions living contiguously on the > media and > 2) The OS and drive supporting scatter-gather lists. The OS and the drive both can do their own batching. If the drive could not do batching there would be no reason for drive cache. > > >>each batch of transactions could be flushed to disk >>together. This could provide as much as a 1000-fold >>increase >>in performance without losing any reliability, and depends >>upon write cache being turned on. > > No, it doesn't. > > scott |