Questions about RAID 6
in [Debian]
|
From: Boyd Stephen Smith Jr. on 29 Apr 2010 10:10 On Wednesday 28 April 2010 20:51:18 Stan Hoeppner wrote: > Mike Bird put forth on 4/28/2010 5:48 PM: > > On Wed April 28 2010 15:10:32 Stan Hoeppner wrote: > >> Given the way most database engines do locking, you'll get zero > >> additional seek benefit on reads, and you'll take a 4x hit on writes. I > >> don't know how you could possibly argue otherwise. > > > > Linux can overlap seeks on multiple spindles, as can most operating > > systems of the last fifty years. > > Of course it can, and it even performs I/Os in parallel on multicore or smp > systems, in addition to overlapped I/O. You're still missing the point > that you have to perform 4x the writes with the 4 disk RAID 1 setup, which > reduces write performance by a factor of 4 vs a single disk, 4x the data to write does not mean that it take 4x the time. Since I/O performance is generally measured in B/s or ops/s, reducing it by a factor of 4 would mean taking 4x the time to write the same amount. That doesn't happen in a 4-way RAID-1. Instead, all the writes to the disk are triggered asynchronously, and virtually simultaneously, by the OS. At some time later, the OS waits for the disks to signal that those writes have completed. We'll assume worse-case and make the OS refuse to do anything else until all those writes have finished, even then. Your average time to having all the writes completed is just a tiny bit more than writing to a single disk. [Assuming identical disks, it's based on the average write performance of a single disk, and the standard deviation of write perform of a single disk. Something like (1 + StdDev/30) * Average. The OS can be smarter and only wait for 2 of the writes to finish, since the array is consistent at that point, which would make that number even better. In short, RAID-1 does hurt your write throughput, but not by much. It can improve both read response rate and read throughput, although the current kernel implementation isn't great at either. > and increases > write bandwidth by a factor of 4 for writes. Assuming software RAID, it does increase the bandwidth required on the PCI-X or PCIe bus -- but either is so much faster than disks to rarely be a bottleneck. Assuming SATA or SAS connection and no port multipliers, it doesn't affect the bandwidth since both are serial interfaces, so all bandwidth is measured per attached device. > Thus, on a loaded multi-user server, compared to a single disk system, > you've actually decreased your overall write throughput compared to a > single disk. In other words, if the single disk server can't handle the > I/O load, running a 4-way RAID 1 will make the situation worse. Whereas > running with RAID 10 you should get almost double the write speed of a > single disk due to the striping, even though the total number of writes to > disk is the same as with RAID 1. While I genuinely agree that RAID-1/0 makes more sense than RAID-1 when dealing with 3 or more disks, it comparative performance greatly depends on the various options you've created your array with. (Particularly, since the current kernel implementation would let you do a 1+2 mirroring [original data + 2 copies] across 4 drives and still call it RAID-1/0) In the simple case where you have 2 pairs of mirrored drives and you do the striping across the pairs (i.e. as most 4-way RAID-1/0 hardware controllers do), your read response is about the same as a single disk (just slightly less than RAID-1), your read throughput is about 4x a single disk (same as RAID-1), and your write throughput is a little less than 2x a single disk (almost 2x RAID-1). Read response (for either/both) could be better, but again, the current kernel implementation isn't that great. [RAID-1 gives you more redundancy, of course.] -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss(a)iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
From: Dan Ritter on 29 Apr 2010 12:50 On Mon, Apr 26, 2010 at 04:44:32PM -0500, Boyd Stephen Smith Jr. wrote: > On Monday 26 April 2010 09:29:28 Tim Clewlow wrote: > > I'm getting ready to build a RAID 6 with 4 x 2TB drives to start, > > but the intention is to add more drives as storage requirements > > increase. > > Since you seem fine with RAID 6, I'll assume you are also fine with RAID 5. > > I don't know what your requirements / levels of paranoia are, but RAID 5 is > probably better than RAID 6 until you are up to 6 or 7 drives; the chance of a > double failure in a 5 (or less) drive array is minuscule. It's not minuscule; it happens all the time. The key is that the double failure won't be simultaneous: first one drive goes, and then the extra stress involved in rebuilding it makes another drive go. Then it's time to replace disks and restore from backup. -dsr- -- http://tao.merseine.nu/~dsr/eula.html is hereby incorporated by reference. You can't defend freedom by getting rid of it. -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/20100429163442.GH23434(a)tao.merseine.nu
From: Ron Johnson on 30 Apr 2010 19:50 On 04/26/2010 09:29 AM, Tim Clewlow wrote: > Hi there, > > I'm getting ready to build a RAID 6 with 4 x 2TB drives to start, Since two of the drives (yes, I know the parity is striped across all the drives, but "two drives" is still the effect) are used by striping, RAID 6 with 4 drives doesn't seem rational. > but the intention is to add more drives as storage requirements > increase. > > My research/googling suggests ext3 supports 16TB volumes if block Why ext3? My kids would graduate college before the fsck completed. ext4 or xfs are the way to go. -- Dissent is patriotic, remember? -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/4BDB6A29.8010009(a)cox.net
From: Mark Allums on 30 Apr 2010 20:20 On 4/30/2010 6:39 PM, Ron Johnson wrote: > On 04/26/2010 09:29 AM, Tim Clewlow wrote: >> Hi there, >> >> I'm getting ready to build a RAID 6 with 4 x 2TB drives to start, > > Since two of the drives (yes, I know the parity is striped across all > the drives, but "two drives" is still the effect) are used by striping, > RAID 6 with 4 drives doesn't seem rational. We've taken OP to task already for this, but I guess it bears repeating. Use multiple HW controllers, and at least 7-8 drives, I believe was the consensus, given that SW RAID 6 is a performance loser and losing a controller during a rebuild is a real ruin-your-week kind of moment. But while some of us were skeptical about just how bad the performance of RAID 5 or 6 really is and wanted citation of references, more of us just questioned the perceived frugality. With four drives, wouldn't a RAID 10 be better use of resources, since you can migrate to bigger setups later? And there we were content to let it lie, until... >> but the intention is to add more drives as storage requirements >> increase. >> >> My research/googling suggests ext3 supports 16TB volumes if block > > Why ext3? My kids would graduate college before the fsck completed. > > ext4 or xfs are the way to go. I have ceased to have an opinion on this, having been taken to task, myself, about it. I believe the discussion degenerated into a nit-picky banter over the general suitability of XFS, but I may be wrong about this... _____ Seriously, ext4 is not suitable if you anticipate possible boot problems, unless you are experienced at these things. The same is true of XFS. If you *are* experienced, then more power to you. Although, I would have assumed a very experienced person would have no need to ask the question. Someone pointed out what I have come to regard as the best solution, and that is to make /boot and / (root) and the usual suspects ext3 for safety, and use ext4 or XFS or even btrfs for the data directories. (Unless OP were talking strictly about the data drives to begin with, a possibility I admit I may have overlooked.) Have I summarized adequately? MAA -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/4BDB718C.3060402(a)allums.com
From: Ron Johnson on 30 Apr 2010 22:20
On 04/30/2010 07:10 PM, Mark Allums wrote: [snip] > > Someone pointed out what I have come to regard as the best solution, and > that is to make /boot and / (root) and the usual suspects ext3 for > safety, and use ext4 or XFS or even btrfs for the data directories. That's what I do. / & /home are ext3 and /data is ext4. > (Unless OP were talking strictly about the data drives to begin with, a > possibility I admit I may have overlooked.) > > > Have I summarized adequately? > :) -- Dissent is patriotic, remember? -- To UNSUBSCRIBE, email to debian-user-REQUEST(a)lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmaster(a)lists.debian.org Archive: http://lists.debian.org/4BDB8E16.8030408(a)cox.net |