9,000 ZFS mount points
in [FreeBSD]
|
Prev: Point-to-point link without PPP on freebsd? Is it possible?
Next: cdrkit vs. dvd+rw-tools and the dvd iso 4gb max filesize problem
From: Erick Engelke on 5 Jan 2010 12:14 Hi, My team is looking at upgrading our FreeBSD fileserver, we've been extrememly happy with FreeBSD, the fileserver serves: umpteen NFS clients (all FreeBSD) 2,000 CIFS clients with Samba 9,000 homedirectories on about 4 TB of RAID disks on InforTrend RAID chassies, 2GB battery backed up RAM on RAID chassis with user quotas Our present box is dated. It's FreeBSD 6.2, dual processor, 4GB RAM running 32 bit OS. We will find some money to upgrade it. We've looked at (and talked to) NetApp and Sun about replacement products, but we're also evaluating an inhouse FreeBSD solution to support: all 9,000 homedirectories on 16 TB RAID disks growable to 92 TB ZFS looks nice, except that to support 9,000 user accounts, we believe we would have to have 9,000 mount points. That's how the Sun system would work. The Sun and Netapp solutions have about 64 MB servers and a lot of SATA drives. It looks like we could build a FreeBSD systems but with SAS for a similar price. The way the users hammer our hard disks, I'm really not thinking slow spinning, slower seeking SATA drives are a good idea, even though I realize ZFS and oodles more RAM to cache would help performance. Sun's Storage 7000 series family uses an 18 GB solid state hard disk which buffers the ZFS writes and is supposed to really speed up the system. Can I do the same with FreeBSD. In the end, we may purchase one of these brand name systems, but I want to give FreeBSD a fighting chance because it has been rock solid for all our server systems. We started on FreeBSD 2.0, so we've seen a lot of progress. Erick Erick Engelke erick(a)uwaterloo.ca Director PHY-3013 Engineering Computing (519) 885-1211 x35893 University of Waterloo http://www.eng.uwaterloo.ca/~erick
From: Michel Talon on 5 Jan 2010 14:49 Erick Engelke <erick(a)engmail.uwaterloo.ca> wrote: > > Sun's Storage 7000 series family uses an 18 GB solid state hard disk which > buffers the ZFS writes and is supposed to really speed up the system. Can > I do the same with FreeBSD. I think you can, and this is the best option to speed up ZFS. You should ask on a FreeBSD mailing list where developers hang up, such as freebsd-stable. The real question is: what is the reliablity of ZFS for such high intensity setup as you envision, be it on FreeBSD or Solaris. The convenience is obvious. If you envision modern hardware, such as SAS, the other question is what is reliability of corresponding driver, here also a question on mailing lists may prove useful. -- Michel TALON
From: Rob Kouwenberg on 5 Jan 2010 17:21 Erick Engelke <erick(a)engmail.uwaterloo.ca> wrote: > My team is looking at upgrading our FreeBSD fileserver, we've been > extrememly happy with FreeBSD, the fileserver serves: > > umpteen NFS clients (all FreeBSD) > 2,000 CIFS clients with Samba > 9,000 homedirectories on about 4 TB of RAID disks on InforTrend RAID > chassies, 2GB battery backed up RAM on RAID chassis > with user quotas > Our present box is dated. It's FreeBSD 6.2, dual processor, 4GB RAM > running 32 bit OS. We will find some money to upgrade it. what do e.g. w, top, lsof, vmstat 10 and netstat -an give on an hourly basis ? Know you current load before heading of to the slaughterhouse ! For getting bigger capacity disks, on 1 standard sata2 port you can get about 40Tb. 16 port sata2 cards, with 5 in the wintel box, would give you 3.2Pb on the same hardware. Would that suffice ? > We started on FreeBSD 2.0, so we've seen a lot of progress. Heck, still running 2.2.5 here, both on a 4 watts server and in a vmware imgae. It's getting about time to upgrade... :) Tough luck that BSD/OS went south and I still can't make it vmized ...
From: Michael Sierchio on 5 Jan 2010 21:38 Erick Engelke wrote: > all 9,000 homedirectories on 16 TB RAID disks growable to 92 TB > > ZFS looks nice, except that to support 9,000 user accounts, we believe > we would have to have 9,000 mount points. That's how the Sun system > would work. Doesn't have to -- it's conventional to make each user's home dir a filesystem, but not at all required. It makes quota enforcement easier, but you could do the old-fashioned /export/home/t/th/thisuser approach, and have 30 mount points, or fewer, or more. If you do go the ZFS approach, don't bother with RAID hardware at all -- JBOD and raidz1 or raidz2. > Sun's Storage 7000 series family uses an 18 GB solid state hard disk > which buffers the ZFS writes and is supposed to really speed up the > system. Can I do the same with FreeBSD. I'm skeptical about a SSD speeding up writes -- ZFS's copy-on-write semantics have numerous advantages for conventional disk arrays, and eliminate the write hole on conventional RAID5. > In the end, we may purchase one of these brand name systems, but I want > to give FreeBSD a fighting chance because it has been rock solid for all > our server systems. We started on FreeBSD 2.0, so we've seen a lot of > progress. FreeBSD 8 would be a nice platform. I don't know that SATA2 disks would be that much slower than SAS, and 2TB SATA2 drives are absurdly cheap. The thing you run out of soonest is available bays to put disks in. Do you factor power consumption and cooling requirements in your cost model? - M
From: Erick Engelke on 6 Jan 2010 09:38
On Tue, 5 Jan 2010, Michael Sierchio wrote: > Erick Engelke wrote: > >> all 9,000 homedirectories on 16 TB RAID disks growable to 92 TB Thanks to all who replied. We've been monitorring our old system for quite a while. The load levels are rarely above four, but at peak times the UFS hardware RAIDed SCSI disks are nearing 100% utilization for sustained periods. If we were to just upgrade our fileserver to FreeBSD 8 + UFS, we'd get multicores, so our CPU load is not a problem. Right now our hardware has us stuck at 4GB, but with new hardware we'd go for 64 GB or more. We won't be cheap. >> ZFS looks nice, except that to support 9,000 user accounts, we believe >> we would have to have 9,000 mount points. That's how the Sun system >> would work. > > Doesn't have to -- it's conventional to make each user's home dir > a filesystem, but not at all required. It makes quota enforcement > easier, but you could do the old-fashioned /export/home/t/th/thisuser approach, > and have 30 mount points, or fewer, or more. I thought I needed separate mount points to enforce quota at a per-user basis. My question is, will FreeBSD be okay with 9,0000 mount points. I know ZFS requires a fair bit of memory, and I don't know if there is some sort of formula for estimating how much memory for a given number of ZFS mount points. > If you do go the ZFS approach, don't bother with RAID hardware at all -- > JBOD and raidz1 or raidz2. Right. > >> Sun's Storage 7000 series family uses an 18 GB solid state hard disk >> which buffers the ZFS writes and is supposed to really speed up the >> system. Can I do the same with FreeBSD. > > I'm skeptical about a SSD speeding up writes -- ZFS's copy-on-write > semantics have numerous advantages for conventional disk arrays, and > eliminate the write hole on conventional RAID5. Interesting. > FreeBSD 8 would be a nice platform. I don't know that SATA2 disks would > be that much slower than SAS, and 2TB SATA2 drives are absurdly cheap. The > thing you run out of soonest is available bays to put disks in. Do you > factor power consumption and cooling requirements in your cost model? No, we have ample cooling, power and space. Thanks, Erick |