ZFS SATA problems
in [Solaris]
|
From: Michael Laajanen on 8 Sep 2009 11:52 Hi, cindy wrote: > haven't seenOn Sep 7, 10:36 pm, Michael Laajanen > <michael_laaja...(a)yahoo.com> wrote: >> Hi, >> >> Richard B. Gilbert wrote: >>> Michael Laajanen wrote: >>>> Hi, >>>> Richard B. Gilbert wrote: >>>>> Michael Laajanen wrote: >>>>>> Hi all, >>>>>> Zfs.. wrote: >>>>>>> On Sep 7, 5:15 pm, Michael Laajanen <michael_laaja...(a)yahoo.com> >>>>>>> wrote: >>>>>>>> Hi, >>>>>>>> I am running Solaris 10 on a number of x86 nodes and have root and a >>>>>>>> second filesystem mounted over SATA to two harddisk with ZFS. >>>>>>>> I am currently testing this SATA link to the harddisks using a simple >>>>>>>> script which >>>>>>>> - creates a 40GB file1 >>>>>>>> - copy from file1 to file2 >>>>>>>> - removes file1 >>>>>>>> - copy file2 to file1 >>>>>>>> This is repeated over and over, this one of the nodes reportes >>>>>>>> error as >>>>>>>> below, anyone know what it actually means? >>>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>>> Sep 5 09:39:13 siu5 Error for command 'read sector' Error >>>>>>>> Level: Fatal >>>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>>> 123893504, Error Block: 123893541 >>>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>>> uncorrectable data error >>>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>>> error code: 0x7 >>>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>>> Sep 5 09:39:17 siu5 Error for command 'read sector' Error >>>>>>>> Level: Fatal >>>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>>> 123893504, Error Block: 123893541 >>>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>>> uncorrectable data error >>>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>>> error code: 0x7 >>>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>>> Sep 5 09:39:21 siu5 Error for command 'read sector' Error >>>>>>>> Level: Fatal >>>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>>> 123893504, Error Block: 123893541 >>>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>>> uncorrectable data error >>>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>>> error code: 0x7 >>>>>>>> Sep 5 09:39:25 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>>> Sep 5 09:39:25 siu5 Error for command 'read sector' Error >>>>>>>> Level: Fatal >>>>>>>> S >>>>>>>> /michael >>>>>>> Scrub your zpool >>>>>>> zpool scrub mypool >>>>>>> And see if it reports any dodgy data on one of the disks. If it does, >>>>>>> replace the disk. >>>>>> Could it also mean that I have a bad connection to the drives, like >>>>>> bad cables and so? Why I am asking this is all 11 nodes/hosts in the >>>>>> system are connected to a inhouse designed SATA chassis backplane, >>>>>> one or two drives per node driven via standard SAS cables from >>>>>> Sun(LSI) HBA's. >>>>>> Does anyone know of some way to enhance the error reports on these >>>>>> SATA interfaces by means of setting some "debug variables" to the >>>>>> drivers in order to get more detailed info? >>>>>> /michael >>>>> What more do you think you need to know? The drive has made multiple >>>>> attempts to read block 123893541 and has encountered an error on each >>>>> attempt! >>>>> The very first thing to do is to try to back up your data. With a >>>>> little bit of luck you may be able to make a good backup. >>>> I do not have any data on the drives, I am testing a hard disk chassis. >>>>> You could try replacing cables but I think it would be a complete >>>>> waste of time. The reported error suggests a problem with the disk. >>>>> A bad cable or a poorly seated connector would almost certainly show >>>>> different error messages. >>>> That is exactly what I would like to know, what error messages can be >>>> seen on a standard S10, and if it possible to make the driver more >>>> verbose in order to get as much detailed info as possible! >>>>> At the very least your disk has a corrupted block. It's conceivable >>>>> that rewriting that block might fix it. I would not want to trust >>>>> such a disk with my valuable data. >>>> /michael >>> I don't think that a vanilla Solaris system is able to do diagnosis at >>> that level! >> Could be so, but quite often there are flags that can be set to enhance >> the report level even on production systems. >> >> >> >>> IF you have a Sun service contract, Sun might be able to diagnose the >>> fault in a little more detail. >> The chassis is own designed and the chassis is what I am trying to verify.> The basic problem is clear and I think you are just wasting time trying >>> to find a work around. A new disk is simply not that expensive, >>> especially if you don't buy it from Sun! If you want to pinch pennies, >>> you could probably find a replacement on e-Bay. >> I have no problem getting new drives, but that assumes that that there >> is a drive fault on not the elctrical connection! >> I just like to pin point as detailed as possible before I decide what to do! >> >> /michael > > > Hi Michael, > > You might try using fmdump -eV to show detailed error messages > involving > this disk. The date/time stamps will identify how long this disk has > had problems. > > Cindy Hi I will look into that once back in the lab thanks. /Michael
From: Michael Laajanen on 8 Sep 2009 11:54 Hi, Chris Ridd wrote: > On 2009-09-08 14:38:30 +0100, "Richard B. Gilbert" > <rgilbert88(a)comcast.net> said: > >> Solaris is not in the business of diagnosing defective hardware. > > Doesn't <http://opensolaris.org/os/community/fm/> suggest otherwise? > That points at the same as the reply by Cindy, thanks both of you :) /michael
From: Richard B. Gilbert on 8 Sep 2009 14:28 Chris Ridd wrote: > On 2009-09-08 14:38:30 +0100, "Richard B. Gilbert" > <rgilbert88(a)comcast.net> said: > >> Solaris is not in the business of diagnosing defective hardware. > > Doesn't <http://opensolaris.org/os/community/fm/> suggest otherwise? > Never seen it! I'm running S8, 9, and 10 on various machines. A newer version of Solaris MIGHT provide a little more detail. The bottom line is still that Sun Service has diagnostic software to test and exercise the hardware. You still either pay Sun to replace the disk or you do it yourself. I don't find that drive replacement is beyond my modest skills as a repair person. It's much cheaper than getting someone from Sun to do it. Now if I had an employer that had a service contract, I'd just pick up the phone and say "Come and fix it." If I were feeling generous I might suggest that they bring a replacement disk. It generally speeds things up if the service guy/gal has the right parts!
From: Richard B. Gilbert on 8 Sep 2009 15:46 Michael Laajanen wrote: > Hi, > > Richard B. Gilbert wrote: >> Michael Laajanen wrote: >>> Hi, >>> Richard B. Gilbert wrote: >>>> Michael Laajanen wrote: >>>>> Hi, >>>>> >>>>> Richard B. Gilbert wrote: > <snip> > >> >>>> >>>> IF you have a Sun service contract, Sun might be able to diagnose >>>> the fault in a little more detail. >>>> >>> The chassis is own designed and the chassis is what I am trying to >>> verify. >>>> The basic problem is clear and I think you are just wasting time >>>> trying to find a work around. A new disk is simply not that >>>> expensive, especially if you don't buy it from Sun! If you want to >>>> pinch pennies, you could probably find a replacement on e-Bay. >>>> >>> I have no problem getting new drives, but that assumes that that >>> there is a drive fault on not the elctrical connection! >>> I just like to pin point as detailed as possible before I decide what >>> to do! >>> >>> /michael >>> >> >> The error message strongly suggests a problem with one block on disk. >> The problem may be more serious, more complicated, or both. >> >> Solaris is not in the business of diagnosing defective hardware. Sun, >> and other manufacturers write diagnostic software to exercise and test >> hardware. Sun Service should have access to such software. You may >> have write your own. Doing so requires both a good knowledge of the >> hardware and good knowledge of the software. >> >> It's easier and cheaper (if your time has value) to replace the drive. > Replacing the drive is not what it is about, if it is defective yes but > so far I will not rule out the others parts se below. > > I think you have not understood what I have done, I have designed a > harddisk chassis with 17 disks which is connected to a number of > servers, during testing of this chassis I have received these errors. > > /michael I don't think YOU understand the error message. It's trying and failing to read a specific block on a specific disk. The most likely cause is a bad block on the disk. A faulty cable would affect multiple blocks or the whole disk.
From: Thomas Tornblom on 8 Sep 2009 17:48
Michael Laajanen <michael_laajanen(a)yahoo.com> writes: .... >> The error message strongly suggests a problem with one block on >> disk. The problem may be more serious, more complicated, or both. >> Solaris is not in the business of diagnosing defective hardware. >> Sun, and other manufacturers write diagnostic software to exercise >> and test hardware. Sun Service should have access to such software. >> You may have write your own. Doing so requires both a good >> knowledge of the hardware and good knowledge of the software. >> It's easier and cheaper (if your time has value) to replace the >> drive. > Replacing the drive is not what it is about, if it is defective yes but > so far I will not rule out the others parts se below. It is the *disk* that is telling the system that it has a problem, not the system detecting a problem with the disk. > > I think you have not understood what I have done, I have designed a > harddisk chassis with 17 disks which is connected to a number of > servers, during testing of this chassis I have received these errors. So? > > /michael Thomas |