ZFS SATA problems
in [Solaris]
|
From: Chris Ridd on 8 Sep 2009 01:05 On 2009-09-08 05:36:11 +0100, Michael Laajanen <michael_laajanen(a)yahoo.com> said: > Hi, > Richard B. Gilbert wrote: >> I don't think that a vanilla Solaris system is able to do diagnosis at >> that level! > Could be so, but quite often there are flags that can be set to enhance > the report level even on production systems. Does FMA (fmadm) show anything? -- Chris
From: Richard B. Gilbert on 8 Sep 2009 09:38 Michael Laajanen wrote: > Hi, > Richard B. Gilbert wrote: >> Michael Laajanen wrote: >>> Hi, >>> >>> Richard B. Gilbert wrote: >>>> Michael Laajanen wrote: >>>>> Hi all, >>>>> >>>>> Zfs.. wrote: >>>>>> On Sep 7, 5:15 pm, Michael Laajanen <michael_laaja...(a)yahoo.com> >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I am running Solaris 10 on a number of x86 nodes and have root and a >>>>>>> second filesystem mounted over SATA to two harddisk with ZFS. >>>>>>> >>>>>>> I am currently testing this SATA link to the harddisks using a >>>>>>> simple >>>>>>> script which >>>>>>> >>>>>>> - creates a 40GB file1 >>>>>>> - copy from file1 to file2 >>>>>>> - removes file1 >>>>>>> - copy file2 to file1 >>>>>>> >>>>>>> This is repeated over and over, this one of the nodes reportes >>>>>>> error as >>>>>>> below, anyone know what it actually means? >>>>>>> >>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>> Sep 5 09:39:13 siu5 Error for command 'read sector' Error >>>>>>> Level: Fatal >>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>> 123893504, Error Block: 123893541 >>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>> uncorrectable data error >>>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>> error code: 0x7 >>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>> Sep 5 09:39:17 siu5 Error for command 'read sector' Error >>>>>>> Level: Fatal >>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>> 123893504, Error Block: 123893541 >>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>> uncorrectable data error >>>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>> error code: 0x7 >>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>> Sep 5 09:39:21 siu5 Error for command 'read sector' Error >>>>>>> Level: Fatal >>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Requested Block >>>>>>> 123893504, Error Block: 123893541 >>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Sense Key: >>>>>>> uncorrectable data error >>>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' >>>>>>> error code: 0x7 >>>>>>> Sep 5 09:39:25 siu5 gda: [ID 107833 kern.warning] WARNING: >>>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): >>>>>>> Sep 5 09:39:25 siu5 Error for command 'read sector' Error >>>>>>> Level: Fatal >>>>>>> S >>>>>>> >>>>>>> /michael >>>>>> >>>>>> Scrub your zpool >>>>>> >>>>>> zpool scrub mypool >>>>>> >>>>>> And see if it reports any dodgy data on one of the disks. If it does, >>>>>> replace the disk. >>>>> Could it also mean that I have a bad connection to the drives, like >>>>> bad cables and so? Why I am asking this is all 11 nodes/hosts in >>>>> the system are connected to a inhouse designed SATA chassis >>>>> backplane, one or two drives per node driven via standard SAS >>>>> cables from Sun(LSI) HBA's. >>>>> >>>>> Does anyone know of some way to enhance the error reports on these >>>>> SATA interfaces by means of setting some "debug variables" to the >>>>> drivers in order to get more detailed info? >>>>> >>>>> /michael >>>> >>>> What more do you think you need to know? The drive has made >>>> multiple attempts to read block 123893541 and has encountered an >>>> error on each attempt! >>>> >>>> The very first thing to do is to try to back up your data. With a >>>> little bit of luck you may be able to make a good backup. >>>> >>> I do not have any data on the drives, I am testing a hard disk chassis. >>> >>>> You could try replacing cables but I think it would be a complete >>>> waste of time. The reported error suggests a problem with the >>>> disk. A bad cable or a poorly seated connector would almost >>>> certainly show different error messages. >>> That is exactly what I would like to know, what error messages can be >>> seen on a standard S10, and if it possible to make the driver more >>> verbose in order to get as much detailed info as possible! >>> >>>> >>>> At the very least your disk has a corrupted block. It's conceivable >>>> that rewriting that block might fix it. I would not want to trust >>>> such a disk with my valuable data. >>> >>> /michael >> >> >> I don't think that a vanilla Solaris system is able to do diagnosis at >> that level! > Could be so, but quite often there are flags that can be set to enhance > the report level even on production systems. > >> >> IF you have a Sun service contract, Sun might be able to diagnose the >> fault in a little more detail. >> > The chassis is own designed and the chassis is what I am trying to verify. >> The basic problem is clear and I think you are just wasting time >> trying to find a work around. A new disk is simply not that >> expensive, especially if you don't buy it from Sun! If you want to >> pinch pennies, you could probably find a replacement on e-Bay. >> > I have no problem getting new drives, but that assumes that that there > is a drive fault on not the elctrical connection! > I just like to pin point as detailed as possible before I decide what to > do! > > /michael > The error message strongly suggests a problem with one block on disk. The problem may be more serious, more complicated, or both. Solaris is not in the business of diagnosing defective hardware. Sun, and other manufacturers write diagnostic software to exercise and test hardware. Sun Service should have access to such software. You may have write your own. Doing so requires both a good knowledge of the hardware and good knowledge of the software. It's easier and cheaper (if your time has value) to replace the drive.
From: Chris Ridd on 8 Sep 2009 10:43 On 2009-09-08 14:38:30 +0100, "Richard B. Gilbert" <rgilbert88(a)comcast.net> said: > Solaris is not in the business of diagnosing defective hardware. Doesn't <http://opensolaris.org/os/community/fm/> suggest otherwise? -- Chris
From: cindy on 8 Sep 2009 11:14 haven't seenOn Sep 7, 10:36 pm, Michael Laajanen <michael_laaja...(a)yahoo.com> wrote: > Hi, > > Richard B. Gilbert wrote: > > Michael Laajanen wrote: > >> Hi, > > >> Richard B. Gilbert wrote: > >>> Michael Laajanen wrote: > >>>> Hi all, > > >>>> Zfs.. wrote: > >>>>> On Sep 7, 5:15 pm, Michael Laajanen <michael_laaja...(a)yahoo.com> > >>>>> wrote: > >>>>>> Hi, > > >>>>>> I am running Solaris 10 on a number of x86 nodes and have root and a > >>>>>> second filesystem mounted over SATA to two harddisk with ZFS. > > >>>>>> I am currently testing this SATA link to the harddisks using a simple > >>>>>> script which > > >>>>>> - creates a 40GB file1 > >>>>>> - copy from file1 to file2 > >>>>>> - removes file1 > >>>>>> - copy file2 to file1 > > >>>>>> This is repeated over and over, this one of the nodes reportes > >>>>>> error as > >>>>>> below, anyone know what it actually means? > > >>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.warning] WARNING: > >>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): > >>>>>> Sep 5 09:39:13 siu5 Error for command 'read sector' Error > >>>>>> Level: Fatal > >>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Requested Block > >>>>>> 123893504, Error Block: 123893541 > >>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Sense Key: > >>>>>> uncorrectable data error > >>>>>> Sep 5 09:39:13 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' > >>>>>> error code: 0x7 > >>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.warning] WARNING: > >>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): > >>>>>> Sep 5 09:39:17 siu5 Error for command 'read sector' Error > >>>>>> Level: Fatal > >>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Requested Block > >>>>>> 123893504, Error Block: 123893541 > >>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Sense Key: > >>>>>> uncorrectable data error > >>>>>> Sep 5 09:39:17 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' > >>>>>> error code: 0x7 > >>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.warning] WARNING: > >>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): > >>>>>> Sep 5 09:39:21 siu5 Error for command 'read sector' Error > >>>>>> Level: Fatal > >>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Requested Block > >>>>>> 123893504, Error Block: 123893541 > >>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Sense Key: > >>>>>> uncorrectable data error > >>>>>> Sep 5 09:39:21 siu5 gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' > >>>>>> error code: 0x7 > >>>>>> Sep 5 09:39:25 siu5 gda: [ID 107833 kern.warning] WARNING: > >>>>>> /pci@0,0/pci-ide(a)1f,2/ide@1/cmdk@0,0 (Disk1): > >>>>>> Sep 5 09:39:25 siu5 Error for command 'read sector' Error > >>>>>> Level: Fatal > >>>>>> S > > >>>>>> /michael > > >>>>> Scrub your zpool > > >>>>> zpool scrub mypool > > >>>>> And see if it reports any dodgy data on one of the disks. If it does, > >>>>> replace the disk. > >>>> Could it also mean that I have a bad connection to the drives, like > >>>> bad cables and so? Why I am asking this is all 11 nodes/hosts in the > >>>> system are connected to a inhouse designed SATA chassis backplane, > >>>> one or two drives per node driven via standard SAS cables from > >>>> Sun(LSI) HBA's. > > >>>> Does anyone know of some way to enhance the error reports on these > >>>> SATA interfaces by means of setting some "debug variables" to the > >>>> drivers in order to get more detailed info? > > >>>> /michael > > >>> What more do you think you need to know? The drive has made multiple > >>> attempts to read block 123893541 and has encountered an error on each > >>> attempt! > > >>> The very first thing to do is to try to back up your data. With a > >>> little bit of luck you may be able to make a good backup. > > >> I do not have any data on the drives, I am testing a hard disk chassis. > > >>> You could try replacing cables but I think it would be a complete > >>> waste of time. The reported error suggests a problem with the disk. > >>> A bad cable or a poorly seated connector would almost certainly show > >>> different error messages. > >> That is exactly what I would like to know, what error messages can be > >> seen on a standard S10, and if it possible to make the driver more > >> verbose in order to get as much detailed info as possible! > > >>> At the very least your disk has a corrupted block. It's conceivable > >>> that rewriting that block might fix it. I would not want to trust > >>> such a disk with my valuable data. > > >> /michael > > > I don't think that a vanilla Solaris system is able to do diagnosis at > > that level! > > Could be so, but quite often there are flags that can be set to enhance > the report level even on production systems. > > > > > IF you have a Sun service contract, Sun might be able to diagnose the > > fault in a little more detail. > > The chassis is own designed and the chassis is what I am trying to verify.> The basic problem is clear and I think you are just wasting time trying > > to find a work around. A new disk is simply not that expensive, > > especially if you don't buy it from Sun! If you want to pinch pennies, > > you could probably find a replacement on e-Bay. > > I have no problem getting new drives, but that assumes that that there > is a drive fault on not the elctrical connection! > I just like to pin point as detailed as possible before I decide what to do! > > /michael Hi Michael, You might try using fmdump -eV to show detailed error messages involving this disk. The date/time stamps will identify how long this disk has had problems. Cindy
From: Michael Laajanen on 8 Sep 2009 11:50
Hi, Richard B. Gilbert wrote: > Michael Laajanen wrote: >> Hi, >> Richard B. Gilbert wrote: >>> Michael Laajanen wrote: >>>> Hi, >>>> >>>> Richard B. Gilbert wrote: <snip> > >>> >>> IF you have a Sun service contract, Sun might be able to diagnose the >>> fault in a little more detail. >>> >> The chassis is own designed and the chassis is what I am trying to >> verify. >>> The basic problem is clear and I think you are just wasting time >>> trying to find a work around. A new disk is simply not that >>> expensive, especially if you don't buy it from Sun! If you want to >>> pinch pennies, you could probably find a replacement on e-Bay. >>> >> I have no problem getting new drives, but that assumes that that there >> is a drive fault on not the elctrical connection! >> I just like to pin point as detailed as possible before I decide what >> to do! >> >> /michael >> > > The error message strongly suggests a problem with one block on disk. > The problem may be more serious, more complicated, or both. > > Solaris is not in the business of diagnosing defective hardware. Sun, > and other manufacturers write diagnostic software to exercise and test > hardware. Sun Service should have access to such software. You may > have write your own. Doing so requires both a good knowledge of the > hardware and good knowledge of the software. > > It's easier and cheaper (if your time has value) to replace the drive. Replacing the drive is not what it is about, if it is defective yes but so far I will not rule out the others parts se below. I think you have not understood what I have done, I have designed a harddisk chassis with 17 disks which is connected to a number of servers, during testing of this chassis I have received these errors. /michael |