"Forever" fsck?
in [Linux Hardware]
|
Prev: SB Audigy SE and linux problems
Next: ata1 timeout
From: Andrew Gideon on 27 Aug 2006 11:25 On a Fedora Core 3 machine, I've an fsck that's lasted many hours now, and shows no sign of ever completing. Cycles look like: Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 7 has illegal block(s). Clear? yes Illegal block #20494 (1156635280) in inode 7. CLEARED. Illegal block #20495 (1137955446) in inode 7. CLEARED. Illegal block #20496 (1156635280) in inode 7. CLEARED. Illegal block #20526 (1156616080) in inode 7. CLEARED. Illegal block #20527 (1156616064) in inode 7. CLEARED. Illegal block #20528 (1156616064) in inode 7. CLEARED. Illegal block #20686 (1137955446) in inode 7. CLEARED. Illegal block #20687 (1137955446) in inode 7. CLEARED. Illegal block #20688 (1137955446) in inode 7. CLEARED. Illegal block #20689 (1156691328) in inode 7. CLEARED. Illegal block #20719 (1137955448) in inode 7. CLEARED. Too many illegal blocks in inode 7. Clear inode? yes Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 7 has illegal block(s). Clear? yes Illegal block #20494 (1156635280) in inode 7. CLEARED. Illegal block #20495 (1137955446) in inode 7. CLEARED. Illegal block #20496 (1156635280) in inode 7. CLEARED. Illegal block #20526 (1156616080) in inode 7. CLEARED. Illegal block #20527 (1156616064) in inode 7. CLEARED. Illegal block #20528 (1156616064) in inode 7. CLEARED. Illegal block #20686 (1137955446) in inode 7. CLEARED. Illegal block #20687 (1137955446) in inode 7. CLEARED. Illegal block #20688 (1137955446) in inode 7. CLEARED. Illegal block #20689 (1156691595) in inode 7. CLEARED. Illegal block #20719 (1137955448) in inode 7. CLEARED. Too many illegal blocks in inode 7. Clear inode? yes .... This is an ext3 file system which is just about empty. However, it has 300+G of capacity and - this part is weird - when I ran df on it it showed about 90+G in use. I'd unmounted it and remounted it on a different mount point, and the df oddity persisted. So it wasn't an artifact of a file being held open. Any thoughts? The file system is largely empty because I copied everything off it so that I could examine this in more detail. So I've no problem leaving the fsck running (though I am worried that the process might outlive me {8^). I do see that the block numbers are increasing, even though it is always inode 7. So this doesn't at least appear to be an *infinite* loop. Is there any better way to fix this problem (whatever the problem is)? And what *is* the problem here? Thanks... Andrew
From: Dances With Crows on 27 Aug 2006 13:50 On Sun, 27 Aug 2006 11:25:21 -0400, Andrew Gideon staggered into the Black Sun and said: > Restarting e2fsck from the beginning... > Pass 1: Checking inodes, blocks, and sizes > Inode 7 has illegal block(s). Clear? yes > Illegal block #20494 (1156635280) in inode 7. CLEARED. [snip] > Too many illegal blocks in inode 7. > Clear inode? yes If you're going to answer "Y" to everything, why not use e2fsck -y ? > Restarting e2fsck from the beginning... > Pass 1: Checking inodes, blocks, and sizes > Inode 7 has illegal block(s). Clear? yes [snip] > Too many illegal blocks in inode 7. > Clear inode? yes > > This is an ext3 file system which is just about empty. However, it > has 300+G of capacity and when I ran df on it it showed about 90+G in > use. There is something beyond normal filesystem problems here. Inode 7 is right before the journal inode (8, usually), and it may have some sort of special meaning/use in ext3. I didn't see anything in a cursory grep through /usr/src/linux/fs/ext3/ , but IANAFilesystemGuru. Check dmesg and make sure you're not seeing messages like "hdXY: error 131072 (BAD_CRUD) reading sector 12". > I do see that the block numbers are increasing, even though it is > always inode 7. So this doesn't at least appear to be an *infinite* > loop. Is there any better way to fix this problem (whatever the > problem is)? And what *is* the problem here? This is probably some sort of hardware error, maybe combined with a bizarre filesystem corruption problem. If you're sure that there's no hardware problem, the quickest/easiest way to get back to normal is probably to mke2fs -j the partition. -- Matt G|There is no Darkness in Eternity/But only Light too dim for us to see Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong http://www.brainbench.com / I don't practice what I preach, because I'm -----------------------------/ not the kind of person I'm preaching to.
From: Andrew Gideon on 29 Aug 2006 09:49 On Sun, 27 Aug 2006 12:50:54 -0500, Dances With Crows wrote: > If you're sure that there's no > hardware problem, the quickest/easiest way to get back to normal is > probably to mke2fs -j the partition. I am *not* sure that there's no hardware problem. I did find on the console messages like: SCSI error : <3 0 0 0> return code = 0x10070000 end_request: I/O error, dev sdd, sector 976794177 SCSI error : <3 0 0 0> return code = 0x10070000 end_request: I/O error, dev sdd, sector 976794185 SCSI error : <3 0 0 0> return code = 0x10070000 end_request: I/O error, dev sdd, sector 976794193 SCSI error : <3 0 0 0> return code = 0x10070000 end_request: I/O error, dev sdd, sector 976794201 SCSI error : <3 0 0 0> return code = 0x10070000 end_request: I/O error, dev sdd, sector 976794209 ... but these only occurred very early in the process. Since the first few runs of fsck, these have ceased. In fact, I think that this was from when I ran fsck with the badblocks option, come to think of it. One more piece of information: the fsck is now (yes, I left it running for a couple of days!) in an infinite loop. It repeats: Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 7 has illegal block(s). Clear? yes Illegal block #20494 (1156635280) in inode 7. CLEARED. Illegal block #20495 (1137955446) in inode 7. CLEARED. Illegal block #20496 (1156635280) in inode 7. CLEARED. Illegal block #20526 (1156616080) in inode 7. CLEARED. Illegal block #20527 (1156616064) in inode 7. CLEARED. Illegal block #20528 (1156616064) in inode 7. CLEARED. Illegal block #20686 (1137955446) in inode 7. CLEARED. Illegal block #20687 (1137955446) in inode 7. CLEARED. Illegal block #20688 (1137955446) in inode 7. CLEARED. Illegal block #20689 (1156857862) in inode 7. CLEARED. Illegal block #20719 (1137955448) in inode 7. CLEARED. Too many illegal blocks in inode 7. Clear inode? yes Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 7 has illegal block(s). Clear? yes ... with the same block numbers now. I didn't use the -k option when I last ran fsck with the badblocks option. I've just started that now. I'm curious if it'll help (if the problem blocks will be marked bad and subsequently ignored). - Andrew
From: Andrew Gideon on 30 Aug 2006 19:47 On Tue, 29 Aug 2006 09:49:32 -0400, Andrew Gideon wrote: > I didn't use the -k option when I last ran fsck with the badblocks option. > I've just started that now. I'm curious if it'll help (if the problem > blocks will be marked bad and subsequently ignored). It looks like inode 7 is the "bad blocks inode". ... Restarting e2fsck from the beginning... Checking for bad blocks (read-only test): done Pass 1: Checking inodes, blocks, and sizes Bad block inode has an indirect block (4) that conflicts with filesystem metadata. CLEARED. The bad block inode has probably been corrupted. You probably should stop now and run e2fsck -c to scan for bad blocks in the filesystem. Continue? yes Inode 7 has illegal block(s). Clear? yes Illegal block #20494 (1156964847) in inode 7. CLEARED. Illegal block #20495 (1137955446) in inode 7. CLEARED. Illegal block #20496 (1156964847) in inode 7. CLEARED. Illegal block #20526 (1156616080) in inode 7. CLEARED. Illegal block #20527 (1156616064) in inode 7. CLEARED. Illegal block #20528 (1156616064) in inode 7. CLEARED. Illegal block #20686 (1137955446) in inode 7. CLEARED. Illegal block #20687 (1137955446) in inode 7. CLEARED. Illegal block #20688 (1137955446) in inode 7. CLEARED. Illegal block #20689 (1156952934) in inode 7. CLEARED. Illegal block #20719 (1137955448) in inode 7. CLEARED. Too many illegal blocks in inode 7. Clear inode? yes Restarting e2fsck from the beginning... ... Again, it's in an infinite look. I don't see anything but to reformat. Any other suggestions? - Andrew
From: Thomas Richter on 31 Aug 2006 08:26
Andrew Gideon wrote: > On Tue, 29 Aug 2006 09:49:32 -0400, Andrew Gideon wrote: > > >>I didn't use the -k option when I last ran fsck with the badblocks option. >>I've just started that now. I'm curious if it'll help (if the problem >>blocks will be marked bad and subsequently ignored). > > > It looks like inode 7 is the "bad blocks inode". > > Again, it's in an infinite look. I don't see anything but to reformat. > Any other suggestions? Yes. Please try to run the smarttools on the drive and see what they report. It could be a dying hard disk. So long, Thomas |