Trying to design low level hard disk manipulation program
|
From: kenney on 23 Sep 2006 12:06 In article <CbqdnRKoh7rCBo7YnZ2dnUVZ_qOdnZ2d(a)metrocastcablevision.com>, spamtrap(a)crayne.org (Bill Todd) wrote: > 1. NTFS is not, in fact, 'newer than FAT32': NTFS debuted > with Windows NT V3.1 in 1993, whereas FAT32 debuted with > Windows 95 OSR2 in 1996. Yes but FAT32 is hardly a new file system. It is just a slightly modified version of FAT system that was used by DOS. The modifications were required to allow the use of large hard drives. FAT partitions were limited to 4GB. FAT is of course still in use for floppies. Ken Young
From: Terje Mathisen on 23 Sep 2006 12:43 kenney(a)cix.compulink.co.uk wrote: > In article > <CbqdnRKoh7rCBo7YnZ2dnUVZ_qOdnZ2d(a)metrocastcablevision.com>, > spamtrap(a)crayne.org (Bill Todd) wrote: > >> 1. NTFS is not, in fact, 'newer than FAT32': NTFS debuted >> with Windows NT V3.1 in 1993, whereas FAT32 debuted with >> Windows 95 OSR2 in 1996. > > Yes but FAT32 is hardly a new file system. It is just a slightly > modified version of FAT system that was used by DOS. The > modifications were required to allow the use of large hard > drives. FAT partitions were limited to 4GB. FAT is of course > still in use for floppies. Even 'regular FAT' is an extension of the original FAT (FAT-12), which used just 12 bits for cluster numbers, thereby limiting it to 4096 blocks. This was enough for the original floppies, but had to be extended to 16-bit cluster numbers to handle the 10 MB hard drive on the IBM XT. FAT-32 was a _much_ more significant upgrade, retrofitting lots of new stuff onto the old file system: a) As the name implies, 32-bit cluster/block numbers. b) Long filenames. These were implemented with a huge hack, in the form of multiple Hidden/System Volume labels. Since regular (sub-)directory scanning code ignored the volume label, this made it possible to stuff the missing info into the existing directory data structures. c) 10-ms file update timestamps, instead of the 2-sec resolution of previous iterations. The funniest part of FAT-32 is that is is actually faster than the much more modern NTFS, as long as you have a small amount of memory avaialable to cache the most critical info. :-) OTOH, it doesn't even try to be safe in the face of power loss or other unexpected stuff. Terje -- - <Terje.Mathisen(a)hda.hydro.com> "almost all programming can be viewed as an exercise in caching"
From: Bill Todd on 23 Sep 2006 13:59 kenney(a)cix.compulink.co.uk wrote: > In article > <CbqdnRKoh7rCBo7YnZ2dnUVZ_qOdnZ2d(a)metrocastcablevision.com>, > spamtrap(a)crayne.org (Bill Todd) wrote: > >> 1. NTFS is not, in fact, 'newer than FAT32': NTFS debuted >> with Windows NT V3.1 in 1993, whereas FAT32 debuted with >> Windows 95 OSR2 in 1996. > > Yes You might better have stopped there. > but FAT32 is hardly a new file system. Indeed not: as I said, it debuted 10 years ago. Perhaps you meant to state that FAT32 was hardly new *technology* when it debuted - not that this would have had any relevance to my correction above (since the subject was specific, named implementations, not generic technology). Of course, NTFS was hardly new technology when *it* debuted, either: journaled file systems had been around for a while (Cedar comes to mind), and journaling had been used for identical purposes in databases for considerably longer. - bill
From: Bill Todd on 23 Sep 2006 14:38 Terje Mathisen wrote: .... > The funniest part of FAT-32 is that is is actually faster than the much > more modern NTFS, as long as you have a small amount of memory > avaialable to cache the most critical info. :-) While NTFS may not be optimal (I've never had the opportunity to look at really detailed implementation information, let alone the actual code), its inability to match FAT32 performance in some cases (cases not involving very large directories - or, for that matter, to some degree very large files: it takes a while to follow several hundred thousand FAT links to find the one you want even when they *are* present in RAM) should not be interpreted as evidence of poor implementation. Many industrial-strength file systems (including the major Unix variants, NTFS, VMS's ODS-2/5...) have a level of indirection between a file's directory entry and the file's data that FAT lacks: the on-disk inode in Unix, the MFT entry in NTFS, the index file entry in ODS2/5). Thus at each stage of a path look-up (directory -> subdirectory > subdirectory... -> file), there's an extra disk access (unless the inode-like structure happens to be cached) before you can get to the data at the next level. Unix requires such a structure to support its 'hard link' feature (a feature which NTFS and ODS2/5 also support and which was a design requirement for NTFS in order to support NT's Posix personality, but in VMS's case it dates back to its RSX predecessor in the early '70s so can't really be attributed to Unix). The other obvious reason for inode-like structures is to offer an alternate path to a file in case the directory structure becomes corrupted: a scan of the inode table (or equivalent) can still reach all files, though they may take a bit of sorting out (and back in the days before mirroring data became common the ability to recover from a bad disk block in a critical area was important). They also remove per-file metadata from the directory and thus allow faster directory scans - though given how frequently some of that metadata is desired while scanning a directory this can be a mixed blessing, and NTFS in fact duplicates some of it in its directories for precisely this reason. - bill
From: Anne & Lynn Wheeler on 23 Sep 2006 15:13
Bill Todd <billtodd(a)metrocast.net> writes: > Many industrial-strength file systems (including the major Unix > variants, NTFS, VMS's ODS-2/5...) have a level of indirection between > a file's directory entry and the file's data that FAT lacks: the > on-disk inode in Unix, the MFT entry in NTFS, the index file entry in > ODS2/5). Thus at each stage of a path look-up (directory -> > subdirectory > subdirectory... -> file), there's an extra disk access > (unless the inode-like structure happens to be cached) before you can > get to the data at the next level. MFD from mid-60s cms filesystem ... which had some number of incremental improvements and then saw the enhanced/extended (EDF) filesystem in the mid-70s. the original mid-60s implementation supported sparse files ... so there were various null pointers for indirect hyperblocks and datablocks that didn't actually exist. one of the unofficial early 70s incremental improvements to the cms filesystem was the directory file pointer would point directly at the data block ... instead of an indirect hyperblock ... for file that had only one data block (for small files, instead of having a minimum of two blocks, one indirect hyperblock and one data block, it would just have the single data block) another unofficial early/mid 70s incremental improvements was various kinds of data compression. I think both of these were originally done by perkin-elmer and made available on the share waterloo tape. there was some performance measurement for the p/e compression changes ... that the filesystem overhead to compress/decompress the data in the file was frequently more than offset by reduction in cpu overhead reading/writing the physical blocks to/from disk. one of the things that the mid-70s EDF extensions brought to the cms filesystem was multiple logical block size (1k, 2k, & 4k) and more levels of indirect hyperblocks ... supporting up to five levels of indirection for large blocks ... i.e. a 4k filesystem with single hyperblock supported up to 1024 four byte data block pointers. a two level hyperblock had the first level pointing to up to 1024 first-level hyperblocks which each then would point to up to 1024 4k data blocks. As a file grew, the filesystem could transition to higher levels of hyperblock indirection. in the early 70s, i had did a page-mapped layer for the original (cp67) cms filesystem ... and then later upgraded the EDF filesystem (by that time morphed into vm370) to also support page-mapped layer construct http://www.garlic.com/~lynn/subtopic.html#mmap there is some folklore that various pieces of ibm/pc and os2 filesystem characteristics were taken from cms. note also that both unix and cms trace some common heritage back to ctss. |