Trying to design low level hard disk manipulation program
|
From: Terje Mathisen on 23 Sep 2006 16:00 Bill Todd wrote: > Terje Mathisen wrote: > > ... > >> The funniest part of FAT-32 is that is is actually faster than the >> much more modern NTFS, as long as you have a small amount of memory >> avaialable to cache the most critical info. :-) > > While NTFS may not be optimal (I've never had the opportunity to look at > really detailed implementation information, let alone the actual code), > its inability to match FAT32 performance in some cases (cases not > involving very large directories - or, for that matter, to some degree > very large files: it takes a while to follow several hundred thousand > FAT links to find the one you want even when they *are* present in RAM) > should not be interpreted as evidence of poor implementation. Bill, I'm sorry if it sounded like critisism of modern file systems: We don't need to start another discussion, I totally agree that NTFS (as well as pretty much every log-based file system) has a _lot_ of improvements compared to FAT32! Terje -- - <Terje.Mathisen(a)hda.hydro.com> "almost all programming can be viewed as an exercise in caching"
From: Bill Todd on 23 Sep 2006 23:41 Anne & Lynn Wheeler wrote: > Bill Todd <billtodd(a)metrocast.net> writes: >> Many industrial-strength file systems (including the major Unix >> variants, NTFS, VMS's ODS-2/5...) have a level of indirection between >> a file's directory entry and the file's data that FAT lacks: the >> on-disk inode in Unix, the MFT entry in NTFS, the index file entry in >> ODS2/5). Thus at each stage of a path look-up (directory -> >> subdirectory > subdirectory... -> file), there's an extra disk access >> (unless the inode-like structure happens to be cached) before you can >> get to the data at the next level. > > MFD from mid-60s cms filesystem It doesn't surprise me at all that DEC (as well as Unix) benefited from earlier IBM work in this area, but it's interesting that DEC changed the terminology (as I noted, cms's 'MFD' became RSX's 'volume index file', while 'MFD' got recycled by RSX to mean the top level of the directory hierarchy on the volume). .... > the original mid-60s implementation supported sparse files Something else that you can't do with a linked-list allocation map like FAT's (unless you build another virtual address map on top of it). Compression (which you mentioned later) is similarly difficult with a file of any real size. - bill
From: Bill Todd on 23 Sep 2006 23:48 Terje Mathisen wrote: > Bill Todd wrote: >> Terje Mathisen wrote: >> >> ... >> >>> The funniest part of FAT-32 is that is is actually faster than the >>> much more modern NTFS, as long as you have a small amount of memory >>> avaialable to cache the most critical info. :-) >> >> While NTFS may not be optimal (I've never had the opportunity to look >> at really detailed implementation information, let alone the actual >> code), its inability to match FAT32 performance in some cases (cases >> not involving very large directories - or, for that matter, to some >> degree very large files: it takes a while to follow several hundred >> thousand FAT links to find the one you want even when they *are* >> present in RAM) should not be interpreted as evidence of poor >> implementation. > > Bill, I'm sorry if it sounded like critisism of modern file systems: I didn't think that's what you had meant at all, but it seemed possible that someone else might think so. Besides, the specifics seemed appropriate given that they concern basic file system architecture (at least to the degree that comp.arch is about software as well as hardware architecture, and my impression is that such discussion is at least amicably tolerated here even if the majority of long-term participants may be more hardware-focused). - bill
From: Terje Mathisen on 24 Sep 2006 10:19 Bill Todd wrote: > Something else that you can't do with a linked-list allocation map like > FAT's (unless you build another virtual address map on top of it). > Compression (which you mentioned later) is similarly difficult with a > file of any real size. So what are the features that good file system should have, besides never silently dropping updates, and never allowing an inconsistent state? a) Extent-based block addressing, ideally with a background task which cleans up fragmented files. b) inode-like separation between filenames and storage allocation. c) Room in the directory structure to skip the inode completely for single-linked files with a small (1-3?) number of extents. Any others? Terje -- - <Terje.Mathisen(a)hda.hydro.com> "almost all programming can be viewed as an exercise in caching"
From: Anton Ertl on 24 Sep 2006 12:42
Terje Mathisen <terje.mathisen(a)hda.hydro.com> writes: >So what are the features that good file system should have, besides >never silently dropping updates, and never allowing an inconsistent state? Well, there are different kinds of consistency. Many file systems people only care for meta-data consistency; as long as the fsck passes, everything is fine. Who needs data, anyway? On the other extreme there is fully synchronous operation of the file system (so you don't even lose a second of work in case of a crash), but this usually results in too-slow implementations. I like the one that I call in-order semantics <http://www.complang.tuwien.ac.at/papers/czezatke&ertl00/#sect-in-order>: |The state of the file system after recovery represents all write()s |(or other changes) that occurred before a specific point in time, and |no write() (or other change) that occurred afterwards. I.e., at most |you lose a minute or so of work. Unfortunately, AFAIK all widely-used file systems provide this guarantee only in fully-synchronous mode, if at all. >a) Extent-based block addressing, ideally with a background task which >cleans up fragmented files. > >b) inode-like separation between filenames and storage allocation. > >c) Room in the directory structure to skip the inode completely for >single-linked files with a small (1-3?) number of extents. Features a and c seem to be low-level implementation details to me, rather than what I would call features. Feature b is an architectural implementation choice, but apart from the availability of hard links (which is not su useful in my experience) it is not a user-visible feature, either. Fup-To: comp.arch - anton -- M. Anton Ertl Some things have to be seen to be believed anton(a)mips.complang.tuwien.ac.at Most things have to be believed to be seen http://www.complang.tuwien.ac.at/anton/home.html |