Buffered RandomAccessFile
in [Java Programming]
|
Prev: Calendar question
Next: Hacking midlet signing
From: Arne Vajhøj on 25 Mar 2010 19:15 On 25-03-2010 12:26, Spud wrote: > Can someone suggest a good approach for caching parts of a large data file? > > I've got a number of very large (multi-gigabyte) files. Each file is > organized into pages of variable length. Pages can be small (< 1k) or > much larger (> 1mb). The organization of the file is similar to a btree, > and the access pattern is similar: you start at the root, skip to pages > scattered in different places in the file, and then when you get to a > leaf node you read sequentially, possibly over many megabytes. > > I need some kind of caching system so I'm not hitting the disk on every > access. Ideally, I'd be able to use a kind of buffered RandomAccessFile > object which would magically cache hunks of the file appropriately. > > I know this problem is similar to the problem databases have: how to > cache pages optimally. There's a difference, though, because my pages > are of variable length and pages in a database are usually of fixed > length. So a fixed pool of uniformly-sized pages organized in an LRU > cache won't work. > > Anyone have any insights on what I should do? Structure it like: business logic buffer layer IO layer Java RT (RandomAccessFile) Pick one of the standard cache packages - they are capable of caching objects of different sizes and can use different eviction algorithms. Arne
From: Arne Vajhøj on 25 Mar 2010 19:17 On 25-03-2010 13:50, Martin Gregorie wrote: > On Thu, 25 Mar 2010 10:24:38 -0700, Lew wrote: >> Spud wrote: >>>> Can someone suggest a good approach for caching parts of a large data >>>> file? >>> > The fastest sequential read I've ever seen was from a RAID 5 disk set > using a hardware RAID controller. This did sequential reads at something > like 3 times the speed expected from a single disk of the same type and > would write at about the same speed as a single disk. The speed gain > seemed to be because the controlled was bright enough to do look-ahead > caching. We measured this performance by serially reading several > thousand fairly small records from a sequential file. > > FWIW this was measured on an IBM AS/400, and IIRC it used 5 disk RAID > sets. I expected good sequential reading performance because RAID 5 was > the only storage method the AS/400 used. However, I didn't expect the > read speed to be 'that' good! 10 should be faster than 5 (or 50) (but also more costly). Arne
From: Arne Vajhøj on 25 Mar 2010 19:27 On 25-03-2010 13:24, Lew wrote: > Knute Johnson wrote: >> If you are sure that RandomAccessFile won't work then I'd look at >> FileChannel.map(). The time required to read a few MB off a disk these >> days is not that much. >> > > To speed up those large sequential reads, i.e., after finding the > starting point of one of those "pages", use a disk with a large > onboard buffer (common disks have buffers ranging from 2MB to 8MB) and > high rotation speed (10K or 15K rpm as opposed to 7200 or 5400). It is more like 8 and 16 MB cache on disks today. You can get high end disks with 64 MB. > Bear in mind that the OS also caches disk reads. There are a lot of > factors influencing read speed: disk-to-buffer transfer rate, buffer- > to-mainboard transfer rate, OS disk-buffering strategy, use of the > disk by other processes, including by the OS itself for memory swap > space, and more. Anything you do by hand to try to improve the > situation risks having the opposite effect. And this a very important point. The OS caches are orders of magnitude bigger than disk caches. And for read there are no big difference. There are potentially a third cache: in the RAID controller. > As with all attempts to optimize, remember the famous maxims: > 1. Don't do it. > 2. Don't do it yet. > 3. Measure performance before and after, under realistic conditions. > 4. Everything you think you know, or you assume, or you predict, is > wrong. > 5. It will work differently in the field. > 6. Nothing can fix a slow algorithm. In general very good advice. Skipping #1-2 may be justified here. If little computation is done on the data, then reading GB's data will be the bottleneck. Arne
From: Martin Gregorie on 25 Mar 2010 20:24 On Thu, 25 Mar 2010 19:17:57 -0400, Arne Vajhøj wrote: > On 25-03-2010 13:50, Martin Gregorie wrote: >> On Thu, 25 Mar 2010 10:24:38 -0700, Lew wrote: >>> Spud wrote: >>>>> Can someone suggest a good approach for caching parts of a large >>>>> data file? >>>> >> The fastest sequential read I've ever seen was from a RAID 5 disk set >> using a hardware RAID controller. This did sequential reads at >> something like 3 times the speed expected from a single disk of the >> same type and would write at about the same speed as a single disk. The >> speed gain seemed to be because the controlled was bright enough to do >> look-ahead caching. We measured this performance by serially reading >> several thousand fairly small records from a sequential file. >> >> FWIW this was measured on an IBM AS/400, and IIRC it used 5 disk RAID >> sets. I expected good sequential reading performance because RAID 5 was >> the only storage method the AS/400 used. However, I didn't expect the >> read speed to be 'that' good! > > 10 should be faster than 5 (or 50) (but also more costly). > Indeed, but again IIRC, AS/400 controllers were built to handle clusters of five hot swappable disks. I never saw the innards, but I wouldn't be surprised if the controller and 5 disk slots was a physical assembly. The machine was surprisingly modular considering that the small to medium ones occupied a single cabinet. Prize the covers off and you found a number of nodes on an SNA LU 6.2 network. Each RAID controller was a node as were the cpu, network interfaces, etc. -- martin@ | Martin Gregorie gregorie. | Essex, UK org |
From: Kevin McMurtrie on 26 Mar 2010 00:52
In article <IqMqn.492$XI1.162(a)newsfe20.iad>, Knute Johnson <nospam(a)rabbitbrush.frazmtn.com> wrote: > On 3/25/2010 9:26 AM, Spud wrote: > > Can someone suggest a good approach for caching parts of a large data file? > > > > I've got a number of very large (multi-gigabyte) files. Each file is > > organized into pages of variable length. Pages can be small (< 1k) or > > much larger (> 1mb). The organization of the file is similar to a btree, > > and the access pattern is similar: you start at the root, skip to pages > > scattered in different places in the file, and then when you get to a > > leaf node you read sequentially, possibly over many megabytes. > > > > I need some kind of caching system so I'm not hitting the disk on every > > access. Ideally, I'd be able to use a kind of buffered RandomAccessFile > > object which would magically cache hunks of the file appropriately. > > > > I know this problem is similar to the problem databases have: how to > > cache pages optimally. There's a difference, though, because my pages > > are of variable length and pages in a database are usually of fixed > > length. So a fixed pool of uniformly-sized pages organized in an LRU > > cache won't work. > > > > Anyone have any insights on what I should do? > > If you are sure that RandomAccessFile won't work then I'd look at > FileChannel.map(). The time required to read a few MB off a disk these > days is not that much. Sadly, MappedByteBuffer can only access 4GB. A 64 bit JVM allows one large file to be mapped to many MappedByteBuffers but they're still 4GB or less each. This can be coded as an LRU cache of small blocks. It's not all that complicated to manage misaligned boundaries. The hard part will be predictive prefetching to make it to perform better than RandomAccessFile. After all, RandomAccessFile does use a filesystem cache. -- I won't see Google Groups replies because I must filter them as spam |