MappedByteBuffer
in [Java Programming]
|
Prev: swing html parser
Next: Class Constants - pros and cons
From: Alan Gutierrez on 22 Jul 2010 22:33 I'm implementing file storage for a B+Tree for deployment on Linux, development on OS X, and wanted to use MappedByteBuffer. This seems like an easy way to address large pages of data and not have implement buffering myself. I've done a little testing... import java.io.IOException; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import java.nio.channels.FileChannel.MapMode; public class MemoryMap { public static void main(String[] args) throws IOException { int count = Integer.parseInt(args[0]); MappedByteBuffer[] mappedByteBuffers = new MappedByteBuffer[count]; for (int i = 0; i < count; i++) { mappedByteBuffers[i] = opened(i); } } public static MappedByteBuffer opened(int count) throws IOException { int size = 1024 * 1024 * 256; // * 1024; // 1 GB RandomAccessFile raf = new RandomAccessFile("hello" + count + ".raf", "rw"); try { raf.setLength(size); FileChannel channel = raf.getChannel(); try { MappedByteBuffer map = channel.map(MapMode.READ_WRITE, 0, size); channel.close(); map.put(size - 1, (byte) 7); } finally { channel.close(); } } finally { raf.close(); } raf = new RandomAccessFile("hello" + count + ".raf", "rw"); try { FileChannel channel = raf.getChannel(); try { MappedByteBuffer map = channel.map(MapMode.READ_WRITE, 0, size); channel.close(); if (map.get(size - 1) != 7) { throw new IllegalStateException(); } return map; } finally { channel.close(); } } finally { raf.close(); } } } On a smallish Linode running Ubuntu 10.4 I can create an array of hundreds of MappedByteBuffer. Here are some timings. [alan(a)maribor ~]$ time java MemoryMap 24 real 0m0.119s user 0m0.067s sys 0m0.029s [alan(a)maribor ~]$ time java MemoryMap 512 real 0m0.225s user 0m0.150s sys 0m0.092s [alan(a)maribor ~]$ On OS X running Java 1.5 I can create an array of 5, the first pass is very slow, the second pass is faster, I assume this is because OS X isn't being clever about creating the empty file. [alan(a)postojna ~]$ time java MemoryMap 5 real 0m20.920s user 0m0.089s sys 0m1.267s [alan(a)postojna ~]$ time java MemoryMap 5 real 0m0.301s user 0m0.091s sys 0m0.044s [alan(a)postojna ~]$ When I attempt to create an array of 6 250 MB `MappedByteBuffer`s I get: [alan(a)postojna ~]$ java MemoryMap 6 Exception in thread "main" java.io.IOException: Cannot allocate memory at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742) at MemoryMap.opened(MemoryMap.java:23) at MemoryMap.main(MemoryMap.java:12) [alan(a)postojna ~]$ However: [alan(a)postojna ~]$ time /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java MemoryMap 6 real 0m0.386s user 0m0.354s sys 0m0.087s [alan(a)postojna ~]$ I've created up to 24 250MB buffers using Java 1.6 on OS X. It takes a while to create them, but once created, performance seems reasonable. [alan(a)postojna ~]$ time /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java MemoryMap 24 real 3m13.779s user 0m0.358s sys 0m5.543s [alan(a)postojna ~]$ time /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java MemoryMap 24 real 0m1.646s user 0m0.364s sys 0m0.100s [alan(a)postojna ~]$ I've tried the program running on Windows XP with JDK 1.6 running in Virtual Box and I can't get an array of three pages. All this leads me to the question: am I on the right track? It seems like I'll be able to get the benefits I seek, * paging managed by the operating system, * a simple memory management strategy. on Linux and OS X with JDK 1.6. Or am I misunderstanding the usage of MappedByteBuffer? -- Alan Gutierrez - alan(a)blogometer.com - http://twitter.com/bigeasy
From: Kevin McMurtrie on 23 Jul 2010 01:30 In article <i2av27$9h7$1(a)news.eternal-september.org>, Alan Gutierrez <alan(a)blogometer.com> wrote: > I'm implementing file storage for a B+Tree for deployment on Linux, > development on OS X, and wanted to use MappedByteBuffer. This seems like > an easy way to address large pages of data and not have implement > buffering myself. > > I've done a little testing... > > import java.io.IOException; > import java.io.RandomAccessFile; > import java.nio.MappedByteBuffer; ... > Or am I misunderstanding the usage of MappedByteBuffer? Java 1.5 is more commonly a 32 bit process while Java 1.6 is more commonly a 64 bit process. You're running out of memory addresses in 32 bit mode. Adding '-d64' as the first JVM option will force the 64 bit version or fail with a message that it's not installed. -- I won't see Google Groups replies because I must filter them as spam
From: Lew on 23 Jul 2010 08:43 Kevin McMurtrie wrote: > Java 1.5 is more commonly a 32 bit process while Java 1.6 is more > commonly a 64 bit process. How do you figure that? > You're running out of memory addresses in 32 bit mode. > > Adding '-d64' as the first JVM option will force the 64 bit version or > fail with a message that it's not installed. -- Lew
From: Alan Gutierrez on 23 Jul 2010 09:27 Kevin McMurtrie wrote: > In article <i2av27$9h7$1(a)news.eternal-september.org>, > Alan Gutierrez <alan(a)blogometer.com> wrote: > >> I'm implementing file storage for a B+Tree for deployment on Linux, >> development on OS X, and wanted to use MappedByteBuffer. This seems like >> an easy way to address large pages of data and not have implement >> buffering myself. >> >> I've done a little testing... >> >> import java.io.IOException; >> import java.io.RandomAccessFile; >> import java.nio.MappedByteBuffer; > ... >> Or am I misunderstanding the usage of MappedByteBuffer? > > > Java 1.5 is more commonly a 32 bit process while Java 1.6 is more > commonly a 64 bit process. You're running out of memory addresses in 32 > bit mode. > > Adding '-d64' as the first JVM option will force the 64 bit version or > fail with a message that it's not installed. Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank you. Can you help me understand what is going on here? Is it that, Even through `MappedByteBuffer` will page in and out of memory, you still need a 64 bits to address entire region, and the addresses cannot overlap? -- Alan Gutierrez - alan(a)blogometer.com - http://twitter.com/bigeasy
From: Lew on 23 Jul 2010 10:00
Kevin McMurtrie wrote: > Adding '-d64' as the first JVM option will force the 64 bit version or > fail with a message that it's not installed. > I see nothing in <http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/ tools/solaris/java.html> that requires the "-d64" option to be the first option. -- Lew |