MappedByteBuffer
in [Java Help]
|
Prev: swing html parser
Next: Class Constants - pros and cons
From: Alan Gutierrez on 23 Jul 2010 10:33 Lew wrote: > Kevin McMurtrie wrote: >> Adding '-d64' as the first JVM option will force the 64 bit version or >> fail with a message that it's not installed. >> > > I see nothing in > <http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/ > tools/solaris/java.html> > that requires the "-d64" option to be the first option. It works though: [alan(a)postojna ~]$ java -d64 MemoryMap 6 [alan(a)postojna ~]$ java MemoryMap 6 Exception in thread "main" java.io.IOException: Cannot allocate memory at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:742) at MemoryMap.opened(MemoryMap.java:23) at MemoryMap.main(MemoryMap.java:12) [alan(a)postojna ~]$ -- Alan Gutierrez - alan(a)blogometer.com - http://twitter.com/bigeasy
From: Nigel Wade on 23 Jul 2010 11:37 On 23/07/10 14:27, Alan Gutierrez wrote: > Kevin McMurtrie wrote: >> In article <i2av27$9h7$1(a)news.eternal-september.org>, >> Alan Gutierrez <alan(a)blogometer.com> wrote: >> >>> I'm implementing file storage for a B+Tree for deployment on Linux, >>> development on OS X, and wanted to use MappedByteBuffer. This seems >>> like an easy way to address large pages of data and not have >>> implement buffering myself. >>> >>> I've done a little testing... >>> >>> import java.io.IOException; >>> import java.io.RandomAccessFile; >>> import java.nio.MappedByteBuffer; >> ... >>> Or am I misunderstanding the usage of MappedByteBuffer? >> >> >> Java 1.5 is more commonly a 32 bit process while Java 1.6 is more >> commonly a 64 bit process. You're running out of memory addresses in >> 32 bit mode. >> >> Adding '-d64' as the first JVM option will force the 64 bit version or >> fail with a message that it's not installed. > > Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank > you. > > Can you help me understand what is going on here? Is it that, Even > through `MappedByteBuffer` will page in and out of memory, you still > need a 64 bits to address entire region, and the addresses cannot overlap? > In the dim and distant past I did some work using memory mapped files in C, and I presume that the mechanism behind Java memory mapped files is similar. A memory mapped file is mapped onto a specific address in the virtual memory address space of the application. Although they are not actually occupying the memory there they still require a unique, and contiguous, address space equal to the size of the memory mapped region. So if you want to map a region of 256MB you need 256MB of contiguous address space. A value is read from a memory mapped region by accessing a particular address in the virtual memory of the application e.g. if the file is mapped at address 0x20000, and you want to read a value at offset 0x500 from that start of the file then the data at 0x20500 is read. This address must be unique, so the mapped regions cannot overlap, they must be contiguous, and all mapped regions must lie within the addressable space of the application. In a 32bit application your maximum address space is 4GB, and could be a good deal less than that due to system dependent memory limitations. Thus, your attempt to allocate multiple regions of that address space in blocks of 256MB will have a limited amount of success, depending on what contiguous regions of the address space are currently available. When you run the 64bit version of the JVM the available address space has increased by a factor of 1 billion. Therefore you are able to allocate those regions of contiguous address space for your application, even though there is still the same amount of physical RAM and system VM. -- Nigel Wade
From: Lew on 23 Jul 2010 13:32 Kevin McMurtrie wrote: >>> Adding '-d64' as the first JVM option will force the 64 bit version or >>> fail with a message that it's not installed. > Lew wrote: >> I see nothing in >> <http://download.oracle.com/docs/cd/E17409_01/javase/6/docs/technotes/ >> tools/solaris/java.html> >> that requires the "-d64" option to be the first option. > Alan Gutierrez wrote: > It works though: > Of course. That isn't the question. -- Lew
From: Peter Duniho on 23 Jul 2010 13:42 Nigel Wade wrote: > [...] > In a 32bit application your maximum address space is 4GB, and could be > a good deal less than that due to system dependent memory limitations. > Thus, your attempt to allocate multiple regions of that address space in > blocks of 256MB will have a limited amount of success, depending on what > contiguous regions of the address space are currently available. Indeed, it's worth pointing out that on Windows, the maximum address space for Java programs is only 2GB, and really only about 1.5GB (give or take) after various overhead involved in a Windows process and Java. And of course, any interesting program will have to allocate other memory objects, reducing that further (through both just the use of memory and fragmentation of the virtual address space). This issue is discussed broadly on the Internet, and I recall that there have been discussions even in this newsgroup on the topic. One should be able to find elaborations on the Windows side of the issue, as well as specific information about limits found in other operating systems (my recollection is that Mac OS X — that is, BSD Unix — and Linux have similar limitations, but one should check that for specifics if it really matters). Pete
From: Patricia Shanahan on 23 Jul 2010 16:29
On 7/23/2010 10:42 AM, Peter Duniho wrote: > Nigel Wade wrote: >> [...] >> In a 32bit application your maximum address space is 4GB, and could be >> a good deal less than that due to system dependent memory limitations. >> Thus, your attempt to allocate multiple regions of that address space in >> blocks of 256MB will have a limited amount of success, depending on what >> contiguous regions of the address space are currently available. > > Indeed, it's worth pointing out that on Windows, the maximum address > space for Java programs is only 2GB, and really only about 1.5GB (give > or take) after various overhead involved in a Windows process and Java. > And of course, any interesting program will have to allocate other > memory objects, reducing that further (through both just the use of > memory and fragmentation of the virtual address space). I assume when you say "on Windows" you mean "on 32 bit Windows"? Patricia |