MappedByteBuffer
in [Java Programming]
|
Prev: swing html parser
Next: Class Constants - pros and cons
From: Tom Anderson on 26 Jul 2010 08:04 On Mon, 26 Jul 2010, Esmond Pitt wrote: >> So, if a memory mapped file is closed, its address space is never >> reclaimed? That seems really wrong.... > > I told you it was a gotcha. See the Bug Parade, endless discussion about > it. Indeed. ISTR the gist of the justification for this is something like 'the JVM can't tell when the address range is truly not referenced any more, so it can't safely reuse it'. Am i right in thinking that on 64-bit machines, it's unlikely to be a problem in practice? tom -- Vive la chimie, en particulier, et la connaissance en general. -- Herve This
From: Alan Gutierrez on 26 Jul 2010 12:27 Tom Anderson wrote: > On Mon, 26 Jul 2010, Esmond Pitt wrote: > >>> So, if a memory mapped file is closed, its address space is never >>> reclaimed? That seems really wrong.... >> >> I told you it was a gotcha. See the Bug Parade, endless discussion >> about it. > > Indeed. ISTR the gist of the justification for this is something like > 'the JVM can't tell when the address range is truly not referenced any > more, so it can't safely reuse it'. > > Am i right in thinking that on 64-bit machines, it's unlikely to be a > problem in practice? I'm not sure, but... For the B+tree I've developed: The B+tree is a paged structure and at this time, it handles only reference records, or records that are of fixed with. When it does do variable length records, I'll do blocks managed by the B+tree storage implementation. Thus, I'm going to open however many `MappedByteBuffer`s as necessary to store the B+tree. Each file is a fixed length containing a fixed number of pages. As pages become free, I'll link into a linked list, but there will be no closing and reopening of the files. I curious to see if having many `MappedByteBuffer` instances of a certain size (~250MB or more) is going to be a problem in practice. In my case, I'm not trusting the leaking address space. Trying to design only for what the documentation says it will do. -- Alan Gutierrez - alan(a)blogometer.com - http://twitter.com/bigeasy
From: Paul Cager on 26 Jul 2010 17:20 On Jul 26, 1:04 pm, Tom Anderson <t...(a)urchin.earth.li> wrote: .... > ISTR the gist of the justification for this is something like 'the > JVM can't tell when the address range is truly not referenced any more, so > it can't safely reuse it'. I had never heard of that gotcha before. Looks like it is described in http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038. The explanation given by Sun sounds a bit odd to me: "Suppose that a thread operating on behalf of Alice maps a file into memory and then unmaps it. A second thread operating on behalf of Bob then maps some other file that the underlying operating system happens to assign to the same memory address. Now Alice's thread can read, and possibly even modify, the contents of Bob's file. Oops." I take that to mean that Alice could subvert Java's security manager (she couldn't subvert the operating system's permissions using this method). Have I got that right? If so then I can't really follow Sun's logic.
From: m_hess on 29 Jul 2010 02:50
On 26 Jul., 23:20, Paul Cager <paul.ca...(a)googlemail.com> wrote: > On Jul 26, 1:04 pm, Tom Anderson <t...(a)urchin.earth.li> wrote: > ... > > > ISTR the gist of the justification for this is something like 'the > > JVM can't tell when the address range is truly not referenced any more, so > > it can't safely reuse it'. > > I had never heard of that gotcha before. Looks like it is described inhttp://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038. The > explanation given by Sun sounds a bit odd to me: > > "Suppose that a thread operating on behalf of Alice maps a file > into memory and > then unmaps it. A second thread operating on behalf of Bob then maps > some > other file that the underlying operating system happens to assign to > the same > memory address. Now Alice's thread can read, and possibly even > modify, the > contents of Bob's file. Oops." > > I take that to mean that Alice could subvert Java's security manager > (she couldn't subvert the operating system's permissions using this > method). Have I got that right? If so then I can't really follow Sun's > logic. Well, I'm glad that I'm not the only person who is confused by this explanation. :-) I stumbled across this a few months back, and too me it sounds fishy. It is my understanding, that the memory that is used for the MemoryMappedFiles is part of the java process memory. It is requested by the JVM additionally to what it already got for the java heap. Sticking with their example, they say that Alice "unmapped" the file. The question is what do they mean by this _exactly_. I would expect, that it means that the JVM process has given up (or free()'d) that memory area that was used for the mapped file. So why should it be a problem if some other process now uses this for (here comes the important part) WHATEVER it wants to do? That's the job of the virtual memory management of the underlying OS to make sure no two processes colide on that level. Bottomline: To me the whole explanation SUN gave does not make sense. Maybe they left out some important details, I don't know... >>> So, if a memory mapped file is closed, its address space is never >>> reclaimed? That seems really wrong.... >> I told you it was a gotcha. See the Bug Parade, endless discussion >> about it. My experience was, that the memory will get reclaimed eventually. But the problem is that you have no control over when this happens, as there is no close() or similar for the MemoryMappedFile. Another problem is, that for as long as the memory mapped file is still "live", the file is still locked. I created a unittest for our component which used the memory mapped file, and I had to fiddle with it somewhat, to just be able to delete the temporary files the tests created. Not very "usable" ... :-( Closing remark from my side: We removed the MemoryMappedFile approach from our software. Not because of the handling problems just discussed, but because this meant giving up virtually all control over when and how I/O (many small chunks vs. large bursts) was happening. This broke our necks so we had to change over to stream based I/O. Michael |