return to the begin of InputStream
in [Java Help]
|
From: Tom Anderson on 9 Dec 2009 16:13 On Wed, 9 Dec 2009, Eric Sosman wrote: > On 12/9/2009 10:49 AM, Tom Anderson wrote: >> On Sat, 5 Dec 2009, Eric Sosman wrote: >>> >>> (Marginally topical) [...] >>> >>> Also, it's a *very* bad idea to purge /tmp blindly, even if you're >>> careful only to purge files that haven't been modified in a while. I >>> recall working with a server application that put files in /tmp and >>> mmap'ed them to share memory between its multiple processes. Since >>> simple paging I/O to and from a file opened a week ago doesn't change >>> the files' modification date, along came the customer's /tmp-purging >>> cron job and BLOOEY went the server ... >> >> Hang on, the files were open, right? So how could they be deleted? Or is >> the point that the directory entries were deleted, so when new processes >> were spawned, they couldn't open the file? And since when did writing to >> a file via an mmap not change its modification time, anyway? > > (The topicality margin gets even thinner) > > You're right: An open file can't be deleted. However, its directory > entry is removed. Then, when the application spawns a new process and > that new process tries to share memory with the others by opening and > mmap'ing the now-unfindable file, BLOOEY. (When the customer first > reported trouble, I immediately asked whether there was a cron job or > some such that periodically purged old files from /tmp. Customer > asserted -- vehemently and a bit angrily -- that OF COURSE there wasn't. > So we cobbled together some DTrace to monitor file deletions in /tmp, > and caught the non-existent cron job red-handed ...) It's at moments like this that i start whistling the Dragnet theme tune. > As for file modification times, I confess an incomplete grasp of > exactly which operations do and do not update them. However, just > poking a new value into a page that's mmap'ed from a file is not enough > to update the time stamp. Can you imagine the overhead if every memory > write trapped to the kernel to update the time? Right, but the kernel could update the mtime when it flushed pages to disk. The timestamp would reflect the time of the flush, not the write to the page, but that's how it is with buffered stream IO too. Or am i misunderstanding again, and there was no writing to disk happening? I take it you didn't have SysV shmget/shmat here? >> Either way, i'd suggest the bad idea here was putting critical >> long-lived files in /tmp. Yes, they're temporary, but not that temporary! > > It wasn't my choice. It wasn't even my company's choice. > The third party who wrote the application chose to do things that > way, and even went so far as to include "do_not_delete" as part > of the files' names. ! My company has a recent and hopefully short-lived tendency to put some fairly non-temporary things in /tmp - various files created during the lifetime of an app installation, like search indices, runtime configuration, etc. It really ought to go in /var or somewhere like that, but /tmp was the first place that sprang to mind, and since this is so far just on development machines with fairly short-lived instances, it hasn't yet gone terribly wrong. tom -- Astronomy's a fascinating subject. You look up ... and it's there. -- Patrick Moore
From: Martin Gregorie on 9 Dec 2009 16:53 On Wed, 09 Dec 2009 17:31:12 +0000, Martin Gregorie wrote: > On Wed, 09 Dec 2009 15:49:26 +0000, Tom Anderson wrote: >> And since when did writing to >> a file via an mmap not change its modification time, anyway? >> > Works OK for Linux and most *nixen, don't know about others. > Just checked with a text editor and Fedora 10: the change timestamp is updated when the file is overwritten, not when the file is closed. I don't have anything to hand that could try this with a mmapped file or for random writes. -- martin@ | Martin Gregorie gregorie. | Essex, UK org |
From: Eric Sosman on 9 Dec 2009 17:35 On 12/9/2009 4:13 PM, Tom Anderson wrote: > On Wed, 9 Dec 2009, Eric Sosman wrote: > >> On 12/9/2009 10:49 AM, Tom Anderson wrote: >>> On Sat, 5 Dec 2009, Eric Sosman wrote: >>>> >>>> (Marginally topical) [...] >> (The topicality margin gets even thinner) [...] >> [...] >> As for file modification times, I confess an incomplete grasp of >> exactly which operations do and do not update them. However, just >> poking a new value into a page that's mmap'ed from a file is not >> enough to update the time stamp. Can you imagine the overhead if every >> memory write trapped to the kernel to update the time? > > Right, but the kernel could update the mtime when it flushed pages to > disk. The timestamp would reflect the time of the flush, not the write > to the page, but that's how it is with buffered stream IO too. Or am i > misunderstanding again, and there was no writing to disk happening? (Topicality margin now thin enough to use in a microtome) No disk writes. The tmpfs file system on Solaris tries to stay memory-resident as much as possible, and only goes to the swap area(s) if RAM becomes awfully scarce. > I take it you didn't have SysV shmget/shmat here? Had 'em, sure, but the folks who wrote the application chose not to use them. (Not by default, anyhow: There was a way to use shm instead of /tmp files, but it involved some serious headaches for the administrator of the application and was seldom done.) Anyhow, this has really strayed pretty far from Java. If for some reason you're interested in pursuing it further, I suggest you E-mail me and we'll take it off-line. -- Eric Sosman esosman(a)ieee-dot-org.invalid
From: Tom Anderson on 10 Dec 2009 13:13 On Wed, 9 Dec 2009, Eric Sosman wrote: > On 12/9/2009 4:13 PM, Tom Anderson wrote: >> On Wed, 9 Dec 2009, Eric Sosman wrote: >> >>> On 12/9/2009 10:49 AM, Tom Anderson wrote: >>>> On Sat, 5 Dec 2009, Eric Sosman wrote: >>>> >>>>> (Marginally topical) [...] >>> (The topicality margin gets even thinner) [...] >>> [...] >>> As for file modification times, I confess an incomplete grasp of >>> exactly which operations do and do not update them. However, just >>> poking a new value into a page that's mmap'ed from a file is not >>> enough to update the time stamp. Can you imagine the overhead if every >>> memory write trapped to the kernel to update the time? >> >> Right, but the kernel could update the mtime when it flushed pages to >> disk. The timestamp would reflect the time of the flush, not the write >> to the page, but that's how it is with buffered stream IO too. Or am i >> misunderstanding again, and there was no writing to disk happening? > > (Topicality margin now thin enough to use in a microtome) I disagree! This group is about being a java programmer. Java programmers do things like work with mmapped working files used by long-lived processes, and as such, this is relevant to them, by whom i mean us. > No disk writes. The tmpfs file system on Solaris tries to > stay memory-resident as much as possible, and only goes to the > swap area(s) if RAM becomes awfully scarce. Ah, yes, that sort of changes the semantics. On a disk-resident FS, changing the contents of an mmapped file doesn't change the bytes on disk immediately; they will usually be written some time after the instructions doing the write are finished. Is the point that in Solaris tmpfs, the same pages are used for the mmap buffer and the definitive contents of the file? In that case, there is no flushing to disk, and so no chance to update a timestamp. Now, i have been assuming that writing to a mapped file which *does* live on disk affects the timestamp. Here is a test: http://urchin.earth.li/~twic/Code/MMapTest.java And here are the results on a machine which identifies itself as "Linux localhost 2.6.18-92.1.6.el5PAE #1 SMP Wed Jun 25 14:21:46 EDT 2008 i686 i686 i386 GNU/Linux": t0 = 1260468714000 t1 = 1260468714000 t2 = 1260468714000 t3 = 1260468714000 file = Mapd file In other words, writing to a mmapped file doesn't change the mtime. Yikes! tom -- POTATO POWER IS UNTRACEABLE POWER
From: Martin Gregorie on 11 Dec 2009 15:29
On Thu, 10 Dec 2009 18:13:48 +0000, Tom Anderson wrote: > Now, i have been assuming that writing to a mapped file which *does* > live on disk affects the timestamp. Here is a test: > ....... > In other words, writing to a mmapped file doesn't change the mtime. > Yikes! > Something about this niggled me - the effect of using coarse (millisecond) time resolution and a fast (1.6 GHz) CPU when checking the effect of single statements, so I rewrote your program to add 1000 mS pauses between interesting groups of statements. This shows that, on a Linux system anyway, the modification timestamp is updated when a RandomAccessFile is written or a MappedByteBuffer is changed. No other operations on an existing file including flushing its buffers (FileDescriptor.sync() and MappedByteBuffer.force()) affect the amended timestamp. Here are the labelled timestamps: $ java MMapTest 1 Create,open raf : = 1260562842000 2 Write : = 1260562843000 3 Sync : = 1260562843000 4 Mapped,put : = 1260562845000 5 Force : = 1260562845000 6 Close : = 1260562845000 7 Reopen raf : = 1260562845000 File content : = Mapd file 8 Read : = 1260562845000 9 Close : = 1260562845000 and here's the modified code: ===============MMapTest.java=================== import java.io.File; import java.io.IOException; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import java.nio.channels.FileChannel.MapMode; public class MMapTest { public static void main(String[] args) throws IOException, InterruptedException { File f = File.createTempFile("test", ".mmap"); RandomAccessFile raf = new RandomAccessFile(f, "rw"); showModTime(1, "Create,open raf ", f); raf.write("Test file\n".getBytes("UTF-8")); showModTime(2, "Write ", f); raf.getFD().sync(); showModTime(3, "Sync ", f); FileChannel channel = raf.getChannel(); MappedByteBuffer buf = channel.map(MapMode.READ_WRITE, 0, f.length()); buf.put("Mapd".getBytes("UTF-8")); showModTime(4, "Mapped,put ", f); buf.force(); showModTime(5, "Force ", f); channel.close(); showModTime(6, "Close ", f); raf = new RandomAccessFile(f, "r"); showModTime(7, "Reopen raf ", f); System.err.println(" File content : = " + raf.readLine()); showModTime(8, "Read ", f); raf.close(); showModTime(9, "Close ", f); } private static void showModTime(int n, String label, File f) throws InterruptedException { long t = f.lastModified(); System.err.println(n + " " + label + ": = " + t); Thread.sleep(1000); } } ===============MMapTest.java=================== -- martin@ | Martin Gregorie gregorie. | Essex, UK org | |