Writing to file delay
in [Ruby]
|
From: Matt Triviski on 21 Jul 2010 19:35 Hello All, I am having a very annoying issue. This is the situation: I am reading a file on one machine that is continuosly being written to and writing out to a socket. On another machine I am reading from the socket and writing it to a file. The problem is that when I tail the original file and the file I am creating, the new file has a delay of around 1 minute until I see the new data. It seems that the OS (Linux Redhat 5) is waiting until it gets a block of data before it actually writes to the file. I want it to write immediately. Here is a code snippet of how I am reading from the socket and writing to the file. file = File.new(fname, "w") file.sync = true buff = "" loop{ begin ios = IO.select([server.socket], [file],[server.socket, file], 0.0001) if ios[0].length > 0 buff = buff + ios[0].first.readpartial(8192) end if ios[1].length > 0 file.write buff buff = "" end rescue Errno::ECONNRESET, SocketError puts "ERROR: Lost connection to server" exit 1 rescue EOFError end sleep 0.01 } } I first used just a regular readline from the socket and then did a file.puts line, but I was having the same issue sot that is why I tried to go to select. I am wondering if this is a Linux issue, since I turned the file sync to true. However, I am not sure, so I am hoping that other code heads out there could help me out. -- Posted via http://www.ruby-forum.com/.
From: Eric Wong on 21 Jul 2010 20:01 Matt Triviski <matt.triviski(a)gmail.com> wrote: > Hello All, > > I am having a very annoying issue. This is the situation: > > I am reading a file on one machine that is continuosly being written to > and writing out to a socket. On another machine I am reading from the > socket and writing it to a file. The problem is that when I tail the > original file and the file I am creating, the new file has a delay of > around 1 minute until I see the new data. It seems that the OS (Linux > Redhat 5) is waiting until it gets a block of data before it actually > writes to the file. I want it to write immediately. > > Here is a code snippet of how I am reading from the socket and writing > to the file. > > file = File.new(fname, "w") > file.sync = true Hi Matt, "file.sync = true" looks good already. Any chance you're writing to a file on NFS or some other remote filesystem? Since you're on Linux, try stracing both the Ruby process and your tail process to see what it's doing. > buff = "" > > loop{ > begin > ios = IO.select([server.socket], [file],[server.socket, file], > 0.0001) There's no point in ever calling IO.select on a regular file, they're always ready for reading/writing. Since you're down to one socket, I wouldn't bother with select at all: file.write(server.socket.readpartial(8192)) > > rescue Errno::ECONNRESET, SocketError > puts "ERROR: Lost connection to server" > exit 1 > rescue EOFError > end > > sleep 0.01 > } > } > > I first used just a regular readline from the socket and then did a > file.puts line, but I was having the same issue sot that is why I tried > to go to select. I am wondering if this is a Linux issue, since I > turned the file sync to true. However, I am not sure, so I am hoping > that other code heads out there could help me out. Whatever it is, there's a high likelyhood that strace-ing one of the processes involved will tell you what the problem is :> -- Eric Wong
From: Joel VanderWerf on 21 Jul 2010 20:32 Eric Wong wrote: > Matt Triviski <matt.triviski(a)gmail.com> wrote: >> Hello All, >> >> I am having a very annoying issue. This is the situation: >> >> I am reading a file on one machine that is continuosly being written to >> and writing out to a socket. On another machine I am reading from the >> socket and writing it to a file. The problem is that when I tail the >> original file and the file I am creating, the new file has a delay of >> around 1 minute until I see the new data. It seems that the OS (Linux >> Redhat 5) is waiting until it gets a block of data before it actually >> writes to the file. I want it to write immediately. >> >> Here is a code snippet of how I am reading from the socket and writing >> to the file. >> >> file = File.new(fname, "w") >> file.sync = true > > Hi Matt, > > "file.sync = true" looks good already. You could try calling fsync as well (it's not a writer, just call "file.fsync"): Implementation from IO ------------------------------------------------------------------------------ ios.fsync -> 0 or nil ------------------------------------------------------------------------------ Immediately writes all buffered data in ios to disk. Note that fsync differs from using IO#sync=. The latter ensures that data is flushed from Ruby's buffers, but doesn't not guarantee that the underlying operating system actually writes it to disk. NotImplementedError is raised if the underlying operating system does not support fsync(2).
From: Matt Triviski on 21 Jul 2010 20:56 Joel VanderWerf wrote: > Eric Wong wrote: >>> Redhat 5) is waiting until it gets a block of data before it actually >> "file.sync = true" looks good already. > You could try calling fsync as well (it's not a writer, just call > "file.fsync"): > > Implementation from IO > ------------------------------------------------------------------------------ > ios.fsync -> 0 or nil > > ------------------------------------------------------------------------------ > > Immediately writes all buffered data in ios to disk. Note that > fsync differs from using IO#sync=. The latter ensures that data > is flushed from Ruby's buffers, but doesn't not guarantee that the > underlying > operating system actually writes it to disk. > > NotImplementedError is raised if the underlying operating system does > not support fsync(2). Hey fellas, thanks for the response. I tried file.fsync, and checked to make sure that the return was not nil, and it wasn't. However, I am seeing the same activity. I took out the select call and as Eric suggested and just used "file.write(server.socket.readpartial(8k))" with and that did not seem to help either. I am writing to NFS, so I am thinking that this may be the issue. I'm starting to think that there is no way for me to get what I want, except to rewrite this in C or Assembly :( -- Posted via http://www.ruby-forum.com/.
From: Eric Wong on 21 Jul 2010 21:26
Matt Triviski <matt.triviski(a)gmail.com> wrote: > I am writing to NFS, so I am thinking that this may be the issue. I'm > starting to think that there is no way for me to get what I want, except > to rewrite this in C or Assembly :( NFS is your problem, not Ruby. See the nfs(5) manpage and see if you (or your friendly systems administrator) can tune the NFS mount options to invalidate the cache more frequently. Ruby IO methods are already thin wrappers around corresponding C functions (and thinner if you use the sys{write,read,...} variants). You normally don't need to go lower-level than that. Heck, with enough time and effort, you even could write your own NFS client implementation in Ruby (not a serious suggestion :) -- Eric Wong |