Intel and AMD RDMA implementation
|
Prev: Simple Hack To Get $2500 To Your PayPal Account.
Next: ARM-based desktop computer ? (Hybrid computers ?: Low + High performance ;))
From: Tim McCaffrey on 27 Jul 2010 13:49 In article <Ba6dne7jX9etdNTRnZ2dnUVZ_qidnZ2d(a)giganews.com>, "newsgroupatcomp-arch.net" says... > >In UNIX terms: >[...] >Occasionally people try to DMA directly into user space. E.g. big files. > >The biggest problem is not the cache snooping, it is the fact that the >standard way of getting burst memory accesses is to use cacheable >memory. And the problem that the buffer cache entry is often touched >only once, during the copy, so it displaces useful data from the cache. Yes, one might say the statement "In UNIX terms" is the biggest problem. With Unix (and, honestly, Windows as well), the user owns the I/O buffer. Which is kind of silly, since the I/O library typically has to copy from/to that buffer from/to what ever variables, etc, are used in the actual write/read. If the buffer was under the OS's control (do a read, get a pointer back, when you do the next read, the previous pointer is invalid). Then the OS would be allowed to DMA directly into user space, because that buffer would be under OS control. It would even allow no-copy access to the disk buffer cache (with appropriate page table remapping & access control). Take, for example, the CDC I/O subsystem. When you make an I/O request it goes to the peripherial processor (via the OS). Circular buffers are used, if you are doing a read, the part NOT written is basically owned by the OS (in other words, don't count on the contents), the part written to is owned by the application. For writes just the opposite. An I/O request is complete when the number of words requested is transferred, or the circular buffer is full. One trick that I thought was neat, was you could request a read & a write from the same circular buffer, if you did it just right, the PPs would do the file copy and never bother the application at all. It was as optimal as you could get for a file copy. Compare that to modern OSs (I think Linux has some voodoo to accomplish something like that for FTPs). - Tim
From: Rick Jones on 27 Jul 2010 15:20 Tim McCaffrey <timcaffrey(a)aol.com> wrote: > With Unix (and, honestly, Windows as well), the user owns the I/O > buffer. Which is kind of silly, since the I/O library typically has > to copy from/to that buffer from/to what ever variables, etc, are > used in the actual write/read. > If the buffer was under the OS's control (do a read, get a pointer > back, when you do the next read, the previous pointer is invalid). > Then the OS would be allowed to DMA directly into user space, > because that buffer would be under OS control. It would even allow > no-copy access to the disk buffer cache (with appropriate page table > remapping & access control). With O_DIRECT or an asynchronous interface, the "owned by the application" buffer can be used for the DMA. rick jones -- The computing industry isn't as much a game of "Follow The Leader" as it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose." - Rick Jones these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
From: Terje Mathisen "terje.mathisen at on 27 Jul 2010 16:00 Tim McCaffrey wrote: > One trick that I thought was neat, was you could request a read& a write from > the same circular buffer, if you did it just right, the PPs would do the file > copy and never bother the application at all. It was as optimal as you could > get for a file copy. Compare that to modern OSs (I think Linux has some > voodoo to accomplish something like that for FTPs). I thought there was a standard unix api called something like CopyFile() which allows the OS to employ all the tricks it can/want, but I can't find it. Wishful thinking? Terje -- - <Terje.Mathisen at tmsw.no> "almost all programming can be viewed as an exercise in caching"
From: Rick Jones on 27 Jul 2010 20:53 Terje Mathisen <"terje.mathisen at tmsw.no"> wrote: > I thought there was a standard unix api called something like > CopyFile() which allows the OS to employ all the tricks it can/want, > but I can't find it. > Wishful thinking? Perhaps. There is a sendfile() call in most *nixes at this point. Depending on the *nix that goes back to the mid-late '90s or so. Not sure if Posix or anyone decided to standardize it. rick jones -- No need to believe in either side, or any side. There is no cause. There's only yourself. The belief is in your own precision. - Joubert these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
From: Benny Amorsen on 28 Jul 2010 04:51
Terje Mathisen <"terje.mathisen at tmsw.no"> writes: > I thought there was a standard unix api called something like > CopyFile() which allows the OS to employ all the tricks it can/want, > but I can't find it. Apart from sendfile(), Linux supports splice() which is really quite brilliant. However, it is completely dependent on hardware support because it does not add any buffering at all. This means that if you try to receive from the network into a file, splice() will do the disk I/O of the chunks received from the network, typically less than 1500 bytes. Smarter NICs can do receive offload and assemble up to 64kB which is somewhat easier for disks to handle. /Benny |