IPC based on name pipe FIFO and transaction log file
in [Unix Programming]
|
Prev: LBW 0.1: Linux Binaries on Windows
Next: socket
From: Scott Lurndal on 2 Apr 2010 15:25 "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: > >"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message >> In your application, I'd frankly avoid file operations in >> favor of queues or ring-buffers in a MAP_SHARED mmap(2) >> region. If you need the >> queues to be persistent, map a file; otherwise map >> anonymous (linux) or > >This may not be flushed to disk often enough to meet my >needs. It seems that append can at least be forced to flush >to disk immediately. Although forcing it to flush to disk >may be very inefficient, I am estimating that it won't cost >much if there are very few bytes being written, far less >than 512 bytes. If you don't set O_SYNC or O_DSYNC when you open your file, the data will _not_ be flushed to disk immediately. It may be delayed by a considerable period unless you call fsync(2) or fdatasync(2) to force the flush. Note that 'fflush' does _not_ require the data to be flushed to disk, just from the user-mode buffers in libc into kernel-mode buffers in the file cache. With mmap, you can explictly call 'msync(2)' on a specific address range to flush that range to the backing device if required. scott
From: Scott Lurndal on 2 Apr 2010 15:30 David W Noon <dwnoon(a)spamtrap.ntlworld.com> writes: >LS0tLS1CRUdJTiBQR1AgU0lHTkVEIE1FU1NBR0UtLS0tLQ0KSGFzaDogU0hBMQ0KDQpPbiBGcmks >IDIgQXByIDIwMTAgMTc6MzY6NTIgKzAyMDAsIEVyc2VrLCBMYXN6bG8gd3JvdGUgYWJvdXQgUmU6 >IElQQw0KYmFzZWQgb24gbmFtZSBwaXBlIEZJRk8gYW5kIHRyYW5zYWN0aW9uIGxvZyBmaWxlOg0K >DQo+T24gRnJpLCAyIEFwciAyMDEwLCBEYXZpZCBXIE5vb24gd3JvdGU6DQpbc25pcF0NCj4+IFZp >dHVzIHdhcyByZWZlcnJpbmcgdG8gT1MvMiwgYW5kIFBGIGlzIFBldGVyIEZpdHpzaW1tb25zLiAg >UGV0ZXIgd2FzDQo+PiBhIHdlbGwga25vd24gZGV2ZWxvcGVyIG9uIE9TLzIgc29tZSAxNSBvciAy >MCB5ZWFycyBhZ28uDQo+DQo+VGhhbmsgeW91IGZvciB0aGlzIGNsYXJpZmljYXRpb24uIEJyaWVm >bHkgc2VhcmNoaW5nIGZvciAnUGV0ZXINCj5GaXR6c2ltbW9ucyBPUy8yIHRocmVhZHMnIGRpZG4n >dCByZXR1cm4gYW55dGhpbmcgZGVmaW5pdGl2ZSwNCg0KUGV0ZXIgd2FzIHByaW1hcmlseSBrbm93 >biBvbiBGaWRvbmV0LiAgSGUgd2FzIHRoZSBtb2RlcmF0b3Igb2YgdGhlDQpPUzJQUk9HIGVjaG8g >KHRoZSBGaWRvbmV0IGFuYWxvZ3VlIG9mIGEgbmV3c2dyb3VwKSB1bnRpbCB0aGUgbGF0ZQ0KMTk5 >MCdzLiAgSSBzdWNjZWVkZWQgaGltIGFzIG1vZGVyYXRvciB3aGVuIGhlIGxvc3QgaGlzIEZpZG9u >ZXQNCmNvbm5lY3Rpb24uDQoNCj5UaGUgbm90aW9uIG9mIGEgdGhyZWFkIGFuZCBhIHRocmVhZCdz >IGRlc2lnbmVkLWluIGNoYXJhY3RlcmlzdGljcyBpbg0KPk9TLzIgc2VlbSBmdW5kYW1lbnRhbGx5 >IGRpZmZlcmVudCBmcm9tIHRob3NlIGluIFVOSVgoUiksIHdoZXJlDQo+ImV2ZXJ5dGhpbmcgaXMg >YSBmaWxlIi4gVGhlcmVmb3JlIEkgYXNzdW1lIHByb2dyYW1taW5nIHRlY2huaXF1ZXMNCj5kZXJp >dmVkIGZyb20gdGhlIE9TLzIgdGhyZWFkIGNvbmNlcHQgY2Fubm90IGJlIGVhc2lseSBhcHBsaWVk >IG9uDQo+bW9kZXJuIFBPU0lYKFIpIHN5c3RlbXMuDQoNCkZvciB0aGUgbW9zdCBwYXJ0LCB0aGUg >YXBwbGljYXRpb24gcHJvZ3JhbW1pbmcgdGVjaG5pcXVlcyBhcmUgY2xvc2VseQ0KY29tcGFyYWJs >ZSwgZXZlbiB0aG91Z2ggdGhlIHVuZGVybHlpbmcgdGhyZWFkIGltcGxlbWVudGF0aW9ucyBjYW4g >YmUNCnF1aXRlIGRpZmZlcmVudC4gIEV2ZW4gdW5kZXIgVU5JWCwgYSB0aHJlYWQgdGhhdCBzaW1w >bHkgd2FpdHMgYSBsb3QgZm9yDQpzb21lIGJhY2tncm91bmQgcHJvY2Vzc2luZyAtLSB0eXBpY2Fs >bHkgZGlzayBvciBuZXR3b3JrIEkvTyAtLSB0bw0KY29tcGxldGUgd29ya3MgYXMgd2VsbCBhcyBp >dCBkb2VzIHVuZGVyIE9TLzIuDQotIC0tIA0KUmVnYXJkcywNCg0KRGF2ZSAgW1JMVSAjMzE0NDY1 >XQ0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 >PT09PT09PT09PT09PT09PT0NCmR3bm9vbkBzcGFtdHJhcC5udGx3b3JsZC5jb20gKERhdmlkIFcg >Tm9vbikNClJlbW92ZSBzcGFtIHRyYXAgdG8gcmVwbHkgYnkgZS1tYWlsLg0KPT09PT09PT09PT09 >PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 >PT0NCi0tLS0tQkVHSU4gUEdQIFNJR05BVFVSRS0tLS0tDQpWZXJzaW9uOiBHbnVQRyB2Mi4wLjE0 >IChHTlUvTGludXgpDQoNCmlFWUVBUkVDQUFZRkFrdTJLWnNBQ2drUTlNcWFVSlF3Mk1tbmVRQ2Zm >Z1pycElIZFlKL2syd3lUY0pnK2F4dTINClFPQUFvSVE0M1hnL2tnQThuL2M5SVUrYmhwc0NNSjQ1 >DQo9RHdJWg0KLS0tLS1FTkQgUEdQIFNJR05BVFVSRS0tLS0tDQo= > base64 encoded messages are frowned upon, being not supported by many older newsreaders. scott
From: Peter Olcott on 2 Apr 2010 15:49 "Joe Beanfish" <joe(a)nospam.duh> wrote in message news:hp5fph$sec(a)news.thunderstone.com... > On 04/01/10 19:23, Peter Olcott wrote: >> I am trying to convert my proprietary OCR software into a >> web application. Initially there will be multiple >> threads, >> one for each web request, and a single threaded process >> servicing these web requests. Eventually there may be >> multiple threads servicing these web requests. > > I'd use a database to maintain the queue. Sometimes you > can > use the filesystem to accomplish database like operations. > One > file per record. Separate directories for pending and > completed > jobs. Mail systems often do that. One file for the mail > msg, one > for the headers, and maybe another for status info. > > If using the filesystem as a database use "mv" to > accomplish > atomic operations: > write to tmpfile.pid > mv tmpfile.pid readytogo.img > queue reader looks for *.img I was going to use a single file with binary data and fixed length records to keep track of all of the web requests. I also proposed named pipes as the means of notification of new web requests, and completed requests.
From: Peter Olcott on 2 Apr 2010 15:50 "Scott Lurndal" <scott(a)slp53.sl.home> wrote in message news:Bwrtn.1$rJ2.0(a)news.usenetserver.com... > "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: >> >>"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message > >>> In your application, I'd frankly avoid file operations >>> in >>> favor of queues or ring-buffers in a MAP_SHARED mmap(2) >>> region. If you need the >>> queues to be persistent, map a file; otherwise map >>> anonymous (linux) or >> >>This may not be flushed to disk often enough to meet my >>needs. It seems that append can at least be forced to >>flush >>to disk immediately. Although forcing it to flush to disk >>may be very inefficient, I am estimating that it won't >>cost >>much if there are very few bytes being written, far less >>than 512 bytes. > > If you don't set O_SYNC or O_DSYNC when you open your > file, > the data will _not_ be flushed to disk immediately. It may > be delayed by a considerable period unless you call > fsync(2) or fdatasync(2) to force the flush. Note that > 'fflush' > does _not_ require the data to be flushed to disk, just > from > the user-mode buffers in libc into kernel-mode buffers in > the > file cache. > > With mmap, you can explictly call 'msync(2)' on a specific > address range to flush that range to the backing device if > required. > > > scott Someone else also brought up the possible issue on flushing the disk's own onboard buffer. Is this really a problem?
From: Ersek, Laszlo on 2 Apr 2010 16:34
On Fri, 2 Apr 2010, Peter Olcott wrote: > "Scott Lurndal" <scott(a)slp53.sl.home> wrote in message > news:Bwrtn.1$rJ2.0(a)news.usenetserver.com... >> With mmap, you can explictly call 'msync(2)' on a specific address >> range to flush that range to the backing device if required. > Someone else also brought up the possible issue on flushing the disk's > own onboard buffer. Is this really a problem? The Linux manual pages for close(2) and fsync(2) allude to this. http://www.kernel.org/doc/man-pages/online/pages/man2/close.2.html http://www.kernel.org/doc/man-pages/online/pages/man2/fsync.2.html I think your server will run on an UPS that will be able to notify the kernel (via a serial port or so) to shut down cleanly if power is failing, so I wouldn't worry about the disk hardware. If you do wish to protect against hardware failures, that's a different weight class. http://sqlite.org/atomiccommit.html ----v---- 2.0 Hardware Assumptions [...] SQLite does not assume that a sector write is atomic. However, it does assume that a sector write is linear. By "linear" we mean that SQLite assumes that when writing a sector, the hardware begins at one end of the data and writes byte by byte until it gets to the other end. The write might go from beginning to end or from end to beginning. If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part was left unchanged. The key assumption by SQLite is that if any part of the sector gets changed, then either the first or the last bytes will be changed. So the hardware will never start writing a sector in the middle and work towards the ends. We do not know if this assumption is always true but it seems reasonable. [...] ----^---- I wouldn't even think of reimplementing this. People have dedicated their lives to research it. I didn't re-read the linked-to page now, but since SQLite is used by many applications run by non-privileged users, I doubt SQLite tries to access any hardware directly. "SQLite is Transactional" nonetheless <http://sqlite.org/transactional.html>, so I'd assume you don't need hardware sync either. .... What about using SQLite for safe job storage, and using the other mechanisms only for notification, so you don't have to poll? <http://www.sqlite.org/threadsafe.html> I apologize if this has already been discussed. lacos |