IPC based on name pipe FIFO and transaction log file
in [Unix Programming]
|
Prev: LBW 0.1: Linux Binaries on Windows
Next: socket
From: Peter Olcott on 1 Apr 2010 12:09 Please critique this design: Multiple threads of one process communicate with one or more threads of another process. The first process appends (O_APPEND flag) transaction records to a transaction log file, and then writes to a named pipe to inform the other process that a transaction is ready for processing. The transaction log file contains all of the details of the transaction as fixed length binary records. Any reads of this file use pread(). The second process receives this message and begins processing. There are three status codes that are written to the transaction log file: 0=New (the first process writes this one using pwrite()) 1=Being Processed (the second process writes this one using pwrite()) 2=Processing is Completed (the second process writes this one using pwrite()) As soon as the second process begins processing it marks this transaction as [Being Processed], when processing is completed it marks this transaction as [Processing is Completed] and notifies the first process using another named pipe.
From: Peter Olcott on 1 Apr 2010 12:18 "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message news:XuGdndDQb8WpWCnWnZ2dnUVZ_v2dnZ2d(a)giganews.com... > Please critique this design: > > Multiple threads of one process communicate with one or > more threads of another process. > > The first process appends (O_APPEND flag) transaction > records to a transaction log file, and then writes to a > named pipe to inform the other process that a transaction > is ready for processing. The transaction log file contains > all of the details of the transaction as fixed length > binary records. Any reads of this file use pread(). > > The second process receives this message and begins > processing. There are three status codes that are written > to the transaction log file: > 0=New (the first process writes this one using pwrite()) Whoops no pwrite() needed here it can be part of the append. > > 1=Being Processed (the second process writes this one > using pwrite()) > > 2=Processing is Completed (the second process writes this > one using pwrite()) > > As soon as the second process begins processing it marks > this transaction as [Being Processed], when processing is > completed it marks this transaction as [Processing is > Completed] and notifies the first process using another > named pipe. >
From: David Schwartz on 1 Apr 2010 18:40 On Apr 1, 9:09 am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > The first process appends (O_APPEND flag) transaction > records to a transaction log file, and then writes to a > named pipe to inform the other process that a transaction is > ready for processing. The transaction log file contains all > of the details of the transaction as fixed length binary > records. Any reads of this file use pread(). Appends are not guaranteed atomic. So each writer would have to have its own transaction log file or you'd need some separate mechanism to lock them. > The second process receives this message and begins > processing. There are three status codes that are written to > the transaction log file: > 0=New (the first process writes this one using pwrite()) > > 1=Being Processed (the second process writes this one using > pwrite()) > > 2=Processing is Completed (the second process writes this > one using pwrite()) > > As soon as the second process begins processing it marks > this transaction as [Being Processed], when processing is > completed it marks this transaction as [Processing is > Completed] and notifies the first process using another > named pipe. How do the threads arbitrate control over the named pipe? How are responses on the named pipe dispatches to the appropriate thread? Or are you suggesting there be one transaction log file and one named pipe for each possible thread-to-thread set? If so, how will they be established in the first place? It's hard to analyze a solution without knowing what problem it's supposed to solve. ;) DS
From: Jens Thoms Toerring on 1 Apr 2010 18:52 [ A bit excessive crossposting removed ] In comp.unix.programmer Peter Olcott <NoSpam(a)ocr4screen.com> wrote: > Please critique this design: > Multiple threads of one process communicate with one or more > threads of another process. > The first process appends (O_APPEND flag) transaction > records to a transaction log file, and then writes to a > named pipe to inform the other process that a transaction is > ready for processing. The transaction log file contains all > of the details of the transaction as fixed length binary > records. Any reads of this file use pread(). > The second process receives this message and begins > processing. There are three status codes that are written to > the transaction log file: > 0=New (the first process writes this one using pwrite()) > 1=Being Processed (the second process writes this one using > pwrite()) > 2=Processing is Completed (the second process writes this > one using pwrite()) > As soon as the second process begins processing it marks > this transaction as [Being Processed], when processing is > completed it marks this transaction as [Processing is > Completed] and notifies the first process using another > named pipe. The problem with your question seems to me that you just de- scribe how you plan to do something but not what you actually want to achieve. So your design might already be the best possible one for your given requirements but without knowing the specific requirements it's a bit hard to be sure. Perhaps you could also be a it more detailed about things like first writing about several threads in each process but in the following only talking about processes. Are there different threads in the first process, all writing to the file and to the pipe and several threads in the second process, also all at once trying to read on the file and the pipe, or is there only a single thread in each process doing that? I guess that having several threads being involved could make quite a bit of a difference. I am also a bit confused why you need a file beside the pipes. Your description could be understand as meaning that you have one proceess that produces some data that have to be acted on by a second process and that the first process needs some ack- nowlegment when the second process starts working and another ther one when it done. If that's the case wouldn't then having the first process put the data into the pipe and the second process sending maybe just a byte when it has read them and is starting to work on them and another one when it's finished be enough (i.e. no file needed at all)? Regards, Jens -- \ Jens Thoms Toerring ___ jt(a)toerring.de \__________________________ http://toerring.de
From: Peter Olcott on 1 Apr 2010 19:23
"David Schwartz" <davids(a)webmaster.com> wrote in message news:f1655750-7bea-4f12-b8f4-bef1f594ec15(a)r1g2000yqj.googlegroups.com... On Apr 1, 9:09 am, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > The first process appends (O_APPEND flag) transaction > records to a transaction log file, and then writes to a > named pipe to inform the other process that a transaction > is > ready for processing. The transaction log file contains > all > of the details of the transaction as fixed length binary > records. Any reads of this file use pread(). --Appends are not guaranteed atomic. So each writer would have to have --its own transaction log file or you'd need some separate mechanism to --lock them. You may be correct, but, if you are then two different editions of Advanced Programming in the Unix Environment would be incorrect: First Edition Chapter 3 Section 3.11 Atomic Operations page 60-61 Appending to a File "Unix provides an atomic way to do this operation if we set the O_APPEND flag when a file is opened." Second Edition Chapter 3 Section 3.11 Atomic Operations page 74 Appending to a File "Unix provides an atomic way to do this operation if we set the O_APPEND flag when a file is opened." > The second process receives this message and begins > processing. There are three status codes that are written > to > the transaction log file: > 0=New (the first process writes this one using pwrite()) > > 1=Being Processed (the second process writes this one > using > pwrite()) > > 2=Processing is Completed (the second process writes this > one using pwrite()) > > As soon as the second process begins processing it marks > this transaction as [Being Processed], when processing is > completed it marks this transaction as [Processing is > Completed] and notifies the first process using another > named pipe. --How do the threads arbitrate control over the named pipe? How are --responses on the named pipe dispatches to the appropriate thread? The last time that I did this (before Unix had threads) it was not an issue. --Or are you suggesting there be one transaction log file and one named Yes one single transaction log file. --pipe for each possible thread-to-thread set? If so, how will they be --established in the first place? Two total pipes, one in each direction. --It's hard to analyze a solution without knowing what problem it's --supposed to solve. ;) DS I am trying to convert my proprietary OCR software into a web application. Initially there will be multiple threads, one for each web request, and a single threaded process servicing these web requests. Eventually there may be multiple threads servicing these web requests. |