IPC based on name pipe FIFO and transaction log file
in [Unix Programming]
|
Prev: LBW 0.1: Linux Binaries on Windows
Next: socket
From: Ersek, Laszlo on 2 Apr 2010 12:18 On Fri, 2 Apr 2010, Vitus Jensen wrote: > On Fri, 2 Apr 2010, Ersek, Laszlo wrote: > >> On Fri, 2 Apr 2010, Vitus Jensen wrote: >> >>> I'm coming from another platform where the maxime was "a thread should >>> do nothing very well" (PF) >> What's the name of that platform? I don't understand the "PF" >> abbreviation. > That's "OS/2", Peter Fitzsimmons (PF) first axiom of threads was "A > thread's most important job is to do nothing very well." As posted on > fidonet OS2PROG, which unfortunately google doesn't search. Happy to be of service: http://fidonet.ozzmosis.com/echomail.php/os2prog/e57e63082ae7ba40.html ----v---- From: Peter Fitzsimmons To: David A. Beisel Date: 1996-08-17 05:33:16 Subject: Semaphore [...] Pete's first thread axiom: A thread's most important job is to do nothing very well. ----^---- http://fidonet.ozzmosis.com/echomail.php/os2prog/107b597bba891cbf.html ----v---- From: Peter Fitzsimmons To: Alexander Kuiper Date: 1996-08-27 03:15:52 Subject: OS/2 Executable system load ?? [...] You may have seen me state my first axiom of threads a few days ago: A thread's most important job is to do nothing very well. I hope I have shed some light on it with this message. [...] ----^---- Yes, google doesn't seem to find it, even when restricting the search to site:fidonet.ozzmosis.com and searching for the verbatim citation. I downloaded the entire archive and grepped it. This is not prohibited: $ hexdump -C fidonet.ozzmosis.com/robots.txt 00000000 55 73 65 72 2d 61 67 65 6e 74 3a 20 2a 0a 44 69 |User-agent: *.Di| 00000010 73 61 6c 6c 6f 77 3a 0a |sallow:.| http://www.robotstxt.org/orig.html ----v---- Disallow [...] Any empty value, indicates that all URLs can be retrieved. [...] ----^---- This was very interesting, thanks. lacos
From: Peter Olcott on 2 Apr 2010 13:14 "Scott Lurndal" <scott(a)slp53.sl.home> wrote in message news:TEotn.1684$MR5.1603(a)news.usenetserver.com... > "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: >> >>"Scott Lurndal" <scott(a)slp53.sl.home> wrote in message >>news:nBatn.1653$OC1.680(a)news.usenetserver.com... >>> David Schwartz <davids(a)webmaster.com> writes: >>>>On Apr 1, 9:09=A0am, "Peter Olcott" >>>><NoS...(a)OCR4Screen.com> wrote: >>>> >>>>> The first process appends (O_APPEND flag) transaction >>>>> records to a transaction log file, and then writes to >>>>> a >>>>> named pipe to inform the other process that a >>>>> transaction is >>>>> ready for processing. The transaction log file >>>>> contains >>>>> all >>>>> of the details of the transaction as fixed length >>>>> binary >>>>> records. Any reads of this file use pread(). >>>> >>>>Appends are not guaranteed atomic. So each writer would >>>>have to have >>>>its own transaction log file or you'd need some separate >>>>mechanism to >>>>lock them. >>> >>> A single write or pwrite call on a file with O_APPEND >>> is required by the SUS to ensure that the write is >>> performed >>> atomically with respect to other writes to the same file >>> which also have >>> the O_APPEND flag set. The order, of course, is not >>> guaranteed. >>> >>> scott >> >>I am taking the above to mean that David was incorrect >>when >>he said: >> >> Appends are not guaranteed atomic. So each writer >> would have to have its own transaction log file or >> you'd >> need some separate mechanism to lock them. >> >>If this is not what was intended please clarify. >> >> > > Read David's response. If two writers are appending to > a single file, each individual write is appended > atomically > to the end of the file. The order of the records when > more > than one process writes to the file is non-deterministic. > > In your application, I'd frankly avoid file operations in > favor of queues or ring-buffers in a MAP_SHARED mmap(2) > region. If you need the > queues to be persistent, map a file; otherwise map > anonymous (linux) or This may not be flushed to disk often enough to meet my needs. It seems that append can at least be forced to flush to disk immediately. Although forcing it to flush to disk may be very inefficient, I am estimating that it won't cost much if there are very few bytes being written, far less than 512 bytes. > shmat (unix). Use pthread_mutex (attr=PROCESS_SHARED), > semop, or > GCC built-in atomic memory access functions (e.g. > __sync_fetch_and_add) > for mutual exclusion/wakup. > > scott
From: David W Noon on 2 Apr 2010 13:29 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 2 Apr 2010 17:36:52 +0200, Ersek, Laszlo wrote about Re: IPC based on name pipe FIFO and transaction log file: >On Fri, 2 Apr 2010, David W Noon wrote: [snip] >> Vitus was referring to OS/2, and PF is Peter Fitzsimmons. Peter was >> a well known developer on OS/2 some 15 or 20 years ago. > >Thank you for this clarification. Briefly searching for 'Peter >Fitzsimmons OS/2 threads' didn't return anything definitive, Peter was primarily known on Fidonet. He was the moderator of the OS2PROG echo (the Fidonet analogue of a newsgroup) until the late 1990's. I succeeded him as moderator when he lost his Fidonet connection. >The notion of a thread and a thread's designed-in characteristics in >OS/2 seem fundamentally different from those in UNIX(R), where >"everything is a file". Therefore I assume programming techniques >derived from the OS/2 thread concept cannot be easily applied on >modern POSIX(R) systems. For the most part, the application programming techniques are closely comparable, even though the underlying thread implementations can be quite different. Even under UNIX, a thread that simply waits a lot for some background processing -- typically disk or network I/O -- to complete works as well as it does under OS/2. - -- Regards, Dave [RLU #314465] ======================================================================= dwnoon(a)spamtrap.ntlworld.com (David W Noon) Remove spam trap to reply by e-mail. ======================================================================= -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEARECAAYFAku2KZsACgkQ9MqaUJQw2MmneQCffgZrpIHdYJ/k2wyTcJg+axu2 QOAAoIQ43Xg/kgA8n/c9IU+bhpsCMJ45 =DwIZ -----END PGP SIGNATURE-----
From: Ersek, Laszlo on 2 Apr 2010 14:58 On Fri, 2 Apr 2010, Peter Olcott wrote: > "Scott Lurndal" <scott(a)slp53.sl.home> wrote in message > news:TEotn.1684$MR5.1603(a)news.usenetserver.com... >> In your application, I'd frankly avoid file operations in favor of >> queues or ring-buffers in a MAP_SHARED mmap(2) region. If you need the >> queues to be persistent, map a file > This may not be flushed to disk often enough to meet my needs. It seems > that append can at least be forced to flush to disk immediately. msync(addr, len, MS_SYNC)? --o-- I think bitwise or-ing MS_INVALIDATE is not necessary, unless you also wish to pick up changes made to the file through write(), in the specified offset range. http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2008-06/msg00341.html The SUSv[34] say in the msync() rationale, "After the data is written out, any cached data may be invalidated if the MS_INVALIDATE flag was specified. This is useful on systems that do not support read/write consistency". I'm curious if "read/write consistency" means "write()'s effect is immediately visible in memory mappings". Would anybody please validate the following table? +-------------+----------------------------------------------------------------+ | change made | change visible via | | through +----------------------------+-------------+---------------------+ | | MAP_SHARED | MAP_PRIVATE | read() | +-------------+----------------------------+-------------+---------------------+ | MAP_SHARED | yes | unspecified | depends on MS_SYNC, | | | | | MS_ASYNC, or normal | | | | | system activity | +-------------+----------------------------+-------------+---------------------+ | MAP_PRIVATE | no | no | no | +-------------+----------------------------+-------------+---------------------+ | write() | depends on MS_INVALIDATE, | unspecified | yes | | | or the system's read/write | | | | | consistency | | | +-------------+----------------------------+-------------+---------------------+ Thanks, lacos
From: Joe Beanfish on 2 Apr 2010 15:16
On 04/01/10 19:23, Peter Olcott wrote: > I am trying to convert my proprietary OCR software into a > web application. Initially there will be multiple threads, > one for each web request, and a single threaded process > servicing these web requests. Eventually there may be > multiple threads servicing these web requests. I'd use a database to maintain the queue. Sometimes you can use the filesystem to accomplish database like operations. One file per record. Separate directories for pending and completed jobs. Mail systems often do that. One file for the mail msg, one for the headers, and maybe another for status info. If using the filesystem as a database use "mv" to accomplish atomic operations: write to tmpfile.pid mv tmpfile.pid readytogo.img queue reader looks for *.img |