Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 29 Mar 2010 11:31 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:lm00r5ddh8n84jmmd65o6n7do5crl4tenq(a)4ax.com... > Ee below... > On Thu, 25 Mar 2010 14:19:58 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >>Right and depending on what is being accomplished one >>might >>discard non crucial integrity for speed. If the sequence >>of >>steps can be easily reconstructed because the request is >>definitely committed to disk, then making sure that >>intervening steps are also committed is not as crucial. > **** > And how is this consistent with your new magical > buzzphrase "fault tolerant"? Either you > want reliability, or you don't. False dilemma fallacy. I want a specific degree of fault tolerance, no more and no less. > If you want fault tolerance, you are going to have to hit > the disk; At least to record incoming transactions, this disk hit can not possibly be avoided and have minimal fault tolerance maintained. This disk hit must be a fully committed write operation.
From: Hector Santos on 29 Mar 2010 11:31 Peter Olcott wrote: > The other Pete had greater insights into my own needs than I > did myself. I will paraphrase what he said. I only need to > avoid losing transactions. When a client makes a request, I > only need to avoid losing this request until it is > completed. Any faults in-between can be restarted from the > beginning. > > The key (not all the details, just the essential basis for > making it work) to providing this level of fault tolerance > is to have the webserver only acknowledge web requests after > the web request have been committed to persistent storage. > > The only remaining essential element (not every little > detail just the essence) is providing a way to keep track of > web requests to make sure that they make it to completed > status in a reasonable amount of time. A timeout threshold > and a generated exception report can provide feedback here. > > Please make any responses to the above statement within the > context of the newly defined much narrower scope of fault > tolerance. //---------------------------------------------- // Prepare Local Request //---------------------------------------------- TRequest req(); //---------------------------------------------- // Record request, user, other data //---------------------------------------------- req.TimePosted = Now(); req.Url = GetUrl(); req.User = GetUserAccount(); req.OtherStuff = GetOtherStuff(); //---------------------------------------------- // Save/Flush to Disk //---------------------------------------------- req.Save(); //---------------------------------------------- // Process it //---------------------------------------------- Req.Process(); //---------------------------------------------- // Send result //---------------------------------------------- Req.SendOutput(); //---------------------------------------------- // Stamp Completion and Update //---------------------------------------------- req.TimeCompleted = Now(); req.Update(); Which part of this *SIMPLE, COMPLETE, MODELING 101 OUTLINE* procedure don't you understand? -- HLS
From: Peter Olcott on 29 Mar 2010 11:33 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:g710r593g0pnhaul2jkdc0rbvmosaqdj8e(a)4ax.com... > See below... > On Thu, 25 Mar 2010 09:35:19 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >>> EnqueueRequest() which takes a description of the >>> request. >>> And is implemented on each >>> platform with platform-specific code that is optimized >>> for >>> that platform. Your concept of >>> "Portable" is incorrect here. you want the FASTEST way >>> that the platform supports, and >>> that may be completely different from every other >>> platform. But if you code has a single >>> abstraction, you are free to implement it in the best >>> way >>> on each platform! >> >>Is this supported on every platform? > *** > That's my entire point: you determine, for every platform, > what 'this' is, and you WRITE > CODE TO SUPPORT IT! In the abstract model, you have a > system, perhaps even MySQL, but it > is considered a low-level transport mechanism completely > invisible to you your application > architecture! You have two abstraction: "Enqueue" and > "Dequeue" and that's what you write > your application in terms of. On some systems, if you > don't care in the slightest about > performance, you might implment them as MySQL calls. In > systems that have > high-performance interprocess communication, you implement > a platform-specific mechanism > that is blindingly fast. Or have you missed the > fundamental concepts of "abstraction"? > joe > ***** I would prefer to start with something that already works if there already is such a thing.
From: Peter Olcott on 29 Mar 2010 11:47 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:q230r5hp161iibaml6pif093en5dti0t2r(a)4ax.com... > See below... > On Thu, 25 Mar 2010 10:35:27 -0400, Hector Santos > <sant9442(a)nospam.gmail.com> wrote: > >>Peter Olcott wrote: >> >>Yes, the world has already put aside this Kindergarden >>concept! and NO >>ONE IN THE WORLD uses SQL for a FIFO! If they do, they >>are PRIMITIVE. > **** > Banking software uses features in products like ORACLE to > guarantee transacted access to > the database of pending transactions, and the audit trail > that follows these transactions. > So while they aren't always using SQL, they are using > transacted databases. Slow as hell, > of course, but utterly robust, because the transactions > either get completed or rolled > back, and the multiple servers that work on the queue can > handle the fact that if server 1 > removes the request, and then crashes, the transaction > that removed the request is rolled > back and the pending transaction reappears in the > database. But this requirement is > incosnsitent with the requirement that there be no disk > accesses. But we expect > consistent requirements? SIlly us! > **** I will need fault tolerance that is as reliable as bank fault tolerance. The specific transaction will be that every client request will cost a dime. This dime will be deducted from the customer's account balance. I can expect at most 100 transactions per second because this is the maximum processing load of the OCR software on a single server when processing requests are of minimal size.
From: Peter Olcott on 29 Mar 2010 11:58
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:4n30r5p0lnqjf7u5tbfodjkjnp9kt05vul(a)4ax.com... > See below... > On Thu, 25 Mar 2010 14:32:19 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >>news:ORrRkLEzKHA.928(a)TK2MSFTNGP05.phx.gbl... >>> >>> >>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >>> news:EqmdnZkWypVi7DbWnZ2dnUVZ_rednZ2d(a)giganews.com... >>>> (By fault tolerance I mean yank the power plug from the >>>> wall and when the machine is re-started it (as much as >>>> possible) picks up right where it left off) >>> >>> ...take a look at transactional NTFS. >>> >>> http://msdn.microsoft.com/en-us/library/aa365738(VS.85).aspx >>> >>> -Pete >>> >> >>Which I bet requires some sort of persistent storage, yup >>it >>does. >>How could I have very fast inter process communication >>that >>is also fault tolerant, or are these two mutually >>exclusive? > **** > Why do you suddenly add "fault tolerance", then add "fast > interprocess communication that > is also fault tolerant"? I think you are totally and > utterly clueless here. You do not > understand ANY of the concepts involved. > > If you have fault tolerance, why does it have to be in the > IPC mechanism? In fact, it > would probably be a Really Bad Idea to try that. Although it may be temporary ignorance on my part that suggested such a thing, I was thinking that it might be simpler to do it this way because every client request will be associated with a financial transaction. Each financial transaction depends upon its corresponding client request and each client request depends upon its corresponding financial transaction. With such mutual dependency it only seemed natural for the underlying representation to be singular. > I guess I'm biased again, having built > several fault-tolerant systems. > joe > **** >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |