Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 28 Mar 2010 22:24 See below... On Thu, 25 Mar 2010 14:32:19 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >news:ORrRkLEzKHA.928(a)TK2MSFTNGP05.phx.gbl... >> >> >> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >> news:EqmdnZkWypVi7DbWnZ2dnUVZ_rednZ2d(a)giganews.com... >>> (By fault tolerance I mean yank the power plug from the >>> wall and when the machine is re-started it (as much as >>> possible) picks up right where it left off) >> >> ...take a look at transactional NTFS. >> >> http://msdn.microsoft.com/en-us/library/aa365738(VS.85).aspx >> >> -Pete >> > >Which I bet requires some sort of persistent storage, yup it >does. >How could I have very fast inter process communication that >is also fault tolerant, or are these two mutually exclusive? **** Why do you suddenly add "fault tolerance", then add "fast interprocess communication that is also fault tolerant"? I think you are totally and utterly clueless here. You do not understand ANY of the concepts involved. If you have fault tolerance, why does it have to be in the IPC mechanism? In fact, it would probably be a Really Bad Idea to try that. I guess I'm biased again, having built several fault-tolerant systems. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:37 See below... On Thu, 25 Mar 2010 15:57:04 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >news:%23Qq40gFzKHA.928(a)TK2MSFTNGP05.phx.gbl... >> >> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >> news:E46dnUbV0LzZJzbWnZ2dnUVZ_judnZ2d(a)giganews.com... >>> >>> "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in >>> message news:ORrRkLEzKHA.928(a)TK2MSFTNGP05.phx.gbl... >>>> >>>> >>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >>>> news:EqmdnZkWypVi7DbWnZ2dnUVZ_rednZ2d(a)giganews.com... >>>>> (By fault tolerance I mean yank the power plug from the >>>>> wall and when the machine is re-started it (as much as >>>>> possible) picks up right where it left off) >>>> >>>> ...take a look at transactional NTFS. >>>> >>>> http://msdn.microsoft.com/en-us/library/aa365738(VS.85).aspx >>>> >>>> -Pete >>>> >>> >>> Which I bet requires some sort of persistent storage, yup >>> it does. >> >> Peter, >> You did not, in this particular portion of a very >> convoluted thread, mention that you wanted fault tolerance >> without persistant storage. I apologize if I have read >> your mind incorrectly, but you simply asked for fault >> tolerance and I gave you a method that you could actually >> use with *very little additional knowledge required*! >> Additionally, I believe that your interpretation of >> "fault tolerance" is that a catastrophic event could >> happen to your system and you application would not lose >> *any* data. Is this the definition that you are using? >> >> >> -Pete >I am looking for fault tolerance implemented entirely in >software, I can not afford much hardware fault tolerance at >the current time. The most hardware fault tolerance that I >will have, initially is RAID 1 on a single core single >server. **** Fault tolerance is almost always in software these days. One site I was at had replaced their $1,000,000 "Nonstop" systems (hardware fault tolerance) with an array of about $10,000 of Windows boxes. I worked with them on dealing with fault tolerance in their device drivers. They had other people working on higher levels. RAID1 isn't really fault tolerant; you need at least RAID5 to get true fault tolerance. **** > >I think that I figured out how to implement the sort of >fault tolerance that I need. The essence is that a request >is not acknowledged as fully accepted until after it is >committed to disk in the transaction log. Every fault after >that can recover using this basis. **** OK, here's a scenario, and you had better handle it: you process the request, and you have asked PayPal to do the deduction, and you fail to get the response back to the end user. In this case, you have to "roll back" the transaction, and credit the account. This is a MINIMUM requirement. I've had to deal with real people's real money in banking and payroll systems, and trust me, this stuff REALLY matters! **** > >I am not sure of the underlying mechanisms, but, most every >database system can handle commits and rollbacks of >transactions. I am not sure exactly what happens when >someone actually pulls the plug during a database update. >Certainly some of the data is garbled and must be repaired. >I don't really know the exact extent that this repair can be >fully automated. **** At a performance cost. This is why transaction processing is inconsistent with your silly requirement that nobody, ever, accesses the disk during your processing. You can use MySQL as a FIFO, but it is going to be EXPENSIVE in terms of performance. REALLY EXPENSIVE. But you clearly no longer care about this subject, in spite of your flogging us to death over how a single page fault was unacceptable. You replace this with a requirement that transactions be committed (each of which is more expensive than dozens of page faults), and think this operation is free. I wonder if you have some really, really interesting stuff you ingest/smoke/drink that allows you to put up such a powerful reality distortion field. It is hard to believe that you can accomplish this without chemical assistance. I have been on some really serious drugs post-surgery (and I have three linear feet of scar tissue on my body to prove how many surgeries I've had) and I think under the most powerful of them I would not be able to create such inconsistent requirements. I might not care about them at all under such conditions, but I think I could recognize that they were inconsistent. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:43 See below... On Fri, 26 Mar 2010 11:01:54 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Mikel" <mikel.luri(a)gmail.com> wrote in message >news:36cb8b85-0cf8-421c-94ce-dd1fd77c3f36(a)e7g2000yqf.googlegroups.com... >> On 26 mar, 15:32, "Peter Olcott" <NoS...(a)OCR4Screen.com> >> wrote: >>> "Pete Delgado" <Peter.Delg...(a)NoSpam.com> wrote in >>> message >>> >>> news:eSOGleKzKHA.264(a)TK2MSFTNGP05.phx.gbl... >>> >>> >>> >>> >>> >>> >>> >>> > "Liviu" <lab...(a)gmail.c0m> wrote in message >>> >news:eUt13uGzKHA.5332(a)TK2MSFTNGP02.phx.gbl... >>> >> "Pete Delgado" <Peter.Delg...(a)NoSpam.com> wrote... >>> >>> >>> I believe that your interpretation of "fault >>> >>> tolerance" >>> >>> is that a >>> >>> catastrophic event could happen to your system and >>> >>> you >>> >>> application >>> >>> would not lose *any* data. Is this the definition >>> >>> that >>> >>> you are using? >>> >>> >> Absent any catastrophic events, a system might still >>> >> be >>> >> called >>> >> "fault tolerant" if it managed at least one successful >>> >> run under >>> >> controlled conditions on developer's machine, despite >>> >> all >>> >> faults >>> >> with its design and implementation ;-) >>> >>> > Given his level of understanding, I sincerely doubt >>> > that >>> > his system can possibly overcome all of the faults that >>> > you mention! ;-) >>> >>> > -Pete >>> >>> Since many of these require redundant hardware and my >>> initial budget can not afford redundant hardware these >>> other >>> faults will not be initially accounted for. My >>> understanding >>> would be much better if people would explain their >>> underlying reasoning.- Ocultar texto de la cita - >>> >>> - Mostrar texto de la cita - >> >> This is a newsgroup, not a place to transcribe books and >> books on MFC, >> Windows programming, OS, databases, TCP/IP protocol, etc. >> It's not a place where people have to explain all their >> background, >> and all their "been there, done that"s, either, though >> it's certainly >> entertaining to read some of those stories. >> If you ask the experts, trust the experts > >The experts are telling me that my real-time process does >not need to be memory resident. > *** I said that (a) real-time has to be defined in terms of a response window (b) there are a large number of features whose time constants have the property that they can interfere with the requirement (a), of which paging MIGHT be one (but you have insisted that it will ALWAYS be a problem, in ANY realtime system, under ANY conditions) but (b) scheduling, cache behavior, priority inversion, and many other charactersitics might ALSO be a problem and (c) anything which interferes with (a) must be eliminated. Paging MIGHT be one of these. Please quote me correctly. joe Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:51 See below... On Fri, 26 Mar 2010 14:19:38 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%23BF%23oYRzKHA.5288(a)TK2MSFTNGP05.phx.gbl... >> Peter Olcott wrote: >> >>> The experts are telling me that my real-time process does >>> not need to be memory resident. >> >> Once again, you are lying to suit your needs. You really >> don't understand how the Intel chip and Preemptive and >> Protected Mode operating systems works which you explained >> over and over again, and even provided LINKS for your >> reading and verification. >> >> If I had to guess, the reason why you don't understand any >> of this is because you are clueless of the history of the >> INTEL chip starting with its Memory Segmentation Model to >> the introduction of Real Mode vs Protected Model hardware >> and operating systems, starting with DMPI. > >Introduced with the 80386, with the 80286 being the prequel. > >> Start reading about the Intel Chip, Memory Segmentation, >> Protected Mode Operating Systems and then maybe, just >> maybe, but I extremely doubt it, you will get some >> inkling of whats going on. >> >> -- >> HLS > >Red Herring! It does not address my need to keep my real >time process memory resident. Its just a bunch of extraneous >details intended to bypass rather than address the point. > **** Yet you persist in telling us that Memory Mapped Files must necessarily INCREASE paging faults, which anyone who understood ANYTHING about how the system works, which we have tried to tell you many, many times, cannot POSSIBLY work because they will exhibit behavior different from what we know they ACTUALLY implement. You hold the belief which is not supported by real behavior that ANY page fault provides unacceptable behavior, then toss out the requirement that you must use a transacted database to provide fault tolerance (or even just a simple FIFO) with no consideration of the relative performance, no DATA to show what the perfomrance is, and a mystical faith that any problem has a magic feature in the software that will make the problem go away. For example, that by disabling MySQL transactioning you can avoid the overhead without actually reducing reliability or, as you call it, "fault tolerance". Then you insist that fault tolerance is non-negotiable. Like page faults and the 500ms window, both of which you clearly don't care about, because if you did, you wouldn't suggest using MySQL as a FIFO, or insist suddenly that fault tolerance is a non-negotiable requirement. Or using PayPal as part of processing the user request. joe **** Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:54
See below... On Fri, 26 Mar 2010 15:31:38 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:uuMACxRzKHA.2436(a)TK2MSFTNGP04.phx.gbl... >> Peter Olcott wrote: >> >> >>> Red Herring! It does not address my need to keep my real >>> time process memory resident. Its just a bunch of >>> extraneous details intended to bypass rather than address >>> the point. >> >> That is the third time you said that. Do you think you >> got your 'Ah ha'? >> >> First of all, you never mentioned you were running this >> this vapor process of yours in real time priority status. > >I kept saying that it had to be as fast as possible. I >mentioned a 500 ms maximum response time. How does this not >add up to real time? > *** THis is definitely "real time" because you have specified T=500ms. Then, for reasons that make no sense whatsoever, you think that small numbers of page faults are going to matter, but the cost of a MySQL FIFO or a PayPal transaction are going to be zero. I guess I can't follow this logic. **** >> >> You were told you can raised the thread and process >> classification but that DOES NOT, I REPEAT IT DOES NOT >> stop the process from being preempted. So it DOES NOT >> have 100% running status. It CAN NOT PHYSICALLY BE >> POSSIBLE with all the THREADS running. This has been one >> of the key points that continues to go over your head. It >> just has higher a priority classification under the >> Windows Scheduling logic. This is why I told you you would >> be better off running DOS under a SINGLE CPU system? > >Yeah right, like DOS can really handle multiple Gigabytes. >The last DOS that I know of had to do all sorts of tricks to >get to 16 MB. (Switching back and forth between protected >and real mode, using tiny shared swap areas) *** And you know the costs of these, right? **** > >> >> 2nd, the issue was NOT your process, but your DATA. That >> was the focus. >> >> 3rd, you haven't tried to any of this (because you don't >> have software, just vaporware) to show that a hog in >> normal thread classification, might be a monster hog in >> Real Time classification. That is why it is NOT >> recommended and you only do so when you are nopt creating >> others pressures in the system. A Real Time process is >> NOT to kill others, it has to appear like its running all >> the time without interrupting anything else. >> >> So please. >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |