Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 28 Mar 2010 21:27 Some people trust information that comes from the horse's mouth. Others seem to consider what comes from the other end. Others choose to act as if they are the other end. joe On Mon, 22 Mar 2010 21:58:07 -0400, Hector Santos <sant9442(a)nospam.gmail.com> wrote: >Peter Olcott wrote: > >>> I don't see that as a contradiction at all. Your process >>> gets 4GB Virtual Memory. There was a moment in space and >>> time that your program did not need the OS to page data >>> into your WORKING SET which is the virtual memory in >>> active use by your program. >> >> It was not a moment in time it was a 12 hour time period. >> The only reason that It ended was that I was convinced that >> twelve hours was enough. > > >You know, you need to really stop this horse stuff of yours. Its >right there , straight from the horse mouth. Why did you choose to >ignore this? I'll show it again: > > http://support.microsoft.com/kb/555223 > > In modern operating systems, including Windows, application > programs and many system processes *ALWAYS* reference memory using > virtual memory addresses which are automatically translated to real > (RAM) addresses by the hardware. Only core parts of the operating > system kernel bypass this address translation and use real memory > addresses directly. > > Virtual Memory is always in use, *EVEN* when the memory required > by all running processes does not exceed the amount of RAM > installed on the system. > >WHY ARE YOU IGNORING THIS? > >You got 8GB on your box and your process is wanting 4GB - it is STILL >virtualize no matter what you do or say. > >The whole point of this thread that you SAID you can not run a 2nd >process because kills your system. > >Well of course, because now you need 8GB for 2 processes! > >If you don't single source the data as a sharable memory, then YOU >WILL NEVER be able to run but only 1 process on your machine. > >Thats not because of the physical limitations of the machine. Its >because your PROGRAM is FLAWED and NOT designed for any kind of >scalability or usage but as a single process application - nes pas! > >I illustrated and proved to you with posted code how to optimized the >process so that YOU can scale and leverage the power of your machine. > >Right now you are utilizing it and using it incorrectly! You really >wasted your money, and if think the solution is to scale out, well, >thats your problem, because you are dumping money down the drain for >nothing. Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 21:39 Ee below... On Thu, 25 Mar 2010 14:19:58 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:seumq5ld090kq8g6430sslcg8th98ocehj(a)4ax.com... >> See below... >> On Thu, 25 Mar 2010 09:08:25 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:OCPux1AzKHA.3884(a)TK2MSFTNGP06.phx.gbl... >>>> Peter Olcott wrote: >>>> >>>>>>> Ah so this is the code that you were suggesting? >>>>>>> I won't be able to implement multi-threading until >>>>>>> volume >>>>>>> grows out of what a single core processor can >>>>>>> accomplish. >>>>>>> I was simply going to use MySQL for the inter-process >>>>>>> communication, building and maintaining my FIFO >>>>>>> queue. >>>>>> **** >>>>>> Well, I can think of worse ways. For example, writing >>>>>> the data to a floppy disk. Or >>>>>> punching it to paper tape and asking the user to >>>>>> re-insert the paper tape. MySQL for >>>>>> interprocess communication? Get serious! >>>>> >>>>> Can you think of any other portable way that this can >>>>> be >>>>> done? >>>>> I would estimate that MySQL would actually keep the >>>>> FIFO >>>>> queue resident in RAM cache. >>>> >>>> Huh? >>>> >>>> Will MySQL will keep a FIFO queue resident? >>>> >>>> WOW! This is unbelievable. >>>> >>>> Do you know what MySQL is? Or even a FIFO queue? >>> >>>Do you know what file caching is? I know that a SQL >>>provider >>>would not be required to always hit disk for a 100K table >>>when multiple GB of RAM are available. >> *** >> Do you know what a "committed transaction" is? And why it >> REQUIRES hitting the disk? Note >> that most table accesses in databases are inherently >> treated as transactions to guarantee >> database integrity even at the single-table level. And >> could you cite a reference that a > >Right and depending on what is being accomplished one might >discard non crucial integrity for speed. If the sequence of >steps can be easily reconstructed because the request is >definitely committed to disk, then making sure that >intervening steps are also committed is not as crucial. **** And how is this consistent with your new magical buzzphrase "fault tolerant"? Either you want reliability, or you don't. If you want fault tolerance, you are going to have to hit the disk; if you don't want to hit the disk, don't come out of left field with a requirement that is INCONSISTENT with your other requirements? And did you check that MySQL allows you to disable its transacted model? **** > >Since the difference in response time can be large it might >be the case that some SQL providers allow you to tradeoff >integrity for performance on a case by case basis. *** Ohh, "some" providers "allow" this! You know this how? And you know *a priori* that MySQL is such a provider, and it allows this? And how is this consistent with the "fault tolerant" requirement that just appeared out of nowhere? I guess this is consistent with your "some sort of" handwave design methodology, the wishful-thinking model you are so fond of. The one that always has "some sort of" magical solution that makes your problem go away. I guess that you have a firm grasp of all these concepts and know that everything can be solved by magical handwave that says "well, given this is a problem, there is a magical solution that will make it go away, and I trust that this massively complex system I am working in provides such a magical solution, so there will be no problem." Kind of scary, actually. **** Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 21:44 See below... On Thu, 25 Mar 2010 09:35:19 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:otnmq553u2c9hdhabfe3ss74uct7pv0flu(a)4ax.com... >> See below... >> On Wed, 24 Mar 2010 23:14:32 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:snnlq5lfv1a59n003k5gkk8a6plb41ruj1(a)4ax.com... >>>> See below... >>>> >>>> On Tue, 23 Mar 2010 14:11:28 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>>Ah so this is the code that you were suggesting? >>>>>I won't be able to implement multi-threading until >>>>>volume >>>>>grows out of what a single core processor can >>>>>accomplish. >>>>>I was simply going to use MySQL for the inter-process >>>>>communication, building and maintaining my FIFO queue. >>>> **** >>>> Well, I can think of worse ways. For example, writing >>>> the >>>> data to a floppy disk. Or >>>> punching it to paper tape and asking the user to >>>> re-insert >>>> the paper tape. MySQL for >>>> interprocess communication? Get serious! >>> >>>Can you think of any other portable way that this can be >>>done? >> **** >> EnqueueRequest() which takes a description of the request. >> And is implemented on each >> platform with platform-specific code that is optimized for >> that platform. Your concept of >> "Portable" is incorrect here. you want the FASTEST way >> that the platform supports, and >> that may be completely different from every other >> platform. But if you code has a single >> abstraction, you are free to implement it in the best way >> on each platform! > >Is this supported on every platform? *** That's my entire point: you determine, for every platform, what 'this' is, and you WRITE CODE TO SUPPORT IT! In the abstract model, you have a system, perhaps even MySQL, but it is considered a low-level transport mechanism completely invisible to you your application architecture! You have two abstraction: "Enqueue" and "Dequeue" and that's what you write your application in terms of. On some systems, if you don't care in the slightest about performance, you might implment them as MySQL calls. In systems that have high-performance interprocess communication, you implement a platform-specific mechanism that is blindingly fast. Or have you missed the fundamental concepts of "abstraction"? joe ***** > >> >> Of course, this is so screamingly obvious that it is also >> clear that you wouldn't want to >> listen to any idea that doesn't fit your preconceived >> notion. >> *** >>>I would estimate that MySQL would actually keep the FIFO >>>queue resident in RAM cache. >> **** >> Oh, it's the guesswork-and-wishful-thinking approach ALL >> OVER AGAIN! DUH! And what part >> of "don't guess, measure" did you fail to understand from >> the last week of messages? WHY >> do you believe MySQL will behave this way? Cite the >> appropriate chapter from the MySQL >> manual, or, since it is open-source, show the subroutine >> that guarantees this! >> >> But feel free to live in your fantasy world! When the >> Real World accidentally matches >> your fantasies, you will win. >> joe >> >> **** >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 21:53 See below... On Thu, 25 Mar 2010 09:46:31 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:otnmq553u2c9hdhabfe3ss74uct7pv0flu(a)4ax.com... >> See below... >> On Wed, 24 Mar 2010 23:14:32 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:snnlq5lfv1a59n003k5gkk8a6plb41ruj1(a)4ax.com... >>>> See below... >>>> >>>> On Tue, 23 Mar 2010 14:11:28 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>>Ah so this is the code that you were suggesting? >>>>>I won't be able to implement multi-threading until >>>>>volume >>>>>grows out of what a single core processor can >>>>>accomplish. >>>>>I was simply going to use MySQL for the inter-process >>>>>communication, building and maintaining my FIFO queue. >>>> **** >>>> Well, I can think of worse ways. For example, writing >>>> the >>>> data to a floppy disk. Or >>>> punching it to paper tape and asking the user to >>>> re-insert >>>> the paper tape. MySQL for >>>> interprocess communication? Get serious! >>> >>>Can you think of any other portable way that this can be >>>done? >> **** >> EnqueueRequest() which takes a description of the request. >> And is implemented on each >> platform with platform-specific code that is optimized for >> that platform. Your concept of >> "Portable" is incorrect here. you want the FASTEST way >> that the platform supports, and > >I know what portable means. *** I doubt it. Your current conversation suggests you have no cllue what "portable" means. Apparently, to you, it means "there are no required source changes ANYWHERE" whereas to me it means "there is an optimum way to implement low-level interprocess data transport on each platform, and I will implement that for each platform". Maybe because I don't care about portability; I care that it is possible to write code that can run on multiple platforms, which is a somewhat different statement. **** > >> that may be completely different from every other >> platform. But if you code has a single >> abstraction, you are free to implement it in the best way >> on each platform! >I already knew that. > >One respondent on the forum that is supposed to know this >stuff >comp.programming.threads >suggested the disk file approach. **** Sure. And whoever this was did not have a CLUE about your issues of performance! Or was, as you have accused us, "playing head games" with you. Why is it that you assume that every forum but this one is going to produce a correct answer? **** > >If EnqueueRequest() is portable across at least >Windows/Unix/Linux then this may be the way that I go. This >will require a learning curve, but, this learning curve may >be justified. > **** It is portable because I have provided a platform-dependent implementation of it, and THAT makes it portable! **** >Do you have any ideas on maximizing fault tolerance? **** Yes. Design for fault-tolerance while you write the code. And be prepared to define "fault" and "tolerance". **** > >I was thinking that the only possible basis for fault >tolerance would require some sort of persistent storage. **** No, only SOME kinds of faults require persistent storage. And persistent storage requires hitting the disk, and that means you are going to pay that price, and you have said you are unwilling to let your database hit the disk. Wow! INCONSISTENT REQUIREMENTS! Who would have guessed? You can't have it both ways. Choose one. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 21:56
See below... On Thu, 25 Mar 2010 09:52:15 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%232%23$yCCzKHA.3264(a)TK2MSFTNGP06.phx.gbl... >> Joseph M. Newcomer wrote: >> >> >>> Or they were testing the limits of your credulity. >>> Reminds me of the Calvin & Hobbs >>> cartoon: The family is in the card. Calvin: "Dad, how do >>> they determine the weight limit >>> of a bridge?" Dad: "They run bigger and bigger trucks >>> over it until it collapses, then >>> they rebuild it exactly and post the weight limit" >> >> >> I like that one. :) >> >>> Can you explain why you would accept, without question, >>> such a patently absurd suggestion >>> from one newsgroup while ignoring all the good advice >>> you've been getting in this one? >> >> Well, he probably didn't ask the right question or they >> haven't had the time yet to pry it out of him. >> >> -- >> HLS > >How else can fault tolerance be provided without persistent >storage? **** This is clearly another case of fastening on some buzzphrase and mistinerpreting it unto death. It is ONE of the POSSIBLE implementations of "fault tolerance", but it is a valid specification only AFTER you have defined "fault" and "tolerance". The "tolerance" can be that the client notices that it has timed out waiting for a response and re-submits the request! This requires NO persisten storage on the server! So before this turns into another "Peter is totally clueless" discussion, you had better decide what "fault" means, what "tolerance" means, and what price you are willing to pay to get it. Thus far, the discussion has been another clueless concatenation of buzzconcepts, handwaves, and vague and contradictory requirements. joe ***** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |