Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 26 Mar 2010 23:44 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:uvgqq5t8e332pqdfncumdd7fdf4oo1b5bo(a)4ax.com... > See below... > > On Fri, 26 Mar 2010 15:55:26 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:ua4qq5dblfdtsm0hdqnnbki8i47dt7o8ci(a)4ax.com... >>> See below... >>> On Fri, 26 Mar 2010 09:47:20 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>message >>>>news:OYBobWLzKHA.5040(a)TK2MSFTNGP02.phx.gbl... >>>>> Peter Olcott wrote: >>>>> >>>>>> >>>>>> You and Joe did give me some excellent help, and I >>>>>> really >>>>>> appreciate that. The idea to base my web application >>>>>> on >>>>>> HTTP was the best. I do not appreciate the rudeness, >>>>>> and >>>>>> denigration. >>>>> >>>>> >>>>> We don't appreciate you telling us to prove something >>>>> that >>>>> is pretty much common knowledge about Windows >>>>> programming, >>>>> and furthermore, we don't appreciate when you still >>>>> don't >>>>> believe us and we advise you explore all yourself even >>>>> to >>>>> he extent of providing simulation code and you still >>>>> hassle us about it without even exploring it. When you >>>>> finally did partially some testing, you have kiddie >>>>> BUGS >>>>> and still come back to us to help you figure it out. >>>>> >>>>> Then you tried to front us with some fictitious >>>>> Specialty >>>>> Group that has all the answers, and LIED about they >>>>> were >>>>> agreeing with you. When asked to tell us what group >>>>> was >>>>> this, silence. >>>> >>>>The group is comp.programming.threads >>>>along with two linux groups and one unix group. >>>> >>>>Outlook express is losing some of the postings. I had to >>>>reply to a reply to Pete's message yesterday because >>>>Pete's >>>>original message never made it to outlook express. >>>> >>>>> >>>>> And even if you still didn't believe us, it isn't like >>>>> the >>>>> world is void of this information. This is all out >>>>> there >>>>> in googleland and you were given countless links, all >>>>> ignored. But its all there, yet you still refuse to >>>>> believe anything. >>>> >>>>I know for a fact that belief and disbelief are both >>>>errors >>>>of reasoning known as fallacies. Only comprehension of >>>>reasoning is a reliable means of discerning truth from >>>>falsehood. I apologize for not showing enough deference >>>>for >>>>the excellent free advice that you are providing. The >>>>advice >>>>that I could verify with reasoning was verifiably >>>>superb. >>> *** >>> And that is why I am getting frustrated with repeatedly >>> telling you the obvious. The >>> "obvious" is RUN THE $%&^ING TEST". You think you can >>> solve problems by "pure reason". >>> This applies only to abstract philosophical situations. >>> A >>> trained engineer knows that you >>> cannot predict large system behavior from small >>> examples; >>> ask any chemical engineer how to >>> scale up a process from a 1-liter lab to an industrial >>> process that can turn out tons a >>> day of whatever the product is. Scale up by three >>> orders >>> of magnitude and you have >>> completely different problems. >> >>The tests continue to pass under Windows. The test passes >>in >>a more limited sense under Linux. I can't understand, and >>apparently the Linux experts do not know why that when I >>hit >>450 MB of a 2 GB Linux system that reports that it is >>using >>300 MB the test slows down by 500%. > **** > My first reaction is "This is characteristic of the > working set manager kicking in" In > fact, there is essentially almost no other explanation. > Sounds like a 300MB working set > limit. The next question is how often the working set > manager kicks in and how it handles > paging out. Windows handles this by explicitly trimming > the working set by paging out the > LRU pages in the process. > > This follows naturally from an understanding of the > fundamental principles of operating > systems. > joe OK so I am convinced. It is a problem that I do have to solve. I shouldn't have skipped that last half of the book, but, I was going for maximum grades, and instead focused my time elsewhere. I still have the original copyright 1985 book, will this be good enough or could I learn more efficiently elsewhere? Operating System Concepts by Peterson and Silberschatz > **** >> >>A 2 GB Linux system using a total of 750 MB slows down the >>test from 90 MB/sec to 17MS/sec as soon as allocation >>exceeds 450 MB. > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 27 Mar 2010 01:39 Pete Delgado wrote: > I hope you mean Peter Olcott and not Pete (me)! ;-) Of course, Pete D. :) -- HLS
From: Joseph M. Newcomer on 27 Mar 2010 23:29 See below... On Fri, 26 Mar 2010 22:44:24 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:uvgqq5t8e332pqdfncumdd7fdf4oo1b5bo(a)4ax.com... >> See below... >> >> On Fri, 26 Mar 2010 15:55:26 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:ua4qq5dblfdtsm0hdqnnbki8i47dt7o8ci(a)4ax.com... >>>> See below... >>>> On Fri, 26 Mar 2010 09:47:20 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> >>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>message >>>>>news:OYBobWLzKHA.5040(a)TK2MSFTNGP02.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>>> >>>>>>> You and Joe did give me some excellent help, and I >>>>>>> really >>>>>>> appreciate that. The idea to base my web application >>>>>>> on >>>>>>> HTTP was the best. I do not appreciate the rudeness, >>>>>>> and >>>>>>> denigration. >>>>>> >>>>>> >>>>>> We don't appreciate you telling us to prove something >>>>>> that >>>>>> is pretty much common knowledge about Windows >>>>>> programming, >>>>>> and furthermore, we don't appreciate when you still >>>>>> don't >>>>>> believe us and we advise you explore all yourself even >>>>>> to >>>>>> he extent of providing simulation code and you still >>>>>> hassle us about it without even exploring it. When you >>>>>> finally did partially some testing, you have kiddie >>>>>> BUGS >>>>>> and still come back to us to help you figure it out. >>>>>> >>>>>> Then you tried to front us with some fictitious >>>>>> Specialty >>>>>> Group that has all the answers, and LIED about they >>>>>> were >>>>>> agreeing with you. When asked to tell us what group >>>>>> was >>>>>> this, silence. >>>>> >>>>>The group is comp.programming.threads >>>>>along with two linux groups and one unix group. >>>>> >>>>>Outlook express is losing some of the postings. I had to >>>>>reply to a reply to Pete's message yesterday because >>>>>Pete's >>>>>original message never made it to outlook express. >>>>> >>>>>> >>>>>> And even if you still didn't believe us, it isn't like >>>>>> the >>>>>> world is void of this information. This is all out >>>>>> there >>>>>> in googleland and you were given countless links, all >>>>>> ignored. But its all there, yet you still refuse to >>>>>> believe anything. >>>>> >>>>>I know for a fact that belief and disbelief are both >>>>>errors >>>>>of reasoning known as fallacies. Only comprehension of >>>>>reasoning is a reliable means of discerning truth from >>>>>falsehood. I apologize for not showing enough deference >>>>>for >>>>>the excellent free advice that you are providing. The >>>>>advice >>>>>that I could verify with reasoning was verifiably >>>>>superb. >>>> *** >>>> And that is why I am getting frustrated with repeatedly >>>> telling you the obvious. The >>>> "obvious" is RUN THE $%&^ING TEST". You think you can >>>> solve problems by "pure reason". >>>> This applies only to abstract philosophical situations. >>>> A >>>> trained engineer knows that you >>>> cannot predict large system behavior from small >>>> examples; >>>> ask any chemical engineer how to >>>> scale up a process from a 1-liter lab to an industrial >>>> process that can turn out tons a >>>> day of whatever the product is. Scale up by three >>>> orders >>>> of magnitude and you have >>>> completely different problems. >>> >>>The tests continue to pass under Windows. The test passes >>>in >>>a more limited sense under Linux. I can't understand, and >>>apparently the Linux experts do not know why that when I >>>hit >>>450 MB of a 2 GB Linux system that reports that it is >>>using >>>300 MB the test slows down by 500%. >> **** >> My first reaction is "This is characteristic of the >> working set manager kicking in" In >> fact, there is essentially almost no other explanation. >> Sounds like a 300MB working set >> limit. The next question is how often the working set >> manager kicks in and how it handles >> paging out. Windows handles this by explicitly trimming >> the working set by paging out the >> LRU pages in the process. >> >> This follows naturally from an understanding of the >> fundamental principles of operating >> systems. >> joe > >OK so I am convinced. It is a problem that I do have to >solve. I shouldn't have skipped that last half of the book, >but, I was going for maximum grades, and instead focused my >time elsewhere. I still have the original copyright 1985 >book, will this be good enough or could I learn more >efficiently elsewhere? > Operating System Concepts by Peterson and Silberschatz > **** Almost every introduction-to-operating-systems book gives only general principles and simplified-for-students overviews. Real systems are never as simple as the textbooks describe, which is why those of us who work in them are aware of the differences. We took those courses too, and we have sometimes taught those courses. I always tell my students, "The course you are about to take is a view from 17,000 feet, sometimes through clouds. I am going to give you general principles, but the reality, when you are down on the ground, can be quite different. But I could take any one-hour module of this course and expand it to five or six hours, and STILL not cover every detail. Discovering the details is up to you. All I'm giving you is the Big Picture, but from 17,000 feet you don't see the potholes, detours, and "under construction" signs that you will from the ground" For example, in one API I say "There are about two dozen options for this parameter. I'm going to show you the three most important. You will need to read about the rest on your own. They apply only in rare and exotic circumstances, which you may actually have. But if I talked about every option and the conditions under which it applied, this section would take four hours, and you would not even understand half of the scenarios. Just know that there is a LOT more going on here than I have time to tell you about. I trust your ability to read about it on your own" For example, most books talk about paging as if it is magic. They don't actually tell you how to set up the x86 control registers, GDT and/or LDT, and the precise sequence in which all of these must be done (I know; I once switched a processor from real to virtual mode, and it took DAYS of study and testing to figure out how to write the four-instruction sequence required! Before the LPSW instruction (Load Processor Status Word) instruction was done, a very, very precise and delicate sequence of instructions had to precede it. And the PDP-11 Unix context-swap code has a famous comment "You are not meant to understand this". You will not find the REAL details in these introductory texts. So be aware that they are simply general overviews, which in an different era would all have been called "Operating Systems Internals for Dummies" because they are so superficial compared to what you need to do to implement REAL operating systems on REAL hardware. I learned the Hydra operating system by having the original implementors give us a very detailed code walkthrough through a six-inch listing, an act which took several weeks of several hours per day, several days per week. A LOT of hours, on the order of 15 a week, sometimes more. I'd guess we spent 100 hours in those walkthroughts reading the source code. And we spent the next several years actually making the system perform well, on a machine architecture orders of magnitude simpler than a Xeon* and an operating system which was in reality smaller than Windows or Unix and in principle less complex, even if it was implemented on a 16-processor PDP-11 system. You have spent less time than this reading about general principles of operating systems from 17,000 feet, through clouds, then start telling the rest of us who have grubbing around on the ground, digging up cans of worms and looking the worms in the eye, and building real systems, that we don't understand what you are talking about. Sorry, it doesn't work that way in the Real World. Those prettied-up-for-students texts are NOT definitive uncompromising principles, but just simplified guidelines that try to explain, in a simple way, what an operating system does internally. Real operating systems are not as pretty as the textbooks describe. And the ones that are build just like in the books don't work very well. They lack critical functionality, and usually are poor performers. There's a reason operating systems have complex APIs: we need those capabilities. Compare the early Unix API with a modern Unixoid (e.g., linux) API, and explain each difference. Then compare these APIs to the APIs described in early Unix-based textbooks. Big surprise: the textbooks describe really simplistic APIs that work in really simplistic ways. Example: if a processor took a fault, and the fault appeared to be hardware, we would run hardware diagnostics on it. The processor was removed from the pool of available processors. If it sucessfully ran diagnostics for N hours, then it would re-enter as an "I/O processor" and the devices attached to it became available; it would handle I/O and continue to run diagnostics. If it could run diagnostics successfully for K hours, for K > n, it reentered as an "application processor" and could run user code. This is REALITY. The cute textbooks don't tell you about this. Or about how bad pages were "mapped out" and recorded on the disk, so we knew which pages should not be used to hold data (in those days, core and DRAM were a good deal less reliable than they are now, and we had a mix of both in our system). Or why we had a timer interrupt on the printer, since the printer usually failed to interrupt when it was finished, so if we didn't get a respons in k ms, we assumed the line had printed and we printed the next line (it was the only way to keep the printer running!). Find THAT in your pretty "Principles of Operating Systems" book (or "Operating System Concepts", or whatever it is called; they are all pretty interchangeable) I give a lecture in the Operating Systems course where I give the grubby details of the Windows driver model to students who have written "device drivers" in the student operating system project. Seriously different models. joe *The PDP-11 ISP was a variable-length-instruction format using 7 registers, one of which was the stack pointer and one of which was the program counter. There was no speculative execution, no caches (we added them), no pipelines, and an absolutely deterministic execution, if you ignore the fact that the time required for a multiply, divide, or carry/borrow propagation for an Add or Subtract instruction would vary with the data; it averaged 0.3 MIPS. We built realtime systems on this architecture with very tight performance deadlines, and yes, paging would have been a disaster on such a slow machine with windows that varied from 100us to 1ms. For example, polling devices was faster than using interrupts (ask Mike Horowitz, who wrote our "terminal concentrators" than handled large numbers of serial ports and Ethernet connections to the mainframe). **** > >> **** >>> >>>A 2 GB Linux system using a total of 750 MB slows down the >>>test from 90 MB/sec to 17MS/sec as soon as allocation >>>exceeds 450 MB. >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 27 Mar 2010 23:33 Seee below... On Thu, 25 Mar 2010 17:27:37 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:1tnnq5tmhhkeb3hv0f8maucmu0ppds3757(a)4ax.com... >> See below... >> On Thu, 25 Mar 2010 14:10:48 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>>> And what did you miss about "scalability"? Oh, that;s >>>> right, you will just throw more >>>> hardware at it. And rely on your ISP to provide >>>> load-balancing. Have you talked to them >>>> about how they do load-balancing when you have multiple >>>> servers? >>>> joe >>> >>>My whole focus was to leverage memory to gain speed. >> **** >> If your only tools is a hammer, all your problems look >> like nails. Guess what: we tried >> to explain to you how to leverage memory to gain speed, >> using a different perspective, but >> you don't want to pay attention to us. Multithreaded >> single process and >> memory-mapped-file multiprocess BOTH leverage memory usage >> to gain speed. This is the >> consequence of deep understanding of reality. >> joe >> **** > >So I have been convinced for several days now that >multi-threads could be a good idea. I still see no benefit >of adding memory mapped files to this mix. You have >explicitly failed to point out any incremental benefit that >I get from memory mapped files over multiple threads sharing >the same memory in the case where data remains resident in >memory as my testing has shown that it does. (I will also >test this on Linux). *** I believe I pointed out that it is likely to result in FEWER page faults, since parts of the DFA you don't access don't actually have to be brought in. And the initialization is going to be faster than several minutes, because there's nothing to "read in" and no storage allocation is going to be required! Is this not screamingly obvious? Just from basic principles? **** > >What you do instead is continue to insist that data does not >remain resident in memory. I will test this in Linux very >soon. **** Interesting that in a later post you reveal that it DOESN'T remain resident in memory, proving that I was right; I stated that you have no guarantees. Or are those numbers about how linux slows down pure fiction? I think they prove what I said, and that it is true in linux (and it looks like a Working Set issue, which I pointed out somewhere in my first response to your post). joe **** > >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 27 Mar 2010 23:42
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:31htq5tcpos7o7vah75d0tp02p6768q8ai(a)4ax.com... > See below... > On Fri, 26 Mar 2010 22:44:24 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: >>OK so I am convinced. It is a problem that I do have to >>solve. I shouldn't have skipped that last half of the >>book, >>but, I was going for maximum grades, and instead focused >>my >>time elsewhere. I still have the original copyright 1985 >>book, will this be good enough or could I learn more >>efficiently elsewhere? >> Operating System Concepts by Peterson and Silberschatz >> > **** > Almost every introduction-to-operating-systems book gives > only general principles and > simplified-for-students overviews. Real systems are never > as simple as the textbooks But then (I guess I will ask explicitly) How much has this stuff changed in 25 years? My book is 25 years old (it was new when I took the course). The only thing that I really need to know about VM right now, is how to minimize its impact on my performance. #include <sys/mman.h> int mlock(const void *addr, size_t len); int munlock(const void *addr, size_t len); int mlockall(int flags); int munlockall(void); Is there anything like this in Windows? > describe, which is why those of us who work in them are > aware of the differences. We took > those courses too, and we have sometimes taught those > courses. I always tell my students, > "The course you are about to take is a view from 17,000 > feet, sometimes through clouds. I > am going to give you general principles, but the reality, > when you are down on the ground, > can be quite different. But I could take any one-hour > module of this course and expand it > to five or six hours, and STILL not cover every detail. > Discovering the details is up to > you. All I'm giving you is the Big Picture, but from > 17,000 feet you don't see the > potholes, detours, and "under construction" signs that you > will from the ground" For > example, in one API I say "There are about two dozen > options for this parameter. I'm > going to show you the three most important. You will need > to read about the rest on your > own. They apply only in rare and exotic circumstances, > which you may actually have. But > if I talked about every option and the conditions under > which it applied, this section > would take four hours, and you would not even understand > half of the scenarios. Just know > that there is a LOT more going on here than I have time to > tell you about. I trust your > ability to read about it on your own" > > For example, most books talk about paging as if it is > magic. They don't actually tell you > how to set up the x86 control registers, GDT and/or LDT, > and the precise sequence in which > all of these must be done (I know; I once switched a > processor from real to virtual mode, > and it took DAYS of study and testing to figure out how to > write the four-instruction > sequence required! Before the LPSW instruction (Load > Processor Status Word) instruction > was done, a very, very precise and delicate sequence of > instructions had to precede it. > And the PDP-11 Unix context-swap code has a famous comment > "You are not meant to > understand this". You will not find the REAL details in > these introductory texts. So be > aware that they are simply general overviews, which in an > different era would all have > been called "Operating Systems Internals for Dummies" > because they are so superficial > compared to what you need to do to implement REAL > operating systems on REAL hardware. > > I learned the Hydra operating system by having the > original implementors give us a very > detailed code walkthrough through a six-inch listing, an > act which took several weeks of > several hours per day, several days per week. A LOT of > hours, on the order of 15 a week, > sometimes more. I'd guess we spent 100 hours in those > walkthroughts reading the source > code. And we spent the next several years actually making > the system perform well, on a > machine architecture orders of magnitude simpler than a > Xeon* and an operating system > which was in reality smaller than Windows or Unix and in > principle less complex, even if > it was implemented on a 16-processor PDP-11 system. You > have spent less time than this > reading about general principles of operating systems from > 17,000 feet, through clouds, > then start telling the rest of us who have grubbing around > on the ground, digging up cans > of worms and looking the worms in the eye, and building > real systems, that we don't > understand what you are talking about. Sorry, it doesn't > work that way in the Real World. > Those prettied-up-for-students texts are NOT definitive > uncompromising principles, but > just simplified guidelines that try to explain, in a > simple way, what an operating system > does internally. Real operating systems are not as pretty > as the textbooks describe. And > the ones that are build just like in the books don't work > very well. They lack critical > functionality, and usually are poor performers. There's a > reason operating systems have > complex APIs: we need those capabilities. Compare the > early Unix API with a modern > Unixoid (e.g., linux) API, and explain each difference. > Then compare these APIs to the > APIs described in early Unix-based textbooks. Big > surprise: the textbooks describe really > simplistic APIs that work in really simplistic ways. > > Example: if a processor took a fault, and the fault > appeared to be hardware, we would run > hardware diagnostics on it. The processor was removed > from the pool of available > processors. If it sucessfully ran diagnostics for N > hours, then it would re-enter as an > "I/O processor" and the devices attached to it became > available; it would handle I/O and > continue to run diagnostics. If it could run diagnostics > successfully for K hours, for > K > n, it reentered as an "application processor" and > could run user code. This is > REALITY. The cute textbooks don't tell you about this. > Or about how bad pages were > "mapped out" and recorded on the disk, so we knew which > pages should not be used to hold > data (in those days, core and DRAM were a good deal less > reliable than they are now, and > we had a mix of both in our system). Or why we had a > timer interrupt on the printer, > since the printer usually failed to interrupt when it was > finished, so if we didn't get a > respons in k ms, we assumed the line had printed and we > printed the next line (it was the > only way to keep the printer running!). Find THAT in your > pretty "Principles of Operating > Systems" book (or "Operating System Concepts", or whatever > it is called; they are all > pretty interchangeable) > > I give a lecture in the Operating Systems course where I > give the grubby details of the > Windows driver model to students who have written "device > drivers" in the student > operating system project. Seriously different models. > joe > > *The PDP-11 ISP was a variable-length-instruction format > using 7 registers, one of which > was the stack pointer and one of which was the program > counter. There was no speculative > execution, no caches (we added them), no pipelines, and an > absolutely deterministic > execution, if you ignore the fact that the time required > for a multiply, divide, or > carry/borrow propagation for an Add or Subtract > instruction would vary with the data; it > averaged 0.3 MIPS. We built realtime systems on this > architecture with very tight > performance deadlines, and yes, paging would have been a > disaster on such a slow machine > with windows that varied from 100us to 1ms. For example, > polling devices was faster than > using interrupts (ask Mike Horowitz, who wrote our > "terminal concentrators" than handled > large numbers of serial ports and Ethernet connections to > the mainframe). > **** >> >>> **** >>>> >>>>A 2 GB Linux system using a total of 750 MB slows down >>>>the >>>>test from 90 MB/sec to 17MS/sec as soon as allocation >>>>exceeds 450 MB. >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |