Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 22 Mar 2010 17:27 "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message news:OIehRQgyKHA.4752(a)TK2MSFTNGP04.phx.gbl... > > "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message > news:osWdnaGZ3q06RTrWnZ2dnUVZ_uudnZ2d(a)giganews.com... >> >> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >> message >> news:ecdfq5lb57qrou47d1ppaupsi6t2guu7nv(a)4ax.com... >>> **** >>> He has NO CLUE as to what a "memory-mapped file" >>> actually is. This last comment indicates >> >> http://en.wikipedia.org/wiki/Memory-mapped_file >> Apparently I do. > > I think you would be far better served by looking at > Windows specific information on memory mapped files such > as that which Joe suggested to you some time ago: > Richter's Programming Applications for Microsoft Windows > 4th. > > -Pete > > Joe kept insisting and continues to insist that my data is not resident in memory. After loading my data and waiting twelve hours the process monitor reports zero page faults, when I execute my process and run it to completion. How does this not prove Joe is wrong (At least in the specific instance of one execution of my process)? (1) The process monitor is lying. (2) Page faults do not measure virtual memory usage.
From: Peter Olcott on 22 Mar 2010 17:32 You tell me all about pages faults, yet the process monitor reports zero page faults, and you continue to claim that its all about page faults, and virtual memory. Pages faults indicate victual memory usage right? A lack of page faults indicates a lack of virtual memory usage right? How does this not prove that my data is in memory and thus you are wrong when you say that my data is not resident in memory? Load data, wait twelve hours check page faults reported execute process to completion same number of pages faults as before the 12 hours therefore data remained resident in memory "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:arnfq5h441qhdl7hnge9rcrfgger1tm0s9(a)4ax.com... > See below... > On Mon, 22 Mar 2010 15:47:46 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:rnjfq5ls8fpma0kvrc6odhuvqfignso8m5(a)4ax.com... >>> See below... >>> On Mon, 22 Mar 2010 14:40:57 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>> >>>> >>>>Try and explain exactly how cache can possibly help when >>>>there is most often essentially no spatial or temporal >>>>locality of reference. >>>> >>> **** >>> While caches work well with locality of reference, that >>> is >>> just a heuristic for predicting >>> cache effects. Locality of reference is not the point; >>> maximizing cache hits is the >>> point. And this can happen, particularly on a shared L3 >>> cache, based solely on the cache >>> replacement algorithm. We use locality of reference as >>> the "easy" approach to determining >>> the likelikhood of cache hits, because it is easy to >>> analyze in applications that process >>> regular data like matrices and arrays. But it is not >>> the >>> theoretical optimum approach. If >>> you took the time to understand how caches work this >>> would >>> be obvious to you. >>> >>> Try to explain why you believe this when you have run no >>> experiments that have any >>> meaning. The difference is that I am saying YOU HAVE NO >>> DATA, and you are saying I KNOW >>> WHAT IS GOING TO HAPPEN, I DON'T NEED NO STINKIN' FACTS. >>> I don't believe you really know >> >>I do have data, and present this data many times and you >>two >>simply blow it off. >>Two processes take 2.75 times as long as one process. What >>could this mean besides resource contention? > *** > An what does a two-process measure tell you about > multithreading? NOTHING! > > You have a flawed experiment that is generalizing from the > wrong premises. > > OF COURSE two huge processes are going to have resource > contention! And are these running > on a single core or multicore machine? But multiple > threads have signficant lower > resource conention, and you ignore this fact and think > your two-process model is the > absolute authoritative experiment. I'd never trust data > like this, and if I were a > product manager I'd send you back to get real data. You > are your own product manager, and > should recognize (give how much we've told you) and you > should send yourself back to get > valid data. > joe > >> >>You tell me all about pages faults, yet the process >>monitor >>reports zero page faults, and you continue to claim that >>its >>all about page faults, and virtual memory. Pages faults >>indicate victual memory usage right? A lack of page faults >>indicates a lack of virtual memory usage right? >> >>> what is going to happen, you are just guessing. I know >>> what I would do: as an egineer >>> (there's that nasty word again) I'd go out at GET the >>> facts. Then, I could say "But I >>> have run this experiment, and it substantiates my >>> theory" >>> and that would be useful >>> knowledge. But you just blindly claim you "know" what >>> is >>> going to happen. I'` `m supposed >>> to take this seriously from someone who dosen't even >>> understand why a Memory Mapped File >>> is going to give superior performance? Given your >>> demonstrated lack of understanding of >>> operating systems, why should I believe ANY assertion >>> you >>> make, unless you have the data >>> to back it up? Hell, I wouldn't believe MY OWN theories >>> about performance without data, >>> and 15 years of performance measurement have convinced >>> me >>> of one absolute fact: "Ask a >>> programmer when the performance bottleneck is in their >>> code, and you will get a wrong >>> answer". That rule NEVER failed me in 15 years of real >>> performance measurement of real >>> programs on real machines, and I believe it today. Botom >>> line: only actual performance >>> data proves anything. Theories about where performance >>> is >>> going are universally wrong >>> unless supported by actual measurements. You have a >>> p-baked theory, for p considerably >>> less than 0.5 (p==0.5 is half-baked), and you refuse to >>> see test your theory. Not a >>> robust approach to building systems. You may be >>> absolutely correct, but you cannot PROVE >>> it without data. >>> joe >>> **** >>> >>>>>> >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 22 Mar 2010 17:32 See below... On Mon, 22 Mar 2010 15:52:54 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:ecdfq5lb57qrou47d1ppaupsi6t2guu7nv(a)4ax.com... >> See below... >> >> On Mon, 22 Mar 2010 10:31:17 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:%23Q4$1KdyKHA.404(a)TK2MSFTNGP02.phx.gbl... >>>> Joseph M. Newcomer wrote: >>>> >>>> >>>>> Note also if you use a memory-mapped file and two >>>>> processes share the same mapping object >>>>> there is only one copy of the data in memory! THis >>>>> has >>>>> not previously come up in >>>>> discussions, but could be critical to your performance >>>>> of >>>>> multiple processes. >>>>> joe >>>> >>>> >>>> He has been told that MMF can help him. >>>> >>>> -- >>>> HLS >>> >>>Since my process (currently) requires unpredictable access >>>to far more memory than can fit into the largest cache, I >>>see no possible way that adding 1000-fold slower disk >>>access >>>could possibly speed things up. This seems absurd to me. >> **** >> He has NO CLUE as to what a "memory-mapped file" actually >> is. This last comment indicates > > http://en.wikipedia.org/wiki/Memory-mapped_file > Apparently I do. *** And obviously you do not. You read one general wikipedia entry and fail to notice how it said that it improves I/O performance, and you have thought that this general article is definitive. OK, since you know all the answers, explain this: if I do CreateFileMapping and name a mapping object, and use the same name in two processes, what happens to my virtual memory maps when I do MapViewOfFile naming that mapping object handle? Please limit your response to fewer than 100 words, which should be enough to explain the implications. Note if you knew the correct answer, you would not have made the ridiculous statements you have made about memory mapped files. And you would have known that they will improve multiprocess performance. I repeat my assertion: you are clueless. Anyone who tries to use a generic wikipedia article as proof of being not clueless is metaclueless. joe **** > >> total and complete cluelessness, plus a startling >> inabilitgy to understand that we are >> making USEFUL suggestions because WE KNOW what is going on >> and he has no idea. >> >> Like you, I'm giving up. There is only so long you can >> beat someone over the head with >> good ideas which they reject because they have no idea >> what you are talking about, but >> won't expend any energy to learn about, or ask questions >> about. Since he doesn't >> understand what shared sections are, or what they buy, and >> that a MMF is the way to get >> shared sections, I'm dropping out of this discussion. He >> has found a set of "experts" who >> agree with him (your example apparently doesn't convey the >> problem correctly), thinks >> memory-mapped files limit access to disk speed (not even >> understanding they are FASTER >> than ReadFile!) and has failed utterly to understand even >> the most basic concepts of an >> operagin system (thinking it is like an automatic >> transmission, where you can use it >> without knowing or caring about how it works, when what he >> is really doing is trying to >> build a competition racing machine and saying "all that >> stuff about the engine is >> irrelevant", whereas anyone who does competition racing >> (like my next-door neighbor did >> for years) knows why all this stuff is critical. If he >> were a racer, and we told him >> about power-shiftting (shifting a manual transmission >> without involving the clutch), he'd >> tell us he didn't need to understand that. >> >> Sad, really. >> joe >> *** >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 22 Mar 2010 17:33 see below... On Mon, 22 Mar 2010 12:15:27 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:uGVpjmdyKHA.5948(a)TK2MSFTNGP06.phx.gbl... >> Peter Olcott wrote: >> >>> Since my process (currently) requires unpredictable >>> access to far more memory than can fit into the largest >>> cache, I see no possible way that adding 1000-fold slower >>> disk access could possibly speed things up. This seems >>> absurd to me. >> >> >> And I would agree it would be seem to be absurd to >> inexperience people. >> >> But you need to TRUST the power of your multi-processor >> computer because YOU are most definitely under utilizing >> it by a long shot. >> >> The code I posted is the proof! > >If it requires essentially nothing besides random access to >entirely different places of 100 MB of memory, thenn (then >and only then) would it be reasonably representative of my >process. Nearly all the my process does is look up in memory >the next place to look up in memory. *** Then the CORRECT approach is not to say "I don't want to waste time reading your example because it doesn't match my problem", the CORRECT approach is to say "I have no modified your example to more closely resemble my access patterns, ran it, and got THIS result" and show the results you got. joe > >> >> Your issue is akin to having a pickup truck, overloading >> the back, piling things on each other, overweight beyond >> the recommended safety levels per specifications of the >> car manufacturer (and city/state ordinances), and now your >> driving, speed, vision of your truck are all altered. >> Your truck won't go as fast now and if even if you could, >> things can fall, people can die, crashes can happen. >> >> You have two choices: >> >> - You can stop and unload stuff and come back and pick >> it up on >> 2nd strip, your total travel time doubled. >> >> - you can get a 2nd pick up truck, split the load and >> get >> on a four lanes highway and drive side by side, >> sometimes >> one creeps ahead, and the other moves ahead, and >> both reach >> the destination at the near same expected time. >> >> Same thing! >> >> You are overloading your machine to the point it working >> very very hard to satisfy your single thread process >> needs. You may "believe" it is working at optimal speeds >> because it has uninterrupted exclusive access but it is >> not reality. You are under utilizing the power of your >> machine. >> >> Whether you realize it or not, the overloaded pickup truck >> is smart and is stopping you every X milliseconds checking >> if you have a 2nd pickup truck to offload some work and do >> some moving for you!! >> >> You need to change your thinking. >> >> However, at this point, I don't think you have any coding >> skills, because if you did, you would be EAGERLY JUMPING >> at the code I provided to see for yourself. >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 22 Mar 2010 17:38
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:42ofq5h3d80a2pmmkmcq46o5hpodb2hi6c(a)4ax.com... > See below... > On Mon, 22 Mar 2010 15:52:54 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:ecdfq5lb57qrou47d1ppaupsi6t2guu7nv(a)4ax.com... >>> See below... >>> >>> On Mon, 22 Mar 2010 10:31:17 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>message >>>>news:%23Q4$1KdyKHA.404(a)TK2MSFTNGP02.phx.gbl... >>>>> Joseph M. Newcomer wrote: >>>>> >>>>> >>>>>> Note also if you use a memory-mapped file and two >>>>>> processes share the same mapping object >>>>>> there is only one copy of the data in memory! THis >>>>>> has >>>>>> not previously come up in >>>>>> discussions, but could be critical to your >>>>>> performance >>>>>> of >>>>>> multiple processes. >>>>>> joe >>>>> >>>>> >>>>> He has been told that MMF can help him. >>>>> >>>>> -- >>>>> HLS >>>> >>>>Since my process (currently) requires unpredictable >>>>access >>>>to far more memory than can fit into the largest cache, >>>>I >>>>see no possible way that adding 1000-fold slower disk >>>>access >>>>could possibly speed things up. This seems absurd to me. >>> **** >>> He has NO CLUE as to what a "memory-mapped file" >>> actually >>> is. This last comment indicates >> >> http://en.wikipedia.org/wiki/Memory-mapped_file >> Apparently I do. > *** > And obviously you do not. You read one general wikipedia > entry and fail to notice how it > said that it improves I/O performance, and you have > thought that this general article is > definitive. > > OK, since you know all the answers, explain this: if I do > CreateFileMapping and name a > mapping object, and use the same name in two processes, > what happens to my virtual memory > maps when I do MapViewOfFile naming that mapping object > handle? Please limit your > response to fewer than 100 words, which should be enough > to explain the implications. > > Note if you knew the correct answer, you would not have > made the ridiculous statements you > have made about memory mapped files. And you would have > known that they will improve > multiprocess performance. Apparently within the context of virtual memory usage, ** which I have shown in at least one instance does not apply. Zero page faults indicates zero virtual memory usage right? ** Do virtual memory maps apply to anything else besides virtual memory? > > I repeat my assertion: you are clueless. Anyone who > tries to use a generic wikipedia > article as proof of being not clueless is metaclueless. > joe > *** >> >>> total and complete cluelessness, plus a startling >>> inabilitgy to understand that we are >>> making USEFUL suggestions because WE KNOW what is going >>> on >>> and he has no idea. >>> >>> Like you, I'm giving up. There is only so long you can >>> beat someone over the head with >>> good ideas which they reject because they have no idea >>> what you are talking about, but >>> won't expend any energy to learn about, or ask questions >>> about. Since he doesn't >>> understand what shared sections are, or what they buy, >>> and >>> that a MMF is the way to get >>> shared sections, I'm dropping out of this discussion. >>> He >>> has found a set of "experts" who >>> agree with him (your example apparently doesn't convey >>> the >>> problem correctly), thinks >>> memory-mapped files limit access to disk speed (not even >>> understanding they are FASTER >>> than ReadFile!) and has failed utterly to understand >>> even >>> the most basic concepts of an >>> operagin system (thinking it is like an automatic >>> transmission, where you can use it >>> without knowing or caring about how it works, when what >>> he >>> is really doing is trying to >>> build a competition racing machine and saying "all that >>> stuff about the engine is >>> irrelevant", whereas anyone who does competition racing >>> (like my next-door neighbor did >>> for years) knows why all this stuff is critical. If he >>> were a racer, and we told him >>> about power-shiftting (shifting a manual transmission >>> without involving the clutch), he'd >>> tell us he didn't need to understand that. >>> >>> Sad, really. >>> joe >>> *** >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |