Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 22 Mar 2010 17:44 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:ifofq5hdvoesvnv8a6v70vi68fk8akcjld(a)4ax.com... > see below... > On Mon, 22 Mar 2010 12:15:27 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>message >>news:uGVpjmdyKHA.5948(a)TK2MSFTNGP06.phx.gbl... >>> Peter Olcott wrote: >>> >>>> Since my process (currently) requires unpredictable >>>> access to far more memory than can fit into the largest >>>> cache, I see no possible way that adding 1000-fold >>>> slower >>>> disk access could possibly speed things up. This seems >>>> absurd to me. >>> >>> >>> And I would agree it would be seem to be absurd to >>> inexperience people. >>> >>> But you need to TRUST the power of your multi-processor >>> computer because YOU are most definitely under utilizing >>> it by a long shot. >>> >>> The code I posted is the proof! >> >>If it requires essentially nothing besides random access >>to >>entirely different places of 100 MB of memory, thenn (then >>and only then) would it be reasonably representative of my >>process. Nearly all the my process does is look up in >>memory >>the next place to look up in memory. > *** > Then the CORRECT approach is not to say "I don't want to > waste time reading your example > because it doesn't match my problem", the CORRECT approach > is to say "I have no modified > your example to more closely resemble my access patterns, > ran it, and got THIS result" and > show the results you got. > joe I think that you must have paraphrased me incorrectly and I don't see the context in this post. I would not likely have concluded that it does not match my problem, and in fact remember asking Hector if it did match my problem. I stated my problem much more concisely than he stated his proof. The single sentence is above. Hector never told me whether or not he thought that it matched my algorithm's memory usage pattern. > >> >>> >>> Your issue is akin to having a pickup truck, overloading >>> the back, piling things on each other, overweight beyond >>> the recommended safety levels per specifications of the >>> car manufacturer (and city/state ordinances), and now >>> your >>> driving, speed, vision of your truck are all altered. >>> Your truck won't go as fast now and if even if you >>> could, >>> things can fall, people can die, crashes can happen. >>> >>> You have two choices: >>> >>> - You can stop and unload stuff and come back and >>> pick >>> it up on >>> 2nd strip, your total travel time doubled. >>> >>> - you can get a 2nd pick up truck, split the load and >>> get >>> on a four lanes highway and drive side by side, >>> sometimes >>> one creeps ahead, and the other moves ahead, and >>> both reach >>> the destination at the near same expected time. >>> >>> Same thing! >>> >>> You are overloading your machine to the point it working >>> very very hard to satisfy your single thread process >>> needs. You may "believe" it is working at optimal >>> speeds >>> because it has uninterrupted exclusive access but it is >>> not reality. You are under utilizing the power of your >>> machine. >>> >>> Whether you realize it or not, the overloaded pickup >>> truck >>> is smart and is stopping you every X milliseconds >>> checking >>> if you have a 2nd pickup truck to offload some work and >>> do >>> some moving for you!! >>> >>> You need to change your thinking. >>> >>> However, at this point, I don't think you have any >>> coding >>> skills, because if you did, you would be EAGERLY JUMPING >>> at the code I provided to see for yourself. >>> >>> -- >>> HLS >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 22 Mar 2010 17:49 Peter Olcott wrote: > Joe kept insisting and continues to insist that my data is > not resident in memory. If you have a 32 bit Windows OS, you are limited to just 2GB RAW ACCESS and 4GB of VIRTUAL MEMORY. If your process is loading 4GB, you are using virtual memory. > After loading my data and waiting twelve hours the process > monitor reports zero page faults, when I execute my process > and run it to completion. You're lying, you told me you have PAGE FAULTS but it settle down to zero, which is NORMAL. But start a 2nd process and you will get page faults. I also asked, now 5 times, to provide the MEMORY LOAD percentage which I even provided with a simple C program that you can compile, and you did not: // File: V:\bin\memload.cpp #include <stdio.h> #include <windows.h> void main(char argc, char *argv[]) { MEMORYSTATUS ms; ms.dwLength = sizeof(ms); GlobalMemoryStatus(&ms); printf("Memory Load: %d%%",ms.dwMemoryLoad); } Why can't you even do that? > How does this not prove Joe is wrong (At least in the > specific instance of one execution of my process)? > (1) The process monitor is lying. > (2) Page faults do not measure virtual memory usage. There are now what 4-5 participants in the thread who are telling your thinking is wrong and lack a understanding of the Windows and Intel hardware. lets get a few more like this guy with a somewhat layman description: http://blogs.sepago.de/helge/2008/01/09/windows-x64-all-the-same-yet-very-different-part-1/ and the #1 guy at Microsoft today! http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx If you DEFY what Mark Russinovich is saying here, you are CRAZY! -- HLS
From: Peter Olcott on 22 Mar 2010 17:59 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:%23F2oLmgyKHA.5360(a)TK2MSFTNGP06.phx.gbl... > Peter Olcott wrote: > >> Joe kept insisting and continues to insist that my data >> is not resident in memory. > > > If you have a 32 bit Windows OS, you are limited to just > 2GB RAW ACCESS and 4GB of VIRTUAL MEMORY. Yes, and that is another thing. I kept saying that I have a 64bit OS, and Joe kept forming his replies in terms of a 32-bit OS. > > If your process is loading 4GB, you are using virtual > memory. > >> After loading my data and waiting twelve hours the >> process monitor reports zero page faults, when I execute >> my process and run it to completion. > > > You're lying, you told me you have PAGE FAULTS but it > settle down to zero, which is NORMAL. But start a 2nd > process and you will get page faults. I only get the page faults until the data is loaded. After the data is loaded I get essentially no more page faults, even after waiting twelve hours before running my process to completion. After proving that my data is resident in RAM Joe continues to chide me for claiming that my data is resident in RAM. You guys just playing head games with me? > > I also asked, now 5 times, to provide the MEMORY LOAD > percentage which I even provided with a simple C program > that you can compile, and you did not: > > // File: V:\bin\memload.cpp > > #include <stdio.h> > #include <windows.h> > > void main(char argc, char *argv[]) > { > MEMORYSTATUS ms; > ms.dwLength = sizeof(ms); > GlobalMemoryStatus(&ms); > printf("Memory Load: %d%%",ms.dwMemoryLoad); > } > > Why can't you even do that? > >> How does this not prove Joe is wrong (At least in the >> specific instance of one execution of my process)? >> (1) The process monitor is lying. >> (2) Page faults do not measure virtual memory usage. > > There are now what 4-5 participants in the thread who are > telling your thinking is wrong and lack a understanding of > the Windows and Intel hardware. > > lets get a few more like this guy with a somewhat layman > description: > > http://blogs.sepago.de/helge/2008/01/09/windows-x64-all-the-same-yet-very-different-part-1/ > > and the #1 guy at Microsoft today! > > http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx > > If you DEFY what Mark Russinovich is saying here, you are > CRAZY! > > -- > HLS
From: Hector Santos on 22 Mar 2010 18:02 Peter Olcott wrote: > You tell me all about pages faults, yet the process monitor > reports zero page faults, and you continue to claim that its > all about page faults, and virtual memory. Its not a claim - its a fact. > Pages faults indicate victual memory usage right? It shows when your PROCESS is asking too much the can provide to you all in memory - it has to virtualize it. > A lack of page faults indicates a lack of virtual memory usage right? No. If its zero or not changing and I know your process is not, it means that your process working set is not demanding more than it can handle or OTHER processes have not chewed up memory, limiting your available memory. You have NO control over this UNLESS you explicitly told windows to use NO-CACHING, NO BUFFER I/O memory. Why do you refuse to believe this? Every Windows programmer has to know this to some degree. > How does this not prove that my data is in memory and thus > you are wrong when you say that my data is not resident in > memory? It doesn't. Allow a Technical Fellow Engineer at Microsoft explain it all to you: Pushing the Limits of Windows: Physical Memory http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx Pushing the Limits of Windows: Virtual Memory http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx Pushing the Limits of Windows: Paged and Nonpaged Pool http://blogs.technet.com/markrussinovich/archive/2009/03/26/3211216.aspx Are you going to argue with Mark too? -- HLS
From: Hector Santos on 22 Mar 2010 18:13
Peter Olcott wrote: > I think that you must have paraphrased me incorrectly and I > don't see the context in this post. I would not likely have > concluded that it does not match my problem, and in fact > remember asking Hector if it did match my problem. I stated > my problem much more concisely than he stated his proof. > > The single sentence is above. Hector never told me whether > or not he thought that it matched my algorithm's memory > usage pattern. I sure did, which you replied to the message, but decided to ignore the message details. The simulator provided the WORST CASE scenario of a serialize reading of your entire payload with single and multiple threads with NO other overhead. This is the highest pressure you can get for X number of threads reading the same memory. It proved your understanding of the memory I/O, caching and multi-threaded nature of a 32 bit preemptive operating system is extremely primitive. I also provided the scenario of random, unpredictable reading with single and multiple threads and proved how it was even faster because you are not reading the entire payload but parts of it randomly. Until you run the simulator, you will never grasp how fundamentally lacking your thinking is about Windows. -- HLS |