Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 22 Mar 2010 21:05 See below... On Mon, 22 Mar 2010 16:38:14 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:42ofq5h3d80a2pmmkmcq46o5hpodb2hi6c(a)4ax.com... >> See below... >> On Mon, 22 Mar 2010 15:52:54 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >>> http://en.wikipedia.org/wiki/Memory-mapped_file >>> Apparently I do. >> *** >> And obviously you do not. You read one general wikipedia >> entry and fail to notice how it >> said that it improves I/O performance, and you have >> thought that this general article is >> definitive. >> >> OK, since you know all the answers, explain this: if I do >> CreateFileMapping and name a >> mapping object, and use the same name in two processes, >> what happens to my virtual memory >> maps when I do MapViewOfFile naming that mapping object >> handle? Please limit your >> response to fewer than 100 words, which should be enough >> to explain the implications. >> >> Note if you knew the correct answer, you would not have >> made the ridiculous statements you >> have made about memory mapped files. And you would have >> known that they will improve >> multiprocess performance. > >Apparently within the context of virtual memory usage, ** >which I have shown in at least one instance does not apply. >Zero page faults indicates zero virtual memory usage right? *** Wrong. For reasons I have tried to explain any number of times! Do you even know what the term "virtual memory" means? You CANNOT avoid virtual memory usage in a Windows, Unix, or linux app. It is the ONLY REALITY that exists at application level. In Windows, it is the only reality that exists, period. Even the kernel runs entirely in virtual memory! **** > >** Do virtual memory maps apply to anything else besides >virtual memory? *** Why would they need to? That's what virtual memory is all about! **** > >> >> I repeat my assertion: you are clueless. Anyone who >> tries to use a generic wikipedia >> article as proof of being not clueless is metaclueless. >> joe >> *** >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 22 Mar 2010 21:17 See below... On Mon, 22 Mar 2010 16:59:34 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%23F2oLmgyKHA.5360(a)TK2MSFTNGP06.phx.gbl... >> Peter Olcott wrote: >> >>> Joe kept insisting and continues to insist that my data >>> is not resident in memory. >> >> >> If you have a 32 bit Windows OS, you are limited to just >> 2GB RAW ACCESS and 4GB of VIRTUAL MEMORY. > >Yes, and that is another thing. I kept saying that I have a >64bit OS, and Joe kept forming his replies in terms of a >32-bit OS. **** And how long did I keep saying "Unless you are running a WIn32 process in Win64" but you did not clarify that you were running on Win64. So in the absence of any explicit statement I had to assume you were running in Win32. **** > >> >> If your process is loading 4GB, you are using virtual >> memory. >> >>> After loading my data and waiting twelve hours the >>> process monitor reports zero page faults, when I execute >>> my process and run it to completion. >> >> >> You're lying, you told me you have PAGE FAULTS but it >> settle down to zero, which is NORMAL. But start a 2nd >> process and you will get page faults. > >I only get the page faults until the data is loaded. After >the data is loaded I get essentially no more page faults, >even after waiting twelve hours before running my process to >completion. After proving that my data is resident in RAM >Joe continues to chide me for claiming that my data is >resident in RAM. **** If you used a memory-mapped file correctly, yu would have very low-cost page faults because you would be mapping to existing pages. But you seem to not want to hear that memory-mapped files will improve performance, particularly in a multiple-process environment. joe **** > >You guys just playing head games with me? **** We are trying to help you, in spite of your best efforts to tell us we are wrong. You insist that simplistic experiments which gave you a single data point give you a basis for extrapolating an entire family of performance information, and we are saying "You don't KNOW until you've MEASURED" and you insist that measurement is not relevant because you MUST be right. All I'm saying is that you MIGHT be right, and once you do the measurements, you might find out that you are completely WRONG, which works to your advantage. So run the damn expeimet, already! joe **** > >> >> I also asked, now 5 times, to provide the MEMORY LOAD >> percentage which I even provided with a simple C program >> that you can compile, and you did not: >> >> // File: V:\bin\memload.cpp >> >> #include <stdio.h> >> #include <windows.h> >> >> void main(char argc, char *argv[]) >> { >> MEMORYSTATUS ms; >> ms.dwLength = sizeof(ms); >> GlobalMemoryStatus(&ms); >> printf("Memory Load: %d%%",ms.dwMemoryLoad); >> } >> >> Why can't you even do that? >> >>> How does this not prove Joe is wrong (At least in the >>> specific instance of one execution of my process)? >>> (1) The process monitor is lying. >>> (2) Page faults do not measure virtual memory usage. >> >> There are now what 4-5 participants in the thread who are >> telling your thinking is wrong and lack a understanding of >> the Windows and Intel hardware. >> >> lets get a few more like this guy with a somewhat layman >> description: >> >> http://blogs.sepago.de/helge/2008/01/09/windows-x64-all-the-same-yet-very-different-part-1/ >> >> and the #1 guy at Microsoft today! >> >> http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx >> >> If you DEFY what Mark Russinovich is saying here, you are >> CRAZY! >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 22 Mar 2010 21:22 On Mon, 22 Mar 2010 16:44:49 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:ifofq5hdvoesvnv8a6v70vi68fk8akcjld(a)4ax.com... >> see below... >> On Mon, 22 Mar 2010 12:15:27 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:uGVpjmdyKHA.5948(a)TK2MSFTNGP06.phx.gbl... >>>> Peter Olcott wrote: >>>> >>>>> Since my process (currently) requires unpredictable >>>>> access to far more memory than can fit into the largest >>>>> cache, I see no possible way that adding 1000-fold >>>>> slower >>>>> disk access could possibly speed things up. This seems >>>>> absurd to me. >>>> >>>> >>>> And I would agree it would be seem to be absurd to >>>> inexperience people. >>>> >>>> But you need to TRUST the power of your multi-processor >>>> computer because YOU are most definitely under utilizing >>>> it by a long shot. >>>> >>>> The code I posted is the proof! >>> >>>If it requires essentially nothing besides random access >>>to >>>entirely different places of 100 MB of memory, thenn (then >>>and only then) would it be reasonably representative of my >>>process. Nearly all the my process does is look up in >>>memory >>>the next place to look up in memory. >> *** >> Then the CORRECT approach is not to say "I don't want to >> waste time reading your example >> because it doesn't match my problem", the CORRECT approach >> is to say "I have no modified >> your example to more closely resemble my access patterns, >> ran it, and got THIS result" and >> show the results you got. >> joe > >I think that you must have paraphrased me incorrectly and I >don't see the context in this post. I would not likely have >concluded that it does not match my problem, and in fact >remember asking Hector if it did match my problem. I stated >my problem much more concisely than he stated his proof. **** So I misremembered >I did not examine the code because I did not want to spend >time looking at something that is not representative of my >process. ? ***** > >The single sentence is above. Hector never told me whether >or not he thought that it matched my algorithm's memory >usage pattern. > >> >>> >>>> >>>> Your issue is akin to having a pickup truck, overloading >>>> the back, piling things on each other, overweight beyond >>>> the recommended safety levels per specifications of the >>>> car manufacturer (and city/state ordinances), and now >>>> your >>>> driving, speed, vision of your truck are all altered. >>>> Your truck won't go as fast now and if even if you >>>> could, >>>> things can fall, people can die, crashes can happen. >>>> >>>> You have two choices: >>>> >>>> - You can stop and unload stuff and come back and >>>> pick >>>> it up on >>>> 2nd strip, your total travel time doubled. >>>> >>>> - you can get a 2nd pick up truck, split the load and >>>> get >>>> on a four lanes highway and drive side by side, >>>> sometimes >>>> one creeps ahead, and the other moves ahead, and >>>> both reach >>>> the destination at the near same expected time. >>>> >>>> Same thing! >>>> >>>> You are overloading your machine to the point it working >>>> very very hard to satisfy your single thread process >>>> needs. You may "believe" it is working at optimal >>>> speeds >>>> because it has uninterrupted exclusive access but it is >>>> not reality. You are under utilizing the power of your >>>> machine. >>>> >>>> Whether you realize it or not, the overloaded pickup >>>> truck >>>> is smart and is stopping you every X milliseconds >>>> checking >>>> if you have a 2nd pickup truck to offload some work and >>>> do >>>> some moving for you!! >>>> >>>> You need to change your thinking. >>>> >>>> However, at this point, I don't think you have any >>>> coding >>>> skills, because if you did, you would be EAGERLY JUMPING >>>> at the code I provided to see for yourself. >>>> >>>> -- >>>> HLS >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 22 Mar 2010 21:28 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OTB08HiyKHA.5360(a)TK2MSFTNGP06.phx.gbl... > Peter Olcott wrote: > >>> It says nothing about one execution; it says that under >>> certain conditions, paging is not >>> an issue. It does not say anything about using multiple >>> threads on multiple cores within >>> a single process. >> >> This is the earlier issue where you claimed that my >> thinking that I needed to have my data resident in RAM >> was absurd and based on ignorance. I do need to have my >> data resident in RAM and indeed my data is resident in >> RAM for extended periods, and there is no ignorance >> associated with this thinking. > > > But is ignorance because your PROCESS MEMORY is VIRTUAL > MEMORY! Virtual memory is essentially disk pretending to be RAM, when disk is not used (no page faults) then it is no longer disk pretending to be RAM. Even though some of the VM infrastructure remains in place and still operates, (requiring a tiny bit of overhead) the part that most significantly impacts performance is not functioning. Thus from a performance point of view VM is essentially not functioning. If you want to get nit picky and refrain from boiling things down to their essence you can say that VM is still operating. For all practical purposes from a pure performance point of view, VM is impacting performance negligibly, and thus can be construed as if it was not functioning. That is one example of the extraneous nit picky details that always boiling everything to its bare essence strips from further consideration. > > Look, for people who have PCs with 1GB or 2GB of RAM, the > PROCESS STILL GETS 4GB. > > Where is it "GHOST RAM" coming from? > > > -- > HLS
From: Joseph M. Newcomer on 22 Mar 2010 21:29
On Mon, 22 Mar 2010 06:28:36 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%23IrS0xYyKHA.2012(a)TK2MSFTNGP04.phx.gbl... >> Here is the result using a 1.5GB readonly memory mapped >> file. I started with 1 single process thread, then switch >> to 2 threads, then 4, 6, 8, 10 and 12 threads. Notice how >> the processing time for the earlier threads started high >> but decreased with the later thread. This was the caching >> effect of the readonly memory file. Also note the Global >> Memory Status *MEMORY LOAD* percentage. For my machine, it >> is at 19% at steady state. But as expected it shuts up >> when dealing with this large memory map file. I probably >> can fine tune the map views better, but they are set as >> read only. Well, I'll leave OP to figure out memory maps >> coding for his patented DFA meta file process. >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 0 >> --------------------------------------- >> Time: 2984 | Elapsed: 0 >> --------------------------------------- >> Total Client Time: 2984 >> >> V:\wc5beta>testpeter3t /s:3000000 /t:2 /r:1 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 0 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> * Wait For Thread Completion >> - Memory Load: 96% >> * Done >> --------------------------------------- >> 0 | Time: 5407 | Elapsed: 0 >> 1 | Time: 4938 | Elapsed: 0 >> --------------------------------------- >> Total Time: 10345 >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 /t:4 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 0 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> - Creating thread 2 >> - Creating thread 3 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> - Resuming thread# 2 in 334 msecs. >> - Resuming thread# 3 in 500 msecs. >> * Wait For Thread Completion >> - Memory Load: 97% >> * Done >> --------------------------------------- >> 0 | Time: 6313 | Elapsed: 0 >> 1 | Time: 5844 | Elapsed: 0 >> 2 | Time: 5500 | Elapsed: 0 >> 3 | Time: 5000 | Elapsed: 0 >> --------------------------------------- >> Total Time: 22657 >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 /t:6 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 0 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> - Creating thread 2 >> - Creating thread 3 >> - Creating thread 4 >> - Creating thread 5 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> - Resuming thread# 2 in 334 msecs. >> - Resuming thread# 3 in 500 msecs. >> - Resuming thread# 4 in 169 msecs. >> - Resuming thread# 5 in 724 msecs. >> * Wait For Thread Completion >> - Memory Load: 97% >> * Done >> --------------------------------------- >> 0 | Time: 6359 | Elapsed: 0 >> 1 | Time: 5891 | Elapsed: 0 >> 2 | Time: 5547 | Elapsed: 0 >> 3 | Time: 5047 | Elapsed: 0 >> 4 | Time: 4875 | Elapsed: 0 >> 5 | Time: 4141 | Elapsed: 0 >> --------------------------------------- >> Total Time: 31860 >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 /t:8 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 16 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> - Creating thread 2 >> - Creating thread 3 >> - Creating thread 4 >> - Creating thread 5 >> - Creating thread 6 >> - Creating thread 7 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> - Resuming thread# 2 in 334 msecs. >> - Resuming thread# 3 in 500 msecs. >> - Resuming thread# 4 in 169 msecs. >> - Resuming thread# 5 in 724 msecs. >> - Resuming thread# 6 in 478 msecs. >> - Resuming thread# 7 in 358 msecs. >> * Wait For Thread Completion >> - Memory Load: 96% >> * Done >> --------------------------------------- >> 0 | Time: 6203 | Elapsed: 0 >> 1 | Time: 5734 | Elapsed: 0 >> 2 | Time: 5391 | Elapsed: 0 >> 3 | Time: 4891 | Elapsed: 0 >> 4 | Time: 4719 | Elapsed: 0 >> 5 | Time: 3984 | Elapsed: 0 >> 6 | Time: 3500 | Elapsed: 0 >> 7 | Time: 3125 | Elapsed: 0 >> --------------------------------------- >> Total Time: 37547 >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 /t:10 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 0 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> - Creating thread 2 >> - Creating thread 3 >> - Creating thread 4 >> - Creating thread 5 >> - Creating thread 6 >> - Creating thread 7 >> - Creating thread 8 >> - Creating thread 9 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> - Resuming thread# 2 in 334 msecs. >> - Resuming thread# 3 in 500 msecs. >> - Resuming thread# 4 in 169 msecs. >> - Resuming thread# 5 in 724 msecs. >> - Resuming thread# 6 in 478 msecs. >> - Resuming thread# 7 in 358 msecs. >> - Resuming thread# 8 in 962 msecs. >> - Resuming thread# 9 in 464 msecs. >> * Wait For Thread Completion >> - Memory Load: 97% >> * Done >> --------------------------------------- >> 0 | Time: 7234 | Elapsed: 0 >> 1 | Time: 6766 | Elapsed: 0 >> 2 | Time: 6422 | Elapsed: 0 >> 3 | Time: 5922 | Elapsed: 0 >> 4 | Time: 5750 | Elapsed: 0 >> 5 | Time: 5016 | Elapsed: 0 >> 6 | Time: 4531 | Elapsed: 0 >> 7 | Time: 4125 | Elapsed: 0 >> 8 | Time: 3203 | Elapsed: 0 >> 9 | Time: 2703 | Elapsed: 0 >> --------------------------------------- >> Total Time: 51672 >> >> V:\wc5beta>testpeter3t /s:3000000 /r:1 /t:12 >> - size : 3000000 >> - memory : 1536000000 (1500000K) >> - repeat : 1 >> - Memory Load : 25% >> - Allocating Data .... 16 >> * Starting threads >> - Creating thread 0 >> - Creating thread 1 >> - Creating thread 2 >> - Creating thread 3 >> - Creating thread 4 >> - Creating thread 5 >> - Creating thread 6 >> - Creating thread 7 >> - Creating thread 8 >> - Creating thread 9 >> - Creating thread 10 >> - Creating thread 11 >> * Resuming threads >> - Resuming thread# 0 in 41 msecs. >> - Resuming thread# 1 in 467 msecs. >> - Resuming thread# 2 in 334 msecs. >> - Resuming thread# 3 in 500 msecs. >> - Resuming thread# 4 in 169 msecs. >> - Resuming thread# 5 in 724 msecs. >> - Resuming thread# 6 in 478 msecs. >> - Resuming thread# 7 in 358 msecs. >> - Resuming thread# 8 in 962 msecs. >> - Resuming thread# 9 in 464 msecs. >> - Resuming thread# 10 in 705 msecs. >> - Resuming thread# 11 in 145 msecs. >> * Wait For Thread Completion >> - Memory Load: 97% >> * Done >> --------------------------------------- >> 0 | Time: 7984 | Elapsed: 0 >> 1 | Time: 7515 | Elapsed: 0 >> 2 | Time: 7188 | Elapsed: 0 >> 3 | Time: 6672 | Elapsed: 0 >> 4 | Time: 6500 | Elapsed: 0 >> 5 | Time: 5781 | Elapsed: 0 >> 6 | Time: 5250 | Elapsed: 0 >> 7 | Time: 4953 | Elapsed: 0 >> 8 | Time: 3953 | Elapsed: 0 >> 9 | Time: 3484 | Elapsed: 0 >> 10 | Time: 2750 | Elapsed: 0 >> 11 | Time: 2547 | Elapsed: 0 >> --------------------------------------- >> Total Time: 64577 >> >> >> -- >> HLS > >OK and where is the summary conclusion? **** Read the data! Don't you see the obvious? If necessary, bring up Excel, plot the data in a graph. It should jump out at you. I fail to see how it challenges your anlytic abilities to infer what is going on here. **** >Also by using a memory mapped file your process would have >entirely different behavior than mine. **** Really? You KNOW THIS? I see nothing that would suggest this conclusion follows. **** > >I known that it is possible that you could have been right >all along about this, and I could be wrong. I know this >because of a term that I coined. [Ignorance Squared]. **** So why did you insist on telling us that our urging to run the experiment was wrong? Why do you insist on an unsubstantiated conclusion in the absence of real data? **** > >[Ignorance Squared] is the process by which a lack of >understanding is perceived by the one whom lacks this >understanding as disagreement. Whereas the one whom has >understanding knows that the ignorant person is lacking >understanding the ignorant person lacks this insight, and is >thus ignorant even of their own ignorance, hence the term >[Ignorance Squared] . > >Now that I have a way to empirically validate your theories >against mine (that I dreamed up last night while sleeping) I >will do this. > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |