Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 22 Mar 2010 20:11 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:4u0gq5t7htjkq7paen7d8hvf8nl6pm82f8(a)4ax.com... > See below... > On Mon, 22 Mar 2010 16:27:48 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >>news:OIehRQgyKHA.4752(a)TK2MSFTNGP04.phx.gbl... >>> >>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >>> news:osWdnaGZ3q06RTrWnZ2dnUVZ_uudnZ2d(a)giganews.com... >>>> >>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>> message >>>> news:ecdfq5lb57qrou47d1ppaupsi6t2guu7nv(a)4ax.com... >>>>> **** >>>>> He has NO CLUE as to what a "memory-mapped file" >>>>> actually is. This last comment indicates >>>> >>>> http://en.wikipedia.org/wiki/Memory-mapped_file >>>> Apparently I do. >>> >>> I think you would be far better served by looking at >>> Windows specific information on memory mapped files such >>> as that which Joe suggested to you some time ago: >>> Richter's Programming Applications for Microsoft Windows >>> 4th. >>> >>> -Pete >>> >>> >> >>Joe kept insisting and continues to insist that my data is >>not resident in memory. > *** > I do not recall asserting that; I pointed out that Windows > pre-outpages unused pages and > marks the slots for reuse, but that is not what you claim > I stated. > **** >> >>After loading my data and waiting twelve hours the process >>monitor reports zero page faults, when I execute my >>process >>and run it to completion. > *** > That is useful data, but has nothing to do with the > multithreading question. IT only > demonstrates that a tiny number of pages had been moved > out (my recollection is you said 5 > page faults, not zero). > > **** >> >>How does this not prove Joe is wrong (At least in the >>specific instance of one execution of my process)? >>(1) The process monitor is lying. >>(2) Page faults do not measure virtual memory usage. >> > **** > It says nothing about one execution; it says that under > certain conditions, paging is not > an issue. It does not say anything about using multiple > threads on multiple cores within > a single process. This is the earlier issue where you claimed that my thinking that I needed to have my data resident in RAM was absurd and based on ignorance. I do need to have my data resident in RAM and indeed my data is resident in RAM for extended periods, and there is no ignorance associated with this thinking. > > You seem to think it does. > joe > **** > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 22 Mar 2010 20:18 Peter Olcott wrote: > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:exGTQtgyKHA.3884(a)TK2MSFTNGP06.phx.gbl... >> Peter Olcott wrote: >> No. If its zero or not changing and I know your process is >> not, it means that your process working set is not >> demanding more than it can handle or OTHER processes have >> not chewed up memory, limiting your available memory. > > OK so zero page faults does not mean that virtual memory is > not being used? > (1) YES zero page faults means that virtual memory is not > active on this process > (2) Not (YES zero page faults means that virtual memory is > not active on this process) > > Which is it (1) or (2) ??? Any hem hawing will be taken > as intentional deceit None of the above: Your process AT THAT MOMENT does not need PAGE anything because it was already in your WORKING SET. Look, YOU SIMPLE PROGRAM IS ALWAYS USING VIRTUALIZE MEMORY! ALWAYS! Please answer these questions: Did you try the memory load program? // File: V:\bin\memload.cpp #include <stdio.h> #include <windows.h> void main(char argc, char *argv[]) { MEMORYSTATUS ms; ms.dwLength = sizeof(ms); GlobalMemoryStatus(&ms); printf("Memory Load: %d%%",ms.dwMemoryLoad); } Did you compile your application to use 64BIT? Did you convert all your 32 BIT variables to 64BIT? Did you read the expert Mark R? Pushing the Limits of Windows: Physical Memory http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx Pushing the Limits of Windows: Virtual Memory http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx Pushing the Limits of Windows: Paged and Nonpaged Pool http://blogs.technet.com/markrussinovich/archive/2009/03/26/3211216.aspx -- HLS
From: Hector Santos on 22 Mar 2010 20:20 Peter Olcott wrote: >> And I suggested waaaaaaaay back in the beginning of this >> thread. :) I even gave him a link for a sweet >> CMemoryMapFile class at MSDN! > > I have proven that this is moot, and this proof continues to > be ignored. What proof? You provided no proof of anything whatsoever, not even the existence of your OCR program. -- HLS
From: Joseph M. Newcomer on 22 Mar 2010 20:19 See below... On Mon, 22 Mar 2010 18:46:24 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:exGTQtgyKHA.3884(a)TK2MSFTNGP06.phx.gbl... >> Peter Olcott wrote: >> >>> You tell me all about pages faults, yet the process >>> monitor >>> reports zero page faults, and you continue to claim that >>> its >>> all about page faults, and virtual memory. >> >> >> Its not a claim - its a fact. >> >>> Pages faults indicate victual memory usage right? >> >> >> It shows when your PROCESS is asking too much the can >> provide to you all in memory - it has to virtualize it. >> >>> A lack of page faults indicates a lack of virtual memory >>> usage right? >> >> >> No. If its zero or not changing and I know your process is >> not, it means that your process working set is not >> demanding more than it can handle or OTHER processes have >> not chewed up memory, limiting your available memory. > >OK so zero page faults does not mean that virtual memory is >not being used? *** OF COURSE virtiual memory is being used; there is NO OTHER KIND OF MEMORY for a process. What it means is that all of the virtual memory has remained resident, something that before you published this results was not something that was demonstrable. You have demonstrated that it is not being paged out and the pages reused, at least under your test scenario. *** >(1) YES zero page faults means that virtual memory is not >active on this process **** But given that there is only virtual memory, you cannot assert that zero page faults mean it is not being used, only that the virtual pages have remained in memory, a useful piece of knowledge. And if you had a clue about memory-mapped files, this would tell you that using a named, shared segment would improver performance of multiple processes using MMF to get the data in, and it might also mean you wouldn't see the several-minute startup transient. You certainly wouldn't see it on the second or higher processes. Perhaps you can explain whay you man by "virtual memory is not active". Alas, for your way of forming the question, virtual memory is ALWAYS active, and that implies the potential for TLB thrashing. Do you know what a TLB is, the role it serves, and how it is managed? Of what TLB thrashing might be? To simplify the task, I will tell you that TLB stands for Translation Lookaside Buffer. The rest is up to you. If you didn't know what at TLB is when you asked the question, it proves that you had no clue about why the question as stated is nonsense. **** >(2) Not (YES zero page faults means that virtual memory is >not active on this process) **** Virtual memory HAS To be "active" because there isn't any other kind of memory available to a process. This is inherent in every operating system. **** > >Which is it (1) or (2) ??? Any hem hawing will be taken >as intentional deceit **** THe question is ill-formed because the terms are being used incorrectly and in some cases in ways that suggest that the answer could be "there is no virtual memory being used" which is a nonsensical statement. There is ONLY virtual memory being used, it just happens that it is not paged out or the pageouts retain their in-memory images so any page faults are "soft" (meaning the data does not have to be read off the disk, because it can be found already in memory, in an un-reused page). You can't hem or haw an ill-formed question, whether the goal is or is not to be deceitful. he question makes no sense as stated and the two alternative answers are both nonsensical. It is almost but not quite as bad as the "have you stopped beating your wife?" style questions, for which any answer that is "yes" or "no" is damning. So I will state that the question is ill-formed and the two options presented as answers are nonsensical, and that is the complete TRUTH. There is no reason to be deceitful here, you asked a question for which the correct answer is: "This demonstrates that Windows retains pages in memory when there is no need to page them out and reuse the page frames" and that is the ONLY truthful and correct answer. Unfortunately, it is not either of the nonsensical alteratives you allow. if you don't undetstand why the question is nonsensical as stated and both alternative answers are nonsensical, you are demonstrating that you really, truly are clueless about how operating systems work. **** > >> >> You have NO control over this UNLESS you explicitly told >> windows to use NO-CACHING, NO BUFFER I/O memory. >> >> Why do you refuse to believe this? Every Windows >> programmer has to know this to some degree. >> >>> How does this not prove that my data is in memory and >>> thus you are wrong when you say that my data is not >>> resident in memory? >> >> >> It doesn't. >> >> Allow a Technical Fellow Engineer at Microsoft explain it >> all to you: >> >> Pushing the Limits of Windows: Physical Memory >> http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx >> >> Pushing the Limits of Windows: Virtual Memory >> http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx >> >> Pushing the Limits of Windows: Paged and Nonpaged Pool >> http://blogs.technet.com/markrussinovich/archive/2009/03/26/3211216.aspx >> >> Are you going to argue with Mark too? >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 22 Mar 2010 20:38
Joseph M. Newcomer wrote: > See below... > On Mon, 22 Mar 2010 16:27:48 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >> "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >> news:OIehRQgyKHA.4752(a)TK2MSFTNGP04.phx.gbl... >>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >>> news:osWdnaGZ3q06RTrWnZ2dnUVZ_uudnZ2d(a)giganews.com... >>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>> message >>>> news:ecdfq5lb57qrou47d1ppaupsi6t2guu7nv(a)4ax.com... >>>>> **** >>>>> He has NO CLUE as to what a "memory-mapped file" >>>>> actually is. This last comment indicates >>>> http://en.wikipedia.org/wiki/Memory-mapped_file >>>> Apparently I do. >>> I think you would be far better served by looking at >>> Windows specific information on memory mapped files such >>> as that which Joe suggested to you some time ago: >>> Richter's Programming Applications for Microsoft Windows >>> 4th. >>> >>> -Pete >>> >>> >> Joe kept insisting and continues to insist that my data is >> not resident in memory. > *** > I do not recall asserting that; I pointed out that Windows pre-outpages unused pages and > marks the slots for reuse, but that is not what you claim I stated. > **** >> After loading my data and waiting twelve hours the process >> monitor reports zero page faults, when I execute my process >> and run it to completion. > *** > That is useful data, but has nothing to do with the multithreading question. IT only > demonstrates that a tiny number of pages had been moved out (my recollection is you said 5 > page faults, not zero). > > **** >> How does this not prove Joe is wrong (At least in the >> specific instance of one execution of my process)? >> (1) The process monitor is lying. >> (2) Page faults do not measure virtual memory usage. >> > **** > It says nothing about one execution; it says that under certain conditions, paging is not > an issue. It does not say anything about using multiple threads on multiple cores within > a single process. > > You seem to think it does. > joe Joe, he changes his statements to suit whatever it is he is trying to prove but can not. I honestly don't think he knows anything Microsoft C/C++, MFC, WIN32 and what he has a OCR compiled code was probably taken from open source and it COMPILED the first time. He has not product, no demo, nothing to show he has anything and even if he does have software copied from a DFA C example code book, he doesn't know anything about optimizing it. All he knows he need 4GB and thought that getting 8GB would be good enough to run at least two instances each with redundant 4GB memory allocations with no lost in speed. But he is finding out otherwise, that is how all this started. So he is wondering and found out the Lx Chip caching didn't help. He concluded that he has an special patented OCR application that exhausted the physical capabilities of the computer, so special, its only possible to run once. Yet, he refuses, and I can only presume its because he doesn't know how, to explore the simulator and doesn't grasp any of the technical writeups and links provided regarding virtual memory and multi-threads operations. It defies logic. -- HLS |