Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 25 Mar 2010 00:39 SEe below... On Tue, 23 Mar 2010 15:53:36 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: >> Run a 2nd instance and you begin to see faults. You saw >> that. You proved that. You told is that. It is why this >> thread got started. > >Four instances of 1.5 GB RAM and zero page faults after the >data is loaded. > >You never know a man with a billion dollars in the bank just >might panic and sell all of his furniture just in case he >loses the billion dollars and won't be able to afford to pay >his electric bill. **** There are people who behave this way. Custodial care and psychoactive drugs (like lithium-based drugs) usually help them. SSRIs sometimes help (selective serotonin reuptake inhibitors). I don't know what an SSRI or lithium equivalent is for an app that becomes depressed. People who behave this way are a danger to themselves; I lost a friend who suicided over less than this, and we all shook our heads afterwards at how we missed the obvious-in-retrospect signs of his growing psychosis. and nobody in his family knew that he'd stopped his meds. And guess what: you run out of space and your processes will suicide, and worse still, they won't even understand why! **** > >If nothing else I will shut the VM off. **** In what wild fantasy world is this possible? You have obviously not been paying attentiont to ANYTHING we have been saying; you CAN'T TURN VIRTUAL MEMORY OFF! IT IS ON ALL THE TIME! Welcome to reality, where your bizarre ideas of what can and cannot be done are irrelevant. You can't shut it off, there is NO POSSIBLE WAY fo "shut the VM off" and your statement shows that you have achived a new level of cluelessness whose existence was previously unsuspected.What next? Powering your computer with orgonic force? Doing OCR using astrology or Ouija boards? Doing interprocess communication with MySQL? Your machine goes virtual before the Windows boot screen appears, and you can't make it NOT be virtual. The kernel would stop working. I tried to explain this, but you clearly were off on your own bizarre idea of what virtual memory is, and not even the Intel docs could persuade you that you are spouting total and complete nonsense! Orongic energy: http://www.orgonics.com/ (this was proven to be a complete fraud decades ago! But the web site is a hoot, as long as you realize it is complete rubbish!) Note: you can set the size of the paging file to 0, which means there will be no paging and you will essentially be limited by the amount of physical RAM; and random programs will fail for no explicable reason other than the fact that because the kernel could not page out something to make room for them, they ran out of memory. How robust is your code against failure induced this way? But the correct way to state this is "I will set the size of the paging file to zero", not "I will turn VM off". The first statement is implementable, the second one is one that tells your listener that you are totally and hopelessly clueless. These two different descriptions are not the same thing, at all, and are not isomorphic to renaming. You can't say one and mean something else. If you set the paging file size to zero, you have disabled paging, which does not disable virtual memory, it disables paging. VIrtual memory continues to exist, whether you want it to or not. It just happens that the sum of all virtual address spaces can't grow beyond physical memory. But every process is STILL running using virtual memory. Get a grip, and try to at least use the technical language correctly! Otherwise, expect people to mock you. And instead of a std::vector of std::vectors (given you have reimplemented std::vector) you could consider building a file that was entirely self-consistent internally. and just map it into your address space. But this would actually improve your performance, and you seem to not be interested in that if it doesn't involve fulfilling fantasies like "turning off the VM" joe **** > > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 25 Mar 2010 00:44 Make sure the addresses are completely independent of where the vector appears in memory. Given you have re-implemented std::vector (presumably as peter::vector) and you have done all the good engineering you claim, this shouldn't take very much time at all. Then you can use memory-mapped files, and share this massive footprint across multiple processes, so although you might have 1.5GB in each process, it is the SAME 1.5GB because every process SHARES that same data with every other process. Seriously, this is one of the exercises in my Systems Programming course; we do it Thursday afternoon. joe On Wed, 24 Mar 2010 10:09:00 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Oliver Regenfelder" <oliver.regenfelder(a)gmx.at> wrote in >message news:c9efd$4ba9f330$547743c7$24260(a)news.inode.at... >> Hello Peter, >> >> Peter Olcott wrote: >>> If I was going to do it for real it would be one large >>> single std::vector of std::vectors shared across multiple >>> threads. I really don't want to deal with Virtual Memory >>> issues at all. I want to do everything that I reasonably >>> can to make them all moot. This may eventually require a >>> real-time OS. >> >> So you are using a std::vector of std::vectors inside your >> application? >> >> Somehting like: >> >> std::vector<std::vector<int> > >> >> Best Regards, >> >> Oliver > >Yes this part of the new design. > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 25 Mar 2010 01:01 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:rdqlq5dv2u8bh308se0td53rk7lqmv0bki(a)4ax.com... > Make sure the addresses are completely independent of > where the vector appears in memory. > > Given you have re-implemented std::vector (presumably as > peter::vector) and you have done > all the good engineering you claim, this shouldn't take > very much time at all. Then you > can use memory-mapped files, and share this massive > footprint across multiple processes, > so although you might have 1.5GB in each process, it is > the SAME 1.5GB because every > process SHARES that same data with every other process. > > Seriously, this is one of the exercises in my Systems > Programming course; we do it > Thursday afternoon. > joe But all that this does is make page faults quicker right? Any page faults at can only degrade my performance. > > On Wed, 24 Mar 2010 10:09:00 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Oliver Regenfelder" <oliver.regenfelder(a)gmx.at> wrote in >>message >>news:c9efd$4ba9f330$547743c7$24260(a)news.inode.at... >>> Hello Peter, >>> >>> Peter Olcott wrote: >>>> If I was going to do it for real it would be one large >>>> single std::vector of std::vectors shared across >>>> multiple >>>> threads. I really don't want to deal with Virtual >>>> Memory >>>> issues at all. I want to do everything that I >>>> reasonably >>>> can to make them all moot. This may eventually require >>>> a >>>> real-time OS. >>> >>> So you are using a std::vector of std::vectors inside >>> your >>> application? >>> >>> Somehting like: >>> >>> std::vector<std::vector<int> > >>> >>> Best Regards, >>> >>> Oliver >> >>Yes this part of the new design. >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 25 Mar 2010 01:07 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:sjolq5dp8kkg42mubvr9kroebcimut3blo(a)4ax.com... > SEe below... > On Tue, 23 Mar 2010 15:53:36 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > > >>> Run a 2nd instance and you begin to see faults. You saw >>> that. You proved that. You told is that. It is why this >>> thread got started. >> >>Four instances of 1.5 GB RAM and zero page faults after >>the >>data is loaded. >> >>You never know a man with a billion dollars in the bank >>just >>might panic and sell all of his furniture just in case he >>loses the billion dollars and won't be able to afford to >>pay >>his electric bill. > **** > There are people who behave this way. Custodial care and > psychoactive drugs (like > lithium-based drugs) usually help them. SSRIs sometimes > help (selective serotonin > reuptake inhibitors). I don't know what an SSRI or > lithium equivalent is for an app that > becomes depressed. Ah so then paging out a process or its data when loads of RAM is still available is crazy right? > > People who behave this way are a danger to themselves; I > lost a friend who suicided over > less than this, and we all shook our heads afterwards at > how we missed the > obvious-in-retrospect signs of his growing psychosis. and > nobody in his family knew that > he'd stopped his meds. And guess what: you run out of > space and your processes will > suicide, and worse still, they won't even understand why! > **** >> >>If nothing else I will shut the VM off. > **** > In what wild fantasy world is this possible? You have > obviously not been paying > attentiont to ANYTHING we have been saying; you CAN'T TURN > VIRTUAL MEMORY OFF! IT IS ON > ALL THE TIME! Welcome to reality, where your bizarre > ideas of what can and cannot be > done are irrelevant. You can't shut it off, there is NO > POSSIBLE WAY fo "shut the VM off" > and your statement shows that you have achived a new level > of cluelessness whose existence > was previously unsuspected.What next? Powering your > computer with orgonic force? Doing > OCR using astrology or Ouija boards? Doing interprocess > communication with MySQL? > > Your machine goes virtual before the Windows boot screen > appears, and you can't make it > NOT be virtual. The kernel would stop working. I tried > to explain this, but you clearly > were off on your own bizarre idea of what virtual memory > is, and not even the Intel docs > could persuade you that you are spouting total and > complete nonsense! > > Orongic energy: http://www.orgonics.com/ (this was proven > to be a complete fraud decades > ago! But the web site is a hoot, as long as you realize it > is complete rubbish!) > > Note: you can set the size of the paging file to 0, which > means there will be no paging > and you will essentially be limited by the amount of > physical RAM; and random programs > will fail for no explicable reason other than the fact > that because the kernel could not > page out something to make room for them, they ran out of > memory. How robust is your code > against failure induced this way? But the correct way to > state this is "I will set the > size of the paging file to zero", not "I will turn VM > off". The first statement is > implementable, the second one is one that tells your > listener that you are totally and > hopelessly clueless. These two different descriptions are > not the same thing, at all, and > are not isomorphic to renaming. You can't say one and mean > something else. If you set the > paging file size to zero, you have disabled paging, which > does not disable virtual memory, > it disables paging. VIrtual memory continues to exist, > whether you want it to or not. It > just happens that the sum of all virtual address spaces > can't grow beyond physical memory. > But every process is STILL running using virtual memory. > Get a grip, and try to at least > use the technical language correctly! Otherwise, expect > people to mock you. > > And instead of a std::vector of std::vectors (given you > have reimplemented std::vector) > you could consider building a file that was entirely > self-consistent internally. and just > map it into your address space. But this would actually > improve your performance, and you > seem to not be interested in that if it doesn't involve > fulfilling fantasies like "turning > off the VM" > joe > > **** >> >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Ismo Salonen on 25 Mar 2010 04:52
Peter Olcott wrote: > "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in > message news:rdqlq5dv2u8bh308se0td53rk7lqmv0bki(a)4ax.com... >> Make sure the addresses are completely independent of >> where the vector appears in memory. >> >> Given you have re-implemented std::vector (presumably as >> peter::vector) and you have done >> all the good engineering you claim, this shouldn't take >> very much time at all. Then you >> can use memory-mapped files, and share this massive >> footprint across multiple processes, >> so although you might have 1.5GB in each process, it is >> the SAME 1.5GB because every >> process SHARES that same data with every other process. >> >> Seriously, this is one of the exercises in my Systems >> Programming course; we do it >> Thursday afternoon. >> joe > > But all that this does is make page faults quicker right? > Any page faults at can only degrade my performance. > just my two cents : memory mapped files are paged in to process memory when the page is 1st time referenced (read or written). This is pagefault mechanism. It is the most efficient way accessing data. Reading memory by ReadFile() or others in that family are slower (well ReadFile directly you buffers would result quite similar performance bause it use the paging mechanism). The pagefault mechanism must be quite optimized as it is the basis for modern operating systems ( windows, linux etc ). The inner workings are quite delicate and I highly suspect you can no way outperform it. It seems that you have plausible product idea but you lack the knowledge how operating system works. Have you read the articles others have referred to ? you should read the Windows Internals (5th edition) by Russinovich et al. to get the real data about how things work. (I've been programmer for living less time than Joe but still about 28 years, you should really trust him as he knows what he is talking about. I've also done more than just windows/linux/unix/vms/ias/rsx/msdos/OS9 stuff , also realtime embedded where 1ms was long time). Probably you could hire someone to help out the implementation stage ? Surely if the product is so good as you have described then using some money to get on to market faster is usually considered good idea. br ismo |