Can extra processing threads help in this case?
in [MFC]
|
From: Liviu on 25 Mar 2010 21:57 "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote... > "Liviu" <lab2k1(a)gmail.c0m> wrote... >> >> You were replying to Pete Delgado, but threaded it under my post. > > His post got lost. It never showed up in my outlook express. Thought you said at some point that you were reading on google groups. http://groups.google.com/group/microsoft.public.vc.mfc/tree/browse_frm/thread/241391c82ac5925f/e8ed13e6e87823e8?hl=en&rnum=131&_done=%2Fgroup%2Fmicrosoft.public.vc.mfc%2Fbrowse_frm%2Fthread%2F241391c82ac5925f%3Fhl%3Den%26#doc_55f6afbb818386a7 >> You must realize of course that fault tolerance comes with a price, >> and transactional databases weigh heavily in terms of disk caching >> and virtual memory usage. If your naive idea that "my app is the >> only thing running and will command all RAM to itself" was wrong >> to begin with, on too many levels to count, yet it is even more wrong >> once you add a database to the picture. > > For all practical purposes this will still be true. The database load > will be negligible compared to the OCR processing load. Something > like 100K of data compared to 4 GB of data. Good luck with that. My advice would be to verify that assumption before you bet the house on it, but then I'd only be repeating what's been said too many times already. FWIW there was a thread on microsoft.public.win32.programmer.kernel not long ago "disable file system cache for a certain file". It had some useful insight and pointers among the replies. That was about windows, though, while your recent questions here have long gone off-topic and don't seem to be about windows or mfc at all. Liviu
From: Hector Santos on 25 Mar 2010 22:03 Peter Olcott wrote: >> I am glad I was able to prove to you how threads using >> shared process memory is better than your multiple single >> process redundant loading. (You are welcome). > > How many times do I have to tell you that I will not be > using multiple single process redundant loading before you > get it? Because you still haven't acknowledge you tested it with threads after the code that was provided to you and you keep talking about FOUR PROCESSORS being good enough and if need be scaling out and now with SINGLE CPU machines. >> But it also MMF was mostly based on you first wanted to >> use single processes. I suggested a SHARED MMF DLL file >> which will also both design models and also give you a >> fast startup time versus what you have now of X minutes to >> start up. > > X minutes of start up once. Not once a day or once a week, > just once. If you used an MMF that would be as fast it talks to open a file. >> But now you bring in disaster recover and wonder how it >> can be reliable. Now you can't avoid considering >> minimizing your caching and buffer I/O that benefits your >> speed. Now MMF plays a bigger role to give you a "mix" >> where tuning factors come into play. > > Pete pointed out that I am really only concerned with not > losing any data. That is correct. Also the only data that I > am concerned with losing is customer transaction data. > Everything else can be reconstructed easily enough. I don't > have to handle this myself, the database provider can handle > this for me. But you also said your SQL server will be your FIFO server. What will you do now? >> The things is MMF works really well if its READ ONLY huge >> data. If you plan to write data, then it helps because of >> caching and delayed commits under solid computer >> environments where power is reliable, but it raises the >> opportunity for lost when disaster occurs. > > I thought that you said in another posting the MMF enables > disaster recovery? You really haven't learn anything yet? Yes by virtue that its virtualize from disk so the only possible lost are the dirty pages that have not been flushed yet. If you minimize the dirty pages (flush every 30 seconds perhaps) then you lower your window of dirty pages being lost to disaster withing that window. The same idea as if you were in an EDITOR and it has an auto-save feature. Wonder why that is? Many applications employ a technique where a MMF hold all the global data and state information. This is one way to share data between processes. So you work with it just like you normally do and your lost is minimize. You can also reload this the file back into memory to restart where you last left off. The only disadvantage is using objects for data which requires you to include a serialization (archive) logic. -- HLS
From: Peter Olcott on 25 Mar 2010 22:17 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:ezfzJiIzKHA.404(a)TK2MSFTNGP02.phx.gbl... > > Peter Olcott wrote: > >>> I am glad I was able to prove to you how threads using >>> shared process memory is better than your multiple >>> single process redundant loading. (You are welcome). >> >> How many times do I have to tell you that I will not be >> using multiple single process redundant loading before >> you get it? > > > Because you still haven't acknowledge you tested it with > threads after the code that was provided to you and you > keep talking about FOUR PROCESSORS being good enough and > if need be scaling out and now with SINGLE CPU machines. I agree that it is probably a good idea without requiring testing. Since the multiple process passed, I assume that the multiple thread will pass. With my current business model this will probably be moot for years. > >>> But it also MMF was mostly based on you first wanted to >>> use single processes. I suggested a SHARED MMF DLL file >>> which will also both design models and also give you a >>> fast startup time versus what you have now of X minutes >>> to start up. >> >> X minutes of start up once. Not once a day or once a >> week, just once. > > > If you used an MMF that would be as fast it talks to open > a file. I am considering this now. I tested Linux again and was somewhat surprised that my process only gets a little less than 500MB of the 2 GB before paging kicks in. > >>> But now you bring in disaster recover and wonder how it >>> can be reliable. Now you can't avoid considering >>> minimizing your caching and buffer I/O that benefits >>> your speed. Now MMF plays a bigger role to give you a >>> "mix" where tuning factors come into play. >> >> Pete pointed out that I am really only concerned with not >> losing any data. That is correct. Also the only data that >> I am concerned with losing is customer transaction data. >> Everything else can be reconstructed easily enough. I >> don't have to handle this myself, the database provider >> can handle this for me. > > > But you also said your SQL server will be your FIFO > server. That is a preliminary idea subject to change. > > What will you do now? > >>> The things is MMF works really well if its READ ONLY >>> huge data. If you plan to write data, then it helps >>> because of caching and delayed commits under solid >>> computer environments where power is reliable, but it >>> raises the opportunity for lost when disaster occurs. >> >> I thought that you said in another posting the MMF >> enables disaster recovery? > > > You really haven't learn anything yet? > > Yes by virtue that its virtualize from disk so the only > possible lost are the dirty pages that have not been > flushed yet. If you minimize the dirty pages (flush every > 30 seconds perhaps) then you lower your window of dirty > pages being lost to disaster withing that window. The > same idea as if you were in an EDITOR and it has an > auto-save feature. Wonder why that is? > > Many applications employ a technique where a MMF hold all > the global data and state information. This is one way to > share data between processes. So you work with it just > like you normally do and your lost is minimize. You can > also reload this the file back into memory to restart > where you last left off. The only disadvantage is using > objects for data which requires you to include a > serialization (archive) logic. > > -- > HLS
From: Hector Santos on 25 Mar 2010 22:44 Of course, the real ultimate question is if you will finished by the time your patent expires. Ok, its been fun. -- Peter Olcott wrote: > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:ezfzJiIzKHA.404(a)TK2MSFTNGP02.phx.gbl... >> Peter Olcott wrote: >> >>>> I am glad I was able to prove to you how threads using >>>> shared process memory is better than your multiple >>>> single process redundant loading. (You are welcome). >>> How many times do I have to tell you that I will not be >>> using multiple single process redundant loading before >>> you get it? >> >> Because you still haven't acknowledge you tested it with >> threads after the code that was provided to you and you >> keep talking about FOUR PROCESSORS being good enough and >> if need be scaling out and now with SINGLE CPU machines. > > I agree that it is probably a good idea without requiring > testing. Since the multiple process passed, I assume that > the multiple thread will pass. With my current business > model this will probably be moot for years. > >>>> But it also MMF was mostly based on you first wanted to >>>> use single processes. I suggested a SHARED MMF DLL file >>>> which will also both design models and also give you a >>>> fast startup time versus what you have now of X minutes >>>> to start up. >>> X minutes of start up once. Not once a day or once a >>> week, just once. >> >> If you used an MMF that would be as fast it talks to open >> a file. > > I am considering this now. I tested Linux again and was > somewhat surprised that my process only gets a little less > than 500MB of the 2 GB before paging kicks in. > >>>> But now you bring in disaster recover and wonder how it >>>> can be reliable. Now you can't avoid considering >>>> minimizing your caching and buffer I/O that benefits >>>> your speed. Now MMF plays a bigger role to give you a >>>> "mix" where tuning factors come into play. >>> Pete pointed out that I am really only concerned with not >>> losing any data. That is correct. Also the only data that >>> I am concerned with losing is customer transaction data. >>> Everything else can be reconstructed easily enough. I >>> don't have to handle this myself, the database provider >>> can handle this for me. >> >> But you also said your SQL server will be your FIFO >> server. > > That is a preliminary idea subject to change. > >> What will you do now? >> >>>> The things is MMF works really well if its READ ONLY >>>> huge data. If you plan to write data, then it helps >>>> because of caching and delayed commits under solid >>>> computer environments where power is reliable, but it >>>> raises the opportunity for lost when disaster occurs. >>> I thought that you said in another posting the MMF >>> enables disaster recovery? >> >> You really haven't learn anything yet? >> >> Yes by virtue that its virtualize from disk so the only >> possible lost are the dirty pages that have not been >> flushed yet. If you minimize the dirty pages (flush every >> 30 seconds perhaps) then you lower your window of dirty >> pages being lost to disaster withing that window. The >> same idea as if you were in an EDITOR and it has an >> auto-save feature. Wonder why that is? >> >> Many applications employ a technique where a MMF hold all >> the global data and state information. This is one way to >> share data between processes. So you work with it just >> like you normally do and your lost is minimize. You can >> also reload this the file back into memory to restart >> where you last left off. The only disadvantage is using >> objects for data which requires you to include a >> serialization (archive) logic. >> >> -- >> HLS > > -- HLS
From: Peter Olcott on 25 Mar 2010 23:00
"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:eP3%2304IzKHA.5940(a)TK2MSFTNGP02.phx.gbl... > Of course, the real ultimate question is if you will > finished by the time your patent expires. The latest improvements would likely qualify for another patent, thus extending the term, plus there is a whole other invention of using this technology to provide an easy to use universal graphical user interface scripting language. > > Ok, its been fun. > > -- > > Peter Olcott wrote: > >> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >> message news:ezfzJiIzKHA.404(a)TK2MSFTNGP02.phx.gbl... >>> Peter Olcott wrote: >>> >>>>> I am glad I was able to prove to you how threads using >>>>> shared process memory is better than your multiple >>>>> single process redundant loading. (You are welcome). >>>> How many times do I have to tell you that I will not be >>>> using multiple single process redundant loading before >>>> you get it? >>> >>> Because you still haven't acknowledge you tested it with >>> threads after the code that was provided to you and you >>> keep talking about FOUR PROCESSORS being good enough and >>> if need be scaling out and now with SINGLE CPU machines. >> >> I agree that it is probably a good idea without requiring >> testing. Since the multiple process passed, I assume that >> the multiple thread will pass. With my current business >> model this will probably be moot for years. >> >>>>> But it also MMF was mostly based on you first wanted >>>>> to use single processes. I suggested a SHARED MMF DLL >>>>> file which will also both design models and also give >>>>> you a fast startup time versus what you have now of X >>>>> minutes to start up. >>>> X minutes of start up once. Not once a day or once a >>>> week, just once. >>> >>> If you used an MMF that would be as fast it talks to >>> open a file. >> >> I am considering this now. I tested Linux again and was >> somewhat surprised that my process only gets a little >> less than 500MB of the 2 GB before paging kicks in. >> >>>>> But now you bring in disaster recover and wonder how >>>>> it can be reliable. Now you can't avoid considering >>>>> minimizing your caching and buffer I/O that benefits >>>>> your speed. Now MMF plays a bigger role to give you a >>>>> "mix" where tuning factors come into play. >>>> Pete pointed out that I am really only concerned with >>>> not losing any data. That is correct. Also the only >>>> data that I am concerned with losing is customer >>>> transaction data. Everything else can be reconstructed >>>> easily enough. I don't have to handle this myself, the >>>> database provider can handle this for me. >>> >>> But you also said your SQL server will be your FIFO >>> server. >> >> That is a preliminary idea subject to change. >> >>> What will you do now? >>> >>>>> The things is MMF works really well if its READ ONLY >>>>> huge data. If you plan to write data, then it helps >>>>> because of caching and delayed commits under solid >>>>> computer environments where power is reliable, but it >>>>> raises the opportunity for lost when disaster occurs. >>>> I thought that you said in another posting the MMF >>>> enables disaster recovery? >>> >>> You really haven't learn anything yet? >>> >>> Yes by virtue that its virtualize from disk so the only >>> possible lost are the dirty pages that have not been >>> flushed yet. If you minimize the dirty pages (flush >>> every 30 seconds perhaps) then you lower your window of >>> dirty pages being lost to disaster withing that window. >>> The same idea as if you were in an EDITOR and it has an >>> auto-save feature. Wonder why that is? >>> >>> Many applications employ a technique where a MMF hold >>> all the global data and state information. This is one >>> way to share data between processes. So you work with it >>> just like you normally do and your lost is minimize. >>> You can also reload this the file back into memory to >>> restart where you last left off. The only disadvantage >>> is using objects for data which requires you to include >>> a serialization (archive) logic. >>> >>> -- >>> HLS >> >> > > > > -- > HLS |