How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Joseph M. Newcomer on 19 Mar 2010 12:14 But note that at this point you have done measurement. Then you go on to say that you believe it SHOULD work in a certain way based on some phiosophical principles that you had no evidence to support. Now you have the evidence. But you can't make statements about how things *should* work, you can only measure how they *do* work. And that is the *only* truth. Opinions of how they *ought* to work are actually pretty meaningless, since there is no reason to believe that is how they *do* work, until you have done actual measurements to validate your opinion, it remains only unsubstantiated opinion. Yet you still insist that unsubstantiated opinion *must* be true. WHat you can say now is that you have real data that substantiates your view, and that's real. Measurement is the only truth. Opinions in the absence of real measurment simply don't matter in the slightest. YOu can safely assert "I think it should work like this" but that's not the same as saying "because I *think* it ought to behave this way, it necessarily MUST behave this way", which is the position you have been taking. joe On Fri, 19 Mar 2010 10:12:43 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:71u5q59q4mi9k4uc4nmv7t6tkp0rdc7r7t(a)4ax.com... >> from my linux expert: linux uses a "two timer" approach. >> When the firt timer fires, the >> code runs thru the page list and notes which pages haven't >> changed. Any page that hasn't >> changed is marked as a candidate for page-out. any page >> that has changed is removed from >> the page-out list. When the second timer fires, pages >> that have been marked are paged >> out. If a page is used before it is paged out, because >> the first timer is still running, >> it is removed from the candidate list for page-out. Pages >> which are paged out are >> candidates for re-use. If the page frame is re-used for >> another page, the page will have >> to be brought back into memory; otherwise, it takes a >> "soft" page fault and the page fault >> finds the page which still happens to be in memory and >> does the appropriate fixup of the >> page table. So it is a hybrid of eager page-out and page >> retention. >> joe >> > >I extended your half hour test to 10 hours and nothing was >paged out on Windows 7 with 8.0 GB RAM. > >To page out any processes with an extra 4.0 GB available >would be like a guy with a million dollars in his bank >account selling some of his furniture just in case he might >need more money. > >All of your ideas and the ideas that you have presented from >others are good ideas, but, probably should not apply when >there are large amounts of excess RAM available. > >> On Thu, 18 Mar 2010 11:40:43 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:OLClPfrxKHA.404(a)TK2MSFTNGP02.phx.gbl... >>>> Joe, Peter, is it just a habit, but you both have to >>>> quote >>>> the entire message just to embed a few lines of response >>>> in the middle of the message? It becomes a hunting >>>> game! >>>> Look for my replies below. >>>> >>> >>>(1) Its less trouble for me to do it this way. >>>(2) When I look for replies I use page-down, not >>>scroll-down, so finding the reply is very quick. >>>(3) It retains the entire context of every reply, thus >>>easily verifying who is at fault in communication errors. >>>(4) Text takes very little room or bandwidth, so the cost >>>of >>>these resources is more than compensated by the savings in >>>human labor. >>> >>>> Honestly, the NNTP Server would work more efficiently. >>>> :) >>>> >>>> Peter Olcott wrote: >>>> >>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>> message >>>>> news:f9h4q5ltbdi70eot2n5me08gij5vg556bc(a)4ax.com... >>>>>> See below... >>>>>> On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" >>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>> >>>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>>>> message >>>>>>> news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>>>>>>> See below... >>>>>>>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>>> >>>>>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote >>>>>>>>> in >>>>>>>>> message >>>>>>>>> news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>>>>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>>>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>>>>> >>>>>>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote >>>>>>>>>>> in >>>>>>>>>>> message >>>>>>>>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>>>>>>> Peter Olcott wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Fastest speed is highest priority, secondary to >>>>>>>>>>>>> this >>>>>>>>>>>>> is >>>>>>>>>>>>> portability, it looks like FastCGI provides >>>>>>>>>>>>> both. >>>>>>>>>>>>> Another very high priority is minimal >>>>>>>>>>>>> development >>>>>>>>>>>>> costs. >>>>>>>>>>>>> As far as I can tell FastCGI is good here too. >>>>>>>>>>>> Dude, FastCGI is not a "Language." Its an idea, >>>>>>>>>>>> a >>>>>>>>>>>> methodology that was designed with EARLY needs >>>>>>>>>>>> to >>>>>>>>>>>> speed >>>>>>>>>>>> up >>>>>>>>>>>> loading time. You won't have that problem >>>>>>>>>>>> today. >>>>>>>>>>> You continue to ignore my repeatedly stated >>>>>>>>>>> requirement >>>>>>>>>>> of >>>>>>>>>>> 4.0 GB of data (a deterministic finite automaton) >>>>>>>>>>> that >>>>>>>>>>> must >>>>>>>>>>> be loaded before execution can begin. If you >>>>>>>>>>> ignore >>>>>>>>>>> my >>>>>>>>>>> fundamental requirements you can't possibly >>>>>>>>>>> provide >>>>>>>>>>> good >>>>>>>>>>> advice. >>>>>>>>>> *** >>>>>>>>>> And you continue to ignore our observations that >>>>>>>>>> there >>>>>>>>>> are >>>>>>>>>> no technologies that guarantee >>>>>>>>>> this, and the timings of program behavior can be >>>>>>>>>> swamped >>>>>>>>>> by communication times (did I >>>>>>>>>> tell you about the 70-second DNS resolution time? >>>>>>>>>> Did >>>>>>>>>> you >>>>>>>>>> know that you can expect delays >>>>>>>>>> in integer number of seconds to locate your >>>>>>>>>> server?) >>>>>>>>>> >>>>>>>>> That is horrendous. I doubt if it is frequent, and >>>>>>>>> local >>>>>>>>> caching would hopefully prevent it from recurring. >>>>>>>> **** >>>>>>>> Depends. Did the end user configure for local DNS >>>>>>>> caching? WHat was the cache retention >>>>>>>> time set by the end user? Note, again, YOU HAVE NO >>>>>>>> CONTROL OVER THESE PARAMETERS! >>>>>>>> **** >>>>>>>>>> You fasten on some technology that says it is >>>>>>>>>> "faster" >>>>>>>>>> and >>>>>>>>>> assume it is the answer to your >>>>>>>>>> problem; you toss jargon around in ways that those >>>>>>>>>> of us >>>>>>>>>> who know the technology don't >>>>>>>>>> even recognize, such as assertions how FastCGI >>>>>>>>>> allows >>>>>>>>>> you >>>>>>>>>> to have browse buttons in the >>>> >>>> sure >>>> >>>>> It seems to me that most of you guys continue to >>>> >>>>> simply >>>> >>>>>>>>> ignore some of my most fundamental design >>>>>>>>> requirements. >>>>>>>>> App >>>>>>>>> load time for me can be several minutes, app >>>>>>>>> execution >>>>>>>>> time >>>>>>>>> of a large job is never more than 100 milliseconds. >>>>>>>> ***** >>>>>>>> No, we hear them, we just don't believe that FastCGI >>>>>>>> is >>>>>>>> going to be the solution that will >>>>>>>> solve every issue for you. Hector says that CGI has >>>>>>>> been >>>>>>>> improved to where it has >>>>>>>> comparable performance to FastCGI. I have no >>>>>>>> information >>>>>>>> to confirm or deny that >>>>>>>> statement. Again, you are not a server expert, so >>>>>>>> you >>>>>>>> should pay attention to the server >>>>>>>> expert. >>>>>>>> **** >>>>>>>>> If the app is the only process besides the OS, by >>>>>>>>> limiting >>>>>>>>> memory usage to no more than 75% of available RAM >>>>>>>>> in >>>>>>>>> actual >>>>>>>>> practice both the app and its data are paged out >>>>>>>>> almost >>>>>>>>> not >>>>>>>>> at all. On a real-time OS this behavior would be >>>>>>>>> guaranteed. >>>>>>>>> On MS Windows it is close enough. Linux probably >>>>>>>>> does >>>>>>>>> not >>>>>>>>> do >>>>>>>>> worse than Windows in this regard. >>>>>>>> **** >>>>>>>> I have no idea why you think this is true. This is >>>>>>>> not an >>>>>>>> assumption I would have >>>>>>>> considered valid until I talked to a linux expert >>>>>>>> (since >>>>>>>> you will be hosted on linux). Yet >>>>>>>> you make this assertion simply because you THINK it >>>>>>>> should >>>>>>>> work, as opposed to getting >>>>>>>> data that confirms that the OS actually works this >>>>>>>> way. >>>>>>>> **** >>>> >>>> whatever >>>> >>>>>>> An OS that unloads any part of an executing process >>>>>>> when it >>>>>>> has no need for additional memory would be an idiotic >>>>>>> OS. It >>>>>>> would be flat out stupid to do this. I am not >>>>>>> convinced >>>>>>> that >>>>>>> either Windows or Linux is this stupid. >>>>>> *** >>>>>> But we learned DECADES ago that "eager page-out" >>>>>> improves overall system performance, >>>>>> because it reduces startup transients when a process >>>>>> (or >>>>>> one of its threads) becomes >>>>>> schedulable. Again, the "working set" has to be paged >>>>>> in, and this is easier if there are >>>>>> available page frames in memory. So what you are >>>>>> calling "stupid" is in fact the >>>>>> DEMONSTRATED INTELLIGENT way to build operating >>>>>> systems, >>>>>> and we've known this since they >>>>>> early 1970s. This is why I have never known a system >>>>>> that did not page out an idle >>>>>> process IN ANITICIPATION OF NEED. Windows does it; >>>>>> I'm >>>>>> trying to find out if linux does >>>>>> it, but again, your opinion of how you THINK an >>>>>> operating system SHOULD work flies in the >>>>>> fact of reality. Just because you don't see a reason >>>>>> doesn't mean there isn't one, and >>>>>> one that has been established by long practice (the >>>>>> "actual practice" you fantasize about >>>>>> doesn't exist). Windows indeed works this way, and so >>>>>> have other operating systems, all >>>>>> based on BEST practice (not someone's fantasized >>>>>> "actual >>>>>> practice"). So I would not >>>>>> predicate actual behavior on fantasies, but learn the >>>>>> facts. >>>>>> ***** >>>>> >>>>> I will simply do whatever it takes to achieve the >>>>> functional result that I require. If there is always an >>>>> extra 4.0 GB of RAM available, then your whole >>>>> anticipation of need concept should become moot. >>>>> >>>>>>> If this same OS unloads part of a process because the >>>>>>> process has been idle for a long time, AND needs more >>>>>>> memory >>>>>>> to do some housekeeping task, then this case I could >>>>>>> see a >>>>>>> reason that would not be stupid. >>>>>> **** >>>>>> It is the antiipation-of-need which in actual practice >>>>>> (not fantasized "actual practive", >>>>>> but ACTUAL practice) has been found to be the best >>>>>> solution for general purpose operating >>>>>> systems, to maximize overall performance. Until you >>>>>> have measured (there's that nasty >>>>>> phrase, implying the obtaining of facts) behavior, you >>>>>> have no reason to assume it >>>>>> corresponds to your fantasies. >>>>>> **** >>>>>>>>>> local Web server, and how it is the perfect answer >>>>>>>>>> to >>>>>>>>>> all >>>>>>>>>> your problems, when we know it >>>>>>>>>> isn't anything but another hack whose purpose is >>>>>>>>>> to >>>>>>>>>> compensate for bad initiali designs >>>>>>>>>> (CGI requiring a new process per request). You >>>>>>>>>> ignore >>>>>>>>>> core issues such as how you are >>>>>>>>>> going to port your app so it runs on a linux or >>>>>>>>>> Unix >>>>>>>>>> server, and how you are going to >>>>>>>>> I am rewriting it to require very little OS >>>>>>>>> specific >>>>>>>>> code. >>>>>>>>> About the only thing that I will have to change is >>>>>>>>> the way >>>>>>>>> that PNG files are read into memory. I will have to >>>>>>>>> switch >>>>>>>>> to LibPNG. All of the MFC code is only development >>>>>>>>> code >>>>>>>>> and >>>>>>>>> will not be part of production. >>>>>>>> **** >>>>>>>> PNG files would not be "read into memory"; instead, >>>>>>>> you >>>>>>>> will need to convert the input >>>>>>>> string (however it is provided, most likely via >>>>>>>> stdin) >>>>>>>> to >>>>>>>> a bitmap, and that requires a >>>>>>>> library that does not read the PNG data from a file >>>>>>>> but >>>>>>>> from a text stream. >>>>>>>> **** >>>>>>> YES, or if I have to write is to disk and read it >>>>>>> back >>>>>>> in >>>>>>> again. Reading and writing 2K files will not kill me. >>>>>>> >>>>>>>>>> rewrite it to be platform-independent and possibly >>>>>>>>>> even >>>>>>>>>> GUI-free, and your unrealistic >>>>>>>>> The web service needs no GUI. It is all PNG in, and >>>>>>>>> UTF-8 >>>>>>>>> out. >>>>>>>> **** >>>>>>>> OK, that's good. You *will* have to write the >>>>>>>> FastCGI >>>>>>>> interface and make sure it works >>>>>>>> multithreaded so it can handle multiple requests. >>>>>>> I am going with Hectors idea of a tiny embedded >>>>>>> webserver >>>>>>> built in to my code. There are several freeware >>>>>>> one's >>>>>>> one >>>>>>> of these looks good enogh for my needs: mongoose. >>>>>>> >>>>>>>> **** >>>>>>>>>> expectations about how process memory is managed, >>>>>>>>>> and >>>>>>>>>> how >>>>>>>>>> every OS is magically going to >>>>>>>>>> behave in the way you wish, without any effort on >>>>>>>>>> your >>>>>>>>>> part. ANd you refuse to listen >>>>>>>>>> when people tell you that you don't understand >>>>>>>>>> either >>>>>>>>>> the >>>>>>>>>> technology or the intrinsic >>>>>>>>>> behavior of how an operating system works. >>>>>>>>>> >>>>>>>>>> You really need to do better research on these >>>>>>>>>> technologies before you start making such >>>>>>>>>> assertions, or ask questions and pay attention to >>>>>>>>>> the >>>>>>>>>> answers. I'm not even a Web server >>>>>>>>>> expert like Hector, and I can tell you are making >>>>>>>>>> nonsensical statements. Listen to him. >>>>>>>>>> He sounds like someone who does this for a living, >>>>>>>>>> and >>>>>>>>>> you >>>>>>>>>> should trust what he is telling >>>>>>>>>> you. >>>>>>>>>> joe >>>>>>>>> All that I know for sure is that some of my >>>>>>>>> fundamental >>>>>>>>> requirements are immutable. Some of Hector's advice >>>>>>>>> seemed >>>>>>>>> to continue to ignore these requirements. I was >>>>>>>>> envisioning >>>>>>>>> the web service as a single application. This may >>>>>>>>> not >>>>>>>>> have >>>>>>>>> been the way that he was envisioning it. >>>>>>>> **** >>>>>>>> You can do it by writing a "front end" that exports >>>>>>>> a >>>>>>>> known port and implements your >>>>>>>> appliction-specific protocol to get the data to send >>>>>>>> to >>>>>>>> your "business logic" component. >>>>>>>> It doesn't solve the paging problem, or the 4GB >>>>>>>> problem. >>>>>>>> joe >>>>>>> I think that I will be going with some sort of >>>>>>> application >>>>>>> specific webserver such as mongoose. Hector tried to >>>>>>> sell me >>>>>>> on his webserver, but, it was too costly and not >>>>>>> flexible >>>>>>> enough for me right now. In the future when I scale >>>>>>> up >>>>>>> to >>>>>>> multiple servers this may change. >>>>>>> >>>>>>>> **** >>>>>>>>>>>> Its nonsense to think this is "answer" to >>>>>>>>>>>> your problem. In addition, if you worry about >>>>>>>>>>>> processing >>>>>>>>>>>> speed, native C/C++ is your choice, so all you >>>>>>>>>>>> are >>>>>>>>>>>> doing >>>>>>>>>>>> is writing a CGI application, a console >>>>>>>>>>>> application >>>>>>>>>>>> that >>>>>>>>>>>> reads standard input. >>>>>>>>>>>> >>>>>>>>>>>> If you want productivity, learn PHP. Simple, >>>>>>>>>>>> well >>>>>>>>>>>> supported. Then if you really find out that >>>>>>>>>>>> FASTCGI >>>>>>>>>>>> is >>>>>>>>>>>> needed for your slow OS and machine, PHP >>>>>>>>>>>> supports >>>>>>>>>>>> it. >>>>>>>>>>>> >>>>>>>>>>>> The point is your SOLUTION is not FASTCGI, it is >>>>>>>>>>>> the >>>>>>>>>>>> actual application that does the work. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> HLS >>>>>>>>>> Joseph M. Newcomer [MVP] >>>>>>>>>> email: newcomer(a)flounder.com >>>>>>>>>> Web: http://www.flounder.com >>>>>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>>>>> Joseph M. Newcomer [MVP] >>>>>>>> email: newcomer(a)flounder.com >>>>>>>> Web: http://www.flounder.com >>>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>>> Joseph M. Newcomer [MVP] >>>>>> email: newcomer(a)flounder.com >>>>>> Web: http://www.flounder.com >>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> HLS >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 19 Mar 2010 12:22 Peter Olcott wrote: > I extended your half hour test to 10 hours and nothing was > paged out on Windows 7 with 8.0 GB RAM. How do you know? What measurement counters did you use? And what is the application doing? -- HLS
From: Peter Olcott on 19 Mar 2010 12:56 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:kc87q5lfipll54ancr4go7tlmlgfi4jknp(a)4ax.com... > But note that at this point you have done measurement. > Then you go on to say that you > believe it SHOULD work in a certain way based on some > phiosophical principles that you had I have never seen it false, not once in 26 years, and I took your simple advice and tested it after ten hours, nothing was paged out. So I can re-iterate that with plenty of excess RAM this should not be a problem, and on Windows 7 with 8.0 GB of RAM and 4.0 GB of excess RAM it is NOT a problem. > no evidence to support. Now you have the evidence. But > you can't make statements about > how things *should* work, you can only measure how they > *do* work. And that is the *only* > truth. Opinions of how they *ought* to work are actually > pretty meaningless, since there > is no reason to believe that is how they *do* work, until > you have done actual > measurements to validate your opinion, it remains only > unsubstantiated opinion. Yet you > still insist that unsubstantiated opinion *must* be true. > WHat you can say now is that > you have real data that substantiates your view, and > that's real. Measurement is the only > truth. Opinions in the absence of real measurment simply > don't matter in the slightest. > YOu can safely assert "I think it should work like this" > but that's not the same as saying > "because I *think* it ought to behave this way, it > necessarily MUST behave this way", > which is the position you have been taking. > joe > > On Fri, 19 Mar 2010 10:12:43 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:71u5q59q4mi9k4uc4nmv7t6tkp0rdc7r7t(a)4ax.com... >>> from my linux expert: linux uses a "two timer" approach. >>> When the firt timer fires, the >>> code runs thru the page list and notes which pages >>> haven't >>> changed. Any page that hasn't >>> changed is marked as a candidate for page-out. any page >>> that has changed is removed from >>> the page-out list. When the second timer fires, pages >>> that have been marked are paged >>> out. If a page is used before it is paged out, because >>> the first timer is still running, >>> it is removed from the candidate list for page-out. >>> Pages >>> which are paged out are >>> candidates for re-use. If the page frame is re-used for >>> another page, the page will have >>> to be brought back into memory; otherwise, it takes a >>> "soft" page fault and the page fault >>> finds the page which still happens to be in memory and >>> does the appropriate fixup of the >>> page table. So it is a hybrid of eager page-out and >>> page >>> retention. >>> joe >>> >> >>I extended your half hour test to 10 hours and nothing was >>paged out on Windows 7 with 8.0 GB RAM. >> >>To page out any processes with an extra 4.0 GB available >>would be like a guy with a million dollars in his bank >>account selling some of his furniture just in case he >>might >>need more money. >> >>All of your ideas and the ideas that you have presented >>from >>others are good ideas, but, probably should not apply when >>there are large amounts of excess RAM available. >> >>> On Thu, 18 Mar 2010 11:40:43 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>message >>>>news:OLClPfrxKHA.404(a)TK2MSFTNGP02.phx.gbl... >>>>> Joe, Peter, is it just a habit, but you both have to >>>>> quote >>>>> the entire message just to embed a few lines of >>>>> response >>>>> in the middle of the message? It becomes a hunting >>>>> game! >>>>> Look for my replies below. >>>>> >>>> >>>>(1) Its less trouble for me to do it this way. >>>>(2) When I look for replies I use page-down, not >>>>scroll-down, so finding the reply is very quick. >>>>(3) It retains the entire context of every reply, thus >>>>easily verifying who is at fault in communication >>>>errors. >>>>(4) Text takes very little room or bandwidth, so the >>>>cost >>>>of >>>>these resources is more than compensated by the savings >>>>in >>>>human labor. >>>> >>>>> Honestly, the NNTP Server would work more efficiently. >>>>> :) >>>>> >>>>> Peter Olcott wrote: >>>>> >>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>>> message >>>>>> news:f9h4q5ltbdi70eot2n5me08gij5vg556bc(a)4ax.com... >>>>>>> See below... >>>>>>> On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" >>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>> >>>>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote >>>>>>>> in >>>>>>>> message >>>>>>>> news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>>>>>>>> See below... >>>>>>>>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>>>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>>>> >>>>>>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> >>>>>>>>>> wrote >>>>>>>>>> in >>>>>>>>>> message >>>>>>>>>> news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>>>>>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter >>>>>>>>>>> Olcott" >>>>>>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> >>>>>>>>>>>> wrote >>>>>>>>>>>> in >>>>>>>>>>>> message >>>>>>>>>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>>>>>>>> Peter Olcott wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Fastest speed is highest priority, secondary >>>>>>>>>>>>>> to >>>>>>>>>>>>>> this >>>>>>>>>>>>>> is >>>>>>>>>>>>>> portability, it looks like FastCGI provides >>>>>>>>>>>>>> both. >>>>>>>>>>>>>> Another very high priority is minimal >>>>>>>>>>>>>> development >>>>>>>>>>>>>> costs. >>>>>>>>>>>>>> As far as I can tell FastCGI is good here >>>>>>>>>>>>>> too. >>>>>>>>>>>>> Dude, FastCGI is not a "Language." Its an >>>>>>>>>>>>> idea, >>>>>>>>>>>>> a >>>>>>>>>>>>> methodology that was designed with EARLY needs >>>>>>>>>>>>> to >>>>>>>>>>>>> speed >>>>>>>>>>>>> up >>>>>>>>>>>>> loading time. You won't have that problem >>>>>>>>>>>>> today. >>>>>>>>>>>> You continue to ignore my repeatedly stated >>>>>>>>>>>> requirement >>>>>>>>>>>> of >>>>>>>>>>>> 4.0 GB of data (a deterministic finite >>>>>>>>>>>> automaton) >>>>>>>>>>>> that >>>>>>>>>>>> must >>>>>>>>>>>> be loaded before execution can begin. If you >>>>>>>>>>>> ignore >>>>>>>>>>>> my >>>>>>>>>>>> fundamental requirements you can't possibly >>>>>>>>>>>> provide >>>>>>>>>>>> good >>>>>>>>>>>> advice. >>>>>>>>>>> *** >>>>>>>>>>> And you continue to ignore our observations that >>>>>>>>>>> there >>>>>>>>>>> are >>>>>>>>>>> no technologies that guarantee >>>>>>>>>>> this, and the timings of program behavior can be >>>>>>>>>>> swamped >>>>>>>>>>> by communication times (did I >>>>>>>>>>> tell you about the 70-second DNS resolution >>>>>>>>>>> time? >>>>>>>>>>> Did >>>>>>>>>>> you >>>>>>>>>>> know that you can expect delays >>>>>>>>>>> in integer number of seconds to locate your >>>>>>>>>>> server?) >>>>>>>>>>> >>>>>>>>>> That is horrendous. I doubt if it is frequent, >>>>>>>>>> and >>>>>>>>>> local >>>>>>>>>> caching would hopefully prevent it from >>>>>>>>>> recurring. >>>>>>>>> **** >>>>>>>>> Depends. Did the end user configure for local DNS >>>>>>>>> caching? WHat was the cache retention >>>>>>>>> time set by the end user? Note, again, YOU HAVE >>>>>>>>> NO >>>>>>>>> CONTROL OVER THESE PARAMETERS! >>>>>>>>> **** >>>>>>>>>>> You fasten on some technology that says it is >>>>>>>>>>> "faster" >>>>>>>>>>> and >>>>>>>>>>> assume it is the answer to your >>>>>>>>>>> problem; you toss jargon around in ways that >>>>>>>>>>> those >>>>>>>>>>> of us >>>>>>>>>>> who know the technology don't >>>>>>>>>>> even recognize, such as assertions how FastCGI >>>>>>>>>>> allows >>>>>>>>>>> you >>>>>>>>>>> to have browse buttons in the >>>>> >>>>> sure >>>>> >>>>> It seems to me that most of you guys continue to >>>>> >>>>> simply >>>>> >>>>>>>>>> ignore some of my most fundamental design >>>>>>>>>> requirements. >>>>>>>>>> App >>>>>>>>>> load time for me can be several minutes, app >>>>>>>>>> execution >>>>>>>>>> time >>>>>>>>>> of a large job is never more than 100 >>>>>>>>>> milliseconds. >>>>>>>>> ***** >>>>>>>>> No, we hear them, we just don't believe that >>>>>>>>> FastCGI >>>>>>>>> is >>>>>>>>> going to be the solution that will >>>>>>>>> solve every issue for you. Hector says that CGI >>>>>>>>> has >>>>>>>>> been >>>>>>>>> improved to where it has >>>>>>>>> comparable performance to FastCGI. I have no >>>>>>>>> information >>>>>>>>> to confirm or deny that >>>>>>>>> statement. Again, you are not a server expert, so >>>>>>>>> you >>>>>>>>> should pay attention to the server >>>>>>>>> expert. >>>>>>>>> **** >>>>>>>>>> If the app is the only process besides the OS, by >>>>>>>>>> limiting >>>>>>>>>> memory usage to no more than 75% of available RAM >>>>>>>>>> in >>>>>>>>>> actual >>>>>>>>>> practice both the app and its data are paged out >>>>>>>>>> almost >>>>>>>>>> not >>>>>>>>>> at all. On a real-time OS this behavior would be >>>>>>>>>> guaranteed. >>>>>>>>>> On MS Windows it is close enough. Linux probably >>>>>>>>>> does >>>>>>>>>> not >>>>>>>>>> do >>>>>>>>>> worse than Windows in this regard. >>>>>>>>> **** >>>>>>>>> I have no idea why you think this is true. This >>>>>>>>> is >>>>>>>>> not an >>>>>>>>> assumption I would have >>>>>>>>> considered valid until I talked to a linux expert >>>>>>>>> (since >>>>>>>>> you will be hosted on linux). Yet >>>>>>>>> you make this assertion simply because you THINK >>>>>>>>> it >>>>>>>>> should >>>>>>>>> work, as opposed to getting >>>>>>>>> data that confirms that the OS actually works this >>>>>>>>> way. >>>>>>>>> **** >>>>> >>>>> whatever >>>>> >>>>>>>> An OS that unloads any part of an executing process >>>>>>>> when it >>>>>>>> has no need for additional memory would be an >>>>>>>> idiotic >>>>>>>> OS. It >>>>>>>> would be flat out stupid to do this. I am not >>>>>>>> convinced >>>>>>>> that >>>>>>>> either Windows or Linux is this stupid. >>>>>>> *** >>>>>>> But we learned DECADES ago that "eager page-out" >>>>>>> improves overall system performance, >>>>>>> because it reduces startup transients when a process >>>>>>> (or >>>>>>> one of its threads) becomes >>>>>>> schedulable. Again, the "working set" has to be >>>>>>> paged >>>>>>> in, and this is easier if there are >>>>>>> available page frames in memory. So what you are >>>>>>> calling "stupid" is in fact the >>>>>>> DEMONSTRATED INTELLIGENT way to build operating >>>>>>> systems, >>>>>>> and we've known this since they >>>>>>> early 1970s. This is why I have never known a >>>>>>> system >>>>>>> that did not page out an idle >>>>>>> process IN ANITICIPATION OF NEED. Windows does it; >>>>>>> I'm >>>>>>> trying to find out if linux does >>>>>>> it, but again, your opinion of how you THINK an >>>>>>> operating system SHOULD work flies in the >>>>>>> fact of reality. Just because you don't see a >>>>>>> reason >>>>>>> doesn't mean there isn't one, and >>>>>>> one that has been established by long practice (the >>>>>>> "actual practice" you fantasize about >>>>>>> doesn't exist). Windows indeed works this way, and >>>>>>> so >>>>>>> have other operating systems, all >>>>>>> based on BEST practice (not someone's fantasized >>>>>>> "actual >>>>>>> practice"). So I would not >>>>>>> predicate actual behavior on fantasies, but learn >>>>>>> the >>>>>>> facts. >>>>>>> ***** >>>>>> >>>>>> I will simply do whatever it takes to achieve the >>>>>> functional result that I require. If there is always >>>>>> an >>>>>> extra 4.0 GB of RAM available, then your whole >>>>>> anticipation of need concept should become moot. >>>>>> >>>>>>>> If this same OS unloads part of a process because >>>>>>>> the >>>>>>>> process has been idle for a long time, AND needs >>>>>>>> more >>>>>>>> memory >>>>>>>> to do some housekeeping task, then this case I >>>>>>>> could >>>>>>>> see a >>>>>>>> reason that would not be stupid. >>>>>>> **** >>>>>>> It is the antiipation-of-need which in actual >>>>>>> practice >>>>>>> (not fantasized "actual practive", >>>>>>> but ACTUAL practice) has been found to be the best >>>>>>> solution for general purpose operating >>>>>>> systems, to maximize overall performance. Until you >>>>>>> have measured (there's that nasty >>>>>>> phrase, implying the obtaining of facts) behavior, >>>>>>> you >>>>>>> have no reason to assume it >>>>>>> corresponds to your fantasies. >>>>>>> **** >>>>>>>>>>> local Web server, and how it is the perfect >>>>>>>>>>> answer >>>>>>>>>>> to >>>>>>>>>>> all >>>>>>>>>>> your problems, when we know it >>>>>>>>>>> isn't anything but another hack whose purpose is >>>>>>>>>>> to >>>>>>>>>>> compensate for bad initiali designs >>>>>>>>>>> (CGI requiring a new process per request). You >>>>>>>>>>> ignore >>>>>>>>>>> core issues such as how you are >>>>>>>>>>> going to port your app so it runs on a linux or >>>>>>>>>>> Unix >>>>>>>>>>> server, and how you are going to >>>>>>>>>> I am rewriting it to require very little OS >>>>>>>>>> specific >>>>>>>>>> code. >>>>>>>>>> About the only thing that I will have to change >>>>>>>>>> is >>>>>>>>>> the way >>>>>>>>>> that PNG files are read into memory. I will have >>>>>>>>>> to >>>>>>>>>> switch >>>>>>>>>> to LibPNG. All of the MFC code is only >>>>>>>>>> development >>>>>>>>>> code >>>>>>>>>> and >>>>>>>>>> will not be part of production. >>>>>>>>> **** >>>>>>>>> PNG files would not be "read into memory"; >>>>>>>>> instead, >>>>>>>>> you >>>>>>>>> will need to convert the input >>>>>>>>> string (however it is provided, most likely via >>>>>>>>> stdin) >>>>>>>>> to >>>>>>>>> a bitmap, and that requires a >>>>>>>>> library that does not read the PNG data from a >>>>>>>>> file >>>>>>>>> but >>>>>>>>> from a text stream. >>>>>>>>> **** >>>>>>>> YES, or if I have to write is to disk and read it >>>>>>>> back >>>>>>>> in >>>>>>>> again. Reading and writing 2K files will not kill >>>>>>>> me. >>>>>>>> >>>>>>>>>>> rewrite it to be platform-independent and >>>>>>>>>>> possibly >>>>>>>>>>> even >>>>>>>>>>> GUI-free, and your unrealistic >>>>>>>>>> The web service needs no GUI. It is all PNG in, >>>>>>>>>> and >>>>>>>>>> UTF-8 >>>>>>>>>> out. >>>>>>>>> **** >>>>>>>>> OK, that's good. You *will* have to write the >>>>>>>>> FastCGI >>>>>>>>> interface and make sure it works >>>>>>>>> multithreaded so it can handle multiple requests. >>>>>>>> I am going with Hectors idea of a tiny embedded >>>>>>>> webserver >>>>>>>> built in to my code. There are several freeware >>>>>>>> one's >>>>>>>> one >>>>>>>> of these looks good enogh for my needs: mongoose. >>>>>>>> >>>>>>>>> **** >>>>>>>>>>> expectations about how process memory is >>>>>>>>>>> managed, >>>>>>>>>>> and >>>>>>>>>>> how >>>>>>>>>>> every OS is magically going to >>>>>>>>>>> behave in the way you wish, without any effort >>>>>>>>>>> on >>>>>>>>>>> your >>>>>>>>>>> part. ANd you refuse to listen >>>>>>>>>>> when people tell you that you don't understand >>>>>>>>>>> either >>>>>>>>>>> the >>>>>>>>>>> technology or the intrinsic >>>>>>>>>>> behavior of how an operating system works. >>>>>>>>>>> >>>>>>>>>>> You really need to do better research on these >>>>>>>>>>> technologies before you start making such >>>>>>>>>>> assertions, or ask questions and pay attention >>>>>>>>>>> to >>>>>>>>>>> the >>>>>>>>>>> answers. I'm not even a Web server >>>>>>>>>>> expert like Hector, and I can tell you are >>>>>>>>>>> making >>>>>>>>>>> nonsensical statements. Listen to him. >>>>>>>>>>> He sounds like someone who does this for a >>>>>>>>>>> living, >>>>>>>>>>> and >>>>>>>>>>> you >>>>>>>>>>> should trust what he is telling >>>>>>>>>>> you. >>>>>>>>>>> joe >>>>>>>>>> All that I know for sure is that some of my >>>>>>>>>> fundamental >>>>>>>>>> requirements are immutable. Some of Hector's >>>>>>>>>> advice >>>>>>>>>> seemed >>>>>>>>>> to continue to ignore these requirements. I was >>>>>>>>>> envisioning >>>>>>>>>> the web service as a single application. This may >>>>>>>>>> not >>>>>>>>>> have >>>>>>>>>> been the way that he was envisioning it. >>>>>>>>> **** >>>>>>>>> You can do it by writing a "front end" that >>>>>>>>> exports >>>>>>>>> a >>>>>>>>> known port and implements your >>>>>>>>> appliction-specific protocol to get the data to >>>>>>>>> send >>>>>>>>> to >>>>>>>>> your "business logic" component. >>>>>>>>> It doesn't solve the paging problem, or the 4GB >>>>>>>>> problem. >>>>>>>>> joe >>>>>>>> I think that I will be going with some sort of >>>>>>>> application >>>>>>>> specific webserver such as mongoose. Hector tried >>>>>>>> to >>>>>>>> sell me >>>>>>>> on his webserver, but, it was too costly and not >>>>>>>> flexible >>>>>>>> enough for me right now. In the future when I scale >>>>>>>> up >>>>>>>> to >>>>>>>> multiple servers this may change. >>>>>>>> >>>>>>>>> **** >>>>>>>>>>>>> Its nonsense to think this is "answer" to >>>>>>>>>>>>> your problem. In addition, if you worry about >>>>>>>>>>>>> processing >>>>>>>>>>>>> speed, native C/C++ is your choice, so all you >>>>>>>>>>>>> are >>>>>>>>>>>>> doing >>>>>>>>>>>>> is writing a CGI application, a console >>>>>>>>>>>>> application >>>>>>>>>>>>> that >>>>>>>>>>>>> reads standard input. >>>>>>>>>>>>> >>>>>>>>>>>>> If you want productivity, learn PHP. Simple, >>>>>>>>>>>>> well >>>>>>>>>>>>> supported. Then if you really find out that >>>>>>>>>>>>> FASTCGI >>>>>>>>>>>>> is >>>>>>>>>>>>> needed for your slow OS and machine, PHP >>>>>>>>>>>>> supports >>>>>>>>>>>>> it. >>>>>>>>>>>>> >>>>>>>>>>>>> The point is your SOLUTION is not FASTCGI, it >>>>>>>>>>>>> is >>>>>>>>>>>>> the >>>>>>>>>>>>> actual application that does the work. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> HLS >>>>>>>>>>> Joseph M. Newcomer [MVP] >>>>>>>>>>> email: newcomer(a)flounder.com >>>>>>>>>>> Web: http://www.flounder.com >>>>>>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>>>>>> Joseph M. Newcomer [MVP] >>>>>>>>> email: newcomer(a)flounder.com >>>>>>>>> Web: http://www.flounder.com >>>>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>>>> Joseph M. Newcomer [MVP] >>>>>>> email: newcomer(a)flounder.com >>>>>>> Web: http://www.flounder.com >>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> HLS >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 19 Mar 2010 13:02 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:ea82ZB4xKHA.3304(a)TK2MSFTNGP06.phx.gbl... > > Peter Olcott wrote: > > >> I extended your half hour test to 10 hours and nothing >> was paged out on Windows 7 with 8.0 GB RAM. > > > How do you know? What measurement counters did you use? > And what is the application doing? > > -- > HLS I had windows report all of its paging, and the number did not grow when I re-invoked the DFA process again after ten idle hours. Also (weirdly enough) the performance was a fraction of a percent faster after ten hours of idleness. Execution time went from 3.75 minutes to 3.73 minutes.
From: Hector Santos on 19 Mar 2010 13:13
Peter Olcott wrote: > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:ea82ZB4xKHA.3304(a)TK2MSFTNGP06.phx.gbl... >> Peter Olcott wrote: >> >> >>> I extended your half hour test to 10 hours and nothing >>> was paged out on Windows 7 with 8.0 GB RAM. >> >> How do you know? What measurement counters did you use? >> And what is the application doing? >> >> -- >> HLS > > I had windows report all of its paging, and the number did > not grow when I re-invoked the DFA process again after ten > idle hours. Also (weirdly enough) the performance was a > fraction of a percent faster after ten hours of idleness. > Execution time went from 3.75 minutes to 3.73 minutes. What were the following counters? - page fault - context switching rate - working set I'm more interested in the page faults. -- HLS |