How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Stephen Myers on 18 Mar 2010 10:13 Peter Olcott wrote: > "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in > message news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>> message news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>> message >>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>> >>>>>>> Fastest speed is highest priority, secondary to this >>>>>>> is >>>>>>> portability, it looks like FastCGI provides both. >>>>>>> Another very high priority is minimal development >>>>>>> costs. >>>>>>> As far as I can tell FastCGI is good here too. >>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>> methodology that was designed with EARLY needs to >>>>>> speed >>>>>> up >>>>>> loading time. You won't have that problem today. >>>>> You continue to ignore my repeatedly stated requirement >>>>> of >>>>> 4.0 GB of data (a deterministic finite automaton) that >>>>> must >>>>> be loaded before execution can begin. If you ignore my >>>>> fundamental requirements you can't possibly provide good >>>>> advice. >>>> *** >>>> And you continue to ignore our observations that there >>>> are >>>> no technologies that guarantee >>>> this, and the timings of program behavior can be swamped >>>> by communication times (did I >>>> tell you about the 70-second DNS resolution time? Did >>>> you >>>> know that you can expect delays >>>> in integer number of seconds to locate your server?) >>>> >>> That is horrendous. I doubt if it is frequent, and local >>> caching would hopefully prevent it from recurring. >> **** >> Depends. Did the end user configure for local DNS >> caching? WHat was the cache retention >> time set by the end user? Note, again, YOU HAVE NO >> CONTROL OVER THESE PARAMETERS! >> **** >>>> You fasten on some technology that says it is "faster" >>>> and >>>> assume it is the answer to your >>>> problem; you toss jargon around in ways that those of us >>>> who know the technology don't >>>> even recognize, such as assertions how FastCGI allows >>>> you >>>> to have browse buttons in the >>> It seems to me that most of you guys continue to simply >>> ignore some of my most fundamental design requirements. >>> App >>> load time for me can be several minutes, app execution >>> time >>> of a large job is never more than 100 milliseconds. >> ***** >> No, we hear them, we just don't believe that FastCGI is >> going to be the solution that will >> solve every issue for you. Hector says that CGI has been >> improved to where it has >> comparable performance to FastCGI. I have no information >> to confirm or deny that >> statement. Again, you are not a server expert, so you >> should pay attention to the server >> expert. >> **** >>> If the app is the only process besides the OS, by limiting >>> memory usage to no more than 75% of available RAM in >>> actual >>> practice both the app and its data are paged out almost >>> not >>> at all. On a real-time OS this behavior would be >>> guaranteed. >>> On MS Windows it is close enough. Linux probably does not >>> do >>> worse than Windows in this regard. >> **** >> I have no idea why you think this is true. This is not an >> assumption I would have >> considered valid until I talked to a linux expert (since >> you will be hosted on linux). Yet >> you make this assertion simply because you THINK it should >> work, as opposed to getting >> data that confirms that the OS actually works this way. >> **** > > An OS that unloads any part of an executing process when it > has no need for additional memory would be an idiotic OS. It > would be flat out stupid to do this. I am not convinced that > either Windows or Linux is this stupid. > > If this same OS unloads part of a process because the > process has been idle for a long time, AND needs more memory > to do some housekeeping task, then this case I could see a > reason that would not be stupid. > Why should any OS keep all of your process in memory at all? The concept of paged virtual memory allows the OS to keep what's currently needed in memory. This allows 10 or 20 or 500 processes to have the illusion that their 2GB is resident whenever they're running. Consider a program that puts up a logo from a resource at startup. Should the OS keep that snippet (page) of memory in physical memory on the off chance that it will be needed again? >>>> local Web server, and how it is the perfect answer to >>>> all >>>> your problems, when we know it >>>> isn't anything but another hack whose purpose is to >>>> compensate for bad initiali designs >>>> (CGI requiring a new process per request). You ignore >>>> core issues such as how you are >>>> going to port your app so it runs on a linux or Unix >>>> server, and how you are going to >>> I am rewriting it to require very little OS specific code. >>> About the only thing that I will have to change is the way >>> that PNG files are read into memory. I will have to switch >>> to LibPNG. All of the MFC code is only development code >>> and >>> will not be part of production. >> **** >> PNG files would not be "read into memory"; instead, you >> will need to convert the input >> string (however it is provided, most likely via stdin) to >> a bitmap, and that requires a >> library that does not read the PNG data from a file but >> from a text stream. >> **** > > YES, or if I have to write is to disk and read it back in > again. Reading and writing 2K files will not kill me. > >>>> rewrite it to be platform-independent and possibly even >>>> GUI-free, and your unrealistic >>> The web service needs no GUI. It is all PNG in, and UTF-8 >>> out. >> **** >> OK, that's good. You *will* have to write the FastCGI >> interface and make sure it works >> multithreaded so it can handle multiple requests. > > I am going with Hectors idea of a tiny embedded webserver > built in to my code. There are several freeware one's one > of these looks good enogh for my needs: mongoose. > >> **** >>>> expectations about how process memory is managed, and >>>> how >>>> every OS is magically going to >>>> behave in the way you wish, without any effort on your >>>> part. ANd you refuse to listen >>>> when people tell you that you don't understand either >>>> the >>>> technology or the intrinsic >>>> behavior of how an operating system works. >>>> >>>> You really need to do better research on these >>>> technologies before you start making such >>>> assertions, or ask questions and pay attention to the >>>> answers. I'm not even a Web server >>>> expert like Hector, and I can tell you are making >>>> nonsensical statements. Listen to him. >>>> He sounds like someone who does this for a living, and >>>> you >>>> should trust what he is telling >>>> you. >>>> joe >>> All that I know for sure is that some of my fundamental >>> requirements are immutable. Some of Hector's advice seemed >>> to continue to ignore these requirements. I was >>> envisioning >>> the web service as a single application. This may not have >>> been the way that he was envisioning it. >> **** >> You can do it by writing a "front end" that exports a >> known port and implements your >> appliction-specific protocol to get the data to send to >> your "business logic" component. >> It doesn't solve the paging problem, or the 4GB problem. >> joe > > I think that I will be going with some sort of application > specific webserver such as mongoose. Hector tried to sell me > on his webserver, but, it was too costly and not flexible > enough for me right now. In the future when I scale up to > multiple servers this may change. > >> **** >>>>>> Its nonsense to think this is "answer" to >>>>>> your problem. In addition, if you worry about >>>>>> processing >>>>>> speed, native C/C++ is your choice, so all you are >>>>>> doing >>>>>> is writing a CGI application, a console application >>>>>> that >>>>>> reads standard input. >>>>>> >>>>>> If you want productivity, learn PHP. Simple, well >>>>>> supported. Then if you really find out that FASTCGI >>>>>> is >>>>>> needed for your slow OS and machine, PHP supports it. >>>>>> >>>>>> The point is your SOLUTION is not FASTCGI, it is the >>>>>> actual application that does the work. >>>>>> >>>>>> -- >>>>>> HLS >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > >
From: Joseph M. Newcomer on 18 Mar 2010 11:19 See below... On Wed, 17 Mar 2010 18:18:55 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:47i2q5tch6oigbaqllh9lk0942mp5pnkt0(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:54:10 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:08i0q513cr5paa3m0mkvlmaoecd9vu7oqu(a)4ax.com... >>>> See below... >>>> On Tue, 16 Mar 2010 17:28:45 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> >>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>message >>>>>news:ea%23kdFVxKHA.5576(a)TK2MSFTNGP05.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>>> >>>>>>> I would be doing FastCGI in C++, if I do it. I am >>>>>>> considering FastCGI because I need a completely >>>>>>> portable >>>>>>> way for a user to browse for a PNG file to send to my >>>>>>> server. HTML has this built in, and it is connected >>>>>>> to >>>>>>> CGI. As far as a protocol goes, this would be >>>>>>> trivial. >>>>>>> The user sends a 24-bit PNG file, the server sends >>>>>>> back >>>>>>> UTF-8. That's the whole protocol. Sometimes the UTF-8 >>>>>>> is >>>>>>> the PNG file translated into text, and other times it >>>>>>> is >>>>>>> an error response. >>>>>> >>>>>> Doesn't sound you understand, or at least are showing >>>>>> you >>>>>> don't, what FastCGI is. >>>>>> >>>>>> Forget fastcgi, what you need is a dedicated http >>>>>> server >>>>>> with the built-in procssing or a http server that >>>>>> supports >>>>>> spawning your cgi EXE application as a special URL. >>>>> >>>>>I can't take the several minutes of CGI process spawn >>>>>time >>>>>for my application, (it has to load gigabytes of data >>>>>into >>>>>memory before it begins) thus conventional CGI is >>>>>infeasible. >>>> *** >>>> So, in other words, if I happen to be so unfrotunate as >>>> to >>>> make a request on your server >>>> and there IS no instance of the process, that in fact, I >>>> cannot expect 500ms reponse time >>>> but instead can expect MINUTES of delay! >>>> >>>> NOte that all FASTCGI promises is that your process does >>>> not exist or require a restart >>>> once it has been launched; it does not promise the pages >>>> will be "held in memory". >>>> joe >>> >>>In actual practice operating systems typically do not page >>>out code or data unless they need room for something else. >>>This will be the only application on the dedicated server, >>>thus little reason to be paged out. If necessary I will >>>run >>>a very tiny little background process that forces the app >>>to >>>do something once in a while. >> *** >> In acutal practice, they DO page out out code or data >> "just in case", in fact, Windows has >> done this for years. So don't cite fictitious "actual >> practice" as your rationale. I'd >> ask a linux expert what linux does before presuming I knew >> what it did. In the history of >> operating systems, I've work on four (including WIndows) >> that paged out idle process pages >> on the general principle that if the process isn't being >> used, it shouldn't be taking up >> any space. So I'm curious where you learned aobut "actual >> practice" and why you believe >> this must be true. Unix was one of the three systems I >> used that did this, and VMS was >> another IBM's TSS/360 was the fourth. so my "actual >> practice" on real operating systems >> says the opposite of your concept of "actual practice", >> and I'm curious what OS you used >> that worked this way. And why you think you can >> generalize from one instance of one OS to >> believe that all operating systems must follow this >> pattern. >> joe > >Like I already said (in a message that you have not gotten >to yet) I can always force it to add up all of its pixels >every once in a while. For example whenever it is otherwise >idle. This would require read access to every memory >location, thus keeping pages paged in. **** Only if you set the working set limits to not force it to page its own pages out! (You do know about working set, right? I've explained it in the past) joe > >> **** >>> >> >>>> >>>> ***** >>>>> >>>>>> >>>>>> You will handle standard INPUT as the encoded data. >>>>>> Simple. >>>>>> >>>>>> Example: >>>>>> >>>>>> int main(char argc, char *argv[]) >>>>>> { >>>>>> DoCGI(); >>>>>> return 0; >>>>>> } >>>>>> >>>>>> where DoCgi() is reads stdin and maybe environment >>>>>> strings. >>>>>> >>>>>> int DoCGI() >>>>>> { >>>>>> // get environment string, if required >>>>>> // read stdin as FORM POSTING encoding data >>>>>> // process, generate response >>>>>> >>>>>> printf("Status: 200\n"); >>>>>> printf("Content-Type: text/text\n\n"); >>>>>> printf"UTF8:%s\n",szResponse); >>>>>> return 0; >>>>>> } >>>>>> >>>>>> For FastCGI, you would prepare your own socket >>>>>> listening >>>>>> server for incoming request from the web server >>>>>> itself. >>>>>> >>>>>> -- >>>>>> HLS >>>>> >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 18 Mar 2010 11:21 See below... On Wed, 17 Mar 2010 14:40:38 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:u2a2q51uc73hcuqv16c4uld4s8957ud4hd(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:40:02 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:r9h0q55c6oml5fhpkrcfs2jtbsof7rf1ld(a)4ax.com... >>>> See below... >>>> On Tue, 16 Mar 2010 20:50:58 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> >>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>message >>>>>news:%23qdTxMXxKHA.4492(a)TK2MSFTNGP05.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>> >>>>>>> You continue to ignore my repeatedly stated >>>>>>> requirement >>>>>>> of 4.0 GB of data (a deterministic finite automaton) >>>>>>> that >>>>>>> must be loaded before execution can begin. If you >>>>>>> ignore >>>>>>> my fundamental requirements you can't possibly >>>>>>> provide >>>>>>> good advice. >>>>>> >>>>>> >>>>>> Heed my advice if you want to get somewhere. Its >>>>>> coming >>>>>> for FREE! >>>>>> >>>>>> If you keeping 4 GB loaded under a 24x7 loaded exe , >>>>>> then >>>>>> the rest of your system will suffer and you will get >>>>>> paging delays during idle times. >>>>>> >>>>>> That calls for a memory map design requirement. It has >>>>>> nothing to do with FASTCGI unless youy think this >>>>>> "FastCGI >>>>>> Language" you wish to use provides a library for you >>>>>> to >>>>>> automatically do this for you - it does not. >>>>>> Independent >>>>>> concepts. >>>>>> >>>>>> -- >>>>>> HLS >>>>> >>>>>Fundamentally FastCGI differs from CGI in that the app >>>>>is >>>>>held in memory rather than re-loaded every time. >>>>>Although >>>>>this feature may have limited alternatives, I see no >>>>>reason >>>>>to consider them. >>>>> >>>> **** >>>> You obviously did not read the specs. NOWHERE does it >>>> say >>>> "the app is held in memory"; >>>> specifically, >>>> http://www.fastcgi.com/drupal/node/6?q=node/15 says >>>> * FastCGI processes are persistent: after finishing a >>>> request, they wait for a new >>>> request instead of exiting. >>> >>>So if you really want to get nitpicky it is held in >>>virtual >>>memory (from the developer's point of view) that is mapped >>>to RAM, if there is no reason to page it out. >> **** >> See what I mean? You are insisting that because YOU don't >> see a reason that it should be >> paged out, the OPERATING SYSTEM will do what YOU think it >> ought to, and not page it out, >> which flies in the face of reality, in which an operating >> system will page out pages of an >> idle process. > >So if I must I keep invoking in by another process to force >it to keep its pages. Every five minutes add up all of your >pixels, forcing it to hit every memory location. **** Only if the working set size the OS maintains exceeds your application footprint. Otherwise, all you do is page-thrash your own app. joe **** > >> **** >>>Which >>>essentially boils down to it is held in RAM. (Even though >>>it >>>can theoretically be paged out at any time, in actual >>>practice this rarely occurs if you have enough RAM. >>>Virtual >>>Memory is an abstraction of RAM that can at times be >>>ignored. >> **** >> Are you ABSOLUTELY CERTAIN that the host OS behaves >> according to your dreams? It might >> not, and it might not actually have tunable parameters you >> can control! THis is what I >> mean about fact-finding; you have to talk to the experts >> (in this case, a linux expert) to >> understand what is going to REALLY happen to an idle >> process, how working set is managed, >> etc., and not rely upon your unfounded opinion of what you >> think SHOULD happen in the >> best-of-all-possible-worlds scenario. Reality has this >> really nasty habit of ignoring our >> wishes. >> joe >> >> **** >>> >>>> >>>> >>>> 1. The Web server creates FastCGI application processes >>>> to >>>> handle requests. The processes >>>> may be created at startup, or created on demand. >>>> 2. The FastCGI program initializes itself, and waits for >>>> a >>>> new connection from the Web >>>> server. >>>> 3. When a client request comes in, the Web server opens >>>> a >>>> connection to the FastCGI >>>> process. The server sends the CGI environment variable >>>> information and standard input over >>>> the connection. >>> >>>Not in the case of FastCGI, FastCGI does not use >>>environment >>>variables. >> **** >> Perhaps you misread the point 3, which clearly states >> "sends the CGI environment variable >> INFORMATION...over the connection" (emphasis added, in >> case you missed it). >> ***** >>> >>>> 4. The FastCGI process sends the standard output and >>>> error >>>> information back to the server >>>> over the same connection. >>>> . When the FastCGI process closes the connection, the >>>> request is complete. The FastCGI >>>> process then waits for another connection from the Web >>>> server. >>>> >>>> NOWHERE in this spec does is make ANY claim that the >>>> "app >>>> is held in memory", only that a >>> >>>That is logically entailed by the statement that the app >>>does not terminate at the end of the request. With enough >>>RAM and few processes running one can virtually ignore >>>virtual memory. It may not be this way in theory, but, I >>>know that it is this way in practice, because in practice >>>I >>>achieve the performance that could not be otherwise >>>achieved. >> ***** >> No, it is NOT "logically entailed" by the staement that >> the app does not terminate; this >> again shows your failure to understand how real operating >> systems work, which is how they >> are designed to work, not how you wish they had been >> designed to work. Windows, for >> example, pages out the pages of an idle process, >> particularly if they become LRU (but on >> general principles, will page them out). Unix used to >> work the same way. You have made >> this incredible leap of faith based on what you WANT it to >> do, with no input from a linux >> expert about how it ACTUALLY works. >> ***** >>> >>>> new app instance is not launched. Why you would leap to >>>> the (erroneous) conclusion that >>>> not starting a second app instance means the app is >>>> "held >>>> in memory" escapes me. What is >>>> "held in memory" is up to the platform's specific >>>> concepts >>>> of address space management for >>>> a process in the context of a certain physical memory >>>> configuration, and it is likely that >>>> some or all of the app's pages will be paged out to make >>>> room for other processes. This >>>> is your typical "design by wishful thinking" approach >>>> where yu make assertions about how >>> >>>I think that you may be getting just a little bit too >>>disrespectful. I do however really appreciate the support >>>that you provided on the quality of my patent. >> **** >> But we keep telling you where you are going wrong, and you >> keep insisting you must be >> right, because your success depends on your assumptions >> being right, even though most of > >I have seen my assumptions consistently be right, and I can >force them to be right if needed. If the OS wants to page >out an idle process then my process will automatically do >busy work whenever it would otherwise be idle. If that does >not work I will convert to a real-time OS. If that does not >work I will write my own OS from scratch and make it work. >Failure is not an option. > >> us know they are actually wrong, based on our own >> experience. And when we tell you that >> you are wrong, and why you are wrong, you come back and >> tell us you *must* be right >> because you *want* the behavior you hypothesize, even >> though real systems don't work that >> way! We are getting very frustrated with having to tell >> you over and over again that your >> assumptions are faulty, and it annoys us that you ignore >> what we say and insist that the >> operating system must behave the way you erroneously state >> it behaves. I can see Hector >> getting frustrated for the same reason. I've been through >> this concerning memory >> allocation and TCP/IP with you, and my patience is running >> thin. You are now repeating >> the scenario with Web servers, where I am expert enough to >> know you are making nonsensical >> statements and Hector actually is an expert and is telling >> you (a) you are wrong and (b) >> how to do it right, and you keep making even more confused >> statements. >> joe >> >> ***** >>> >>>> things work, which are wrong, and then proceed to >>>> systematically ignore eveyone who tells >>>> you "but it doesn't work that way". This discussion is >>>> shaping up just like the "physical >>>> contiguous memory" discussion of a few months ago where >>>> you insisted that storage >>>> allocators *had* to work in a way that, historically, >>>> storage allocators have NEVER >>>> worked, and that virtual memory had to work in a fashion >>>> that has never been recognized as >>>> a way any operating system designer would implement >>>> virtual memory. Now you are telling >>>> us how FASTCGI works, which is, unfortunately, not the >>>> way >>>> it works at all. And that it >>>> will solve problems it is incapable of (not to mention >>>> irrelevant to) solving. >>>> >>>> So start paying attention to the answers. It will save >>>> you a lot of time and effort as >>>> you attempt to make technologies do things they never >>>> were >>>> designed to do, or try to get >>>> them to meet specifications they never pretended to >>>> meet. >>>> joe >>>> ***** >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 18 Mar 2010 11:30 See below... On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> >>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>message >>>>>news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>> >>>>>>> Fastest speed is highest priority, secondary to this >>>>>>> is >>>>>>> portability, it looks like FastCGI provides both. >>>>>>> Another very high priority is minimal development >>>>>>> costs. >>>>>>> As far as I can tell FastCGI is good here too. >>>>>> >>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>> methodology that was designed with EARLY needs to >>>>>> speed >>>>>> up >>>>>> loading time. You won't have that problem today. >>>>> >>>>>You continue to ignore my repeatedly stated requirement >>>>>of >>>>>4.0 GB of data (a deterministic finite automaton) that >>>>>must >>>>>be loaded before execution can begin. If you ignore my >>>>>fundamental requirements you can't possibly provide good >>>>>advice. >>>> *** >>>> And you continue to ignore our observations that there >>>> are >>>> no technologies that guarantee >>>> this, and the timings of program behavior can be swamped >>>> by communication times (did I >>>> tell you about the 70-second DNS resolution time? Did >>>> you >>>> know that you can expect delays >>>> in integer number of seconds to locate your server?) >>>> >>> >>>That is horrendous. I doubt if it is frequent, and local >>>caching would hopefully prevent it from recurring. >> **** >> Depends. Did the end user configure for local DNS >> caching? WHat was the cache retention >> time set by the end user? Note, again, YOU HAVE NO >> CONTROL OVER THESE PARAMETERS! >> **** >>> >>>> You fasten on some technology that says it is "faster" >>>> and >>>> assume it is the answer to your >>>> problem; you toss jargon around in ways that those of us >>>> who know the technology don't >>>> even recognize, such as assertions how FastCGI allows >>>> you >>>> to have browse buttons in the >>> >>>It seems to me that most of you guys continue to simply >>>ignore some of my most fundamental design requirements. >>>App >>>load time for me can be several minutes, app execution >>>time >>>of a large job is never more than 100 milliseconds. >> ***** >> No, we hear them, we just don't believe that FastCGI is >> going to be the solution that will >> solve every issue for you. Hector says that CGI has been >> improved to where it has >> comparable performance to FastCGI. I have no information >> to confirm or deny that >> statement. Again, you are not a server expert, so you >> should pay attention to the server >> expert. >> **** >>> >>>If the app is the only process besides the OS, by limiting >>>memory usage to no more than 75% of available RAM in >>>actual >>>practice both the app and its data are paged out almost >>>not >>>at all. On a real-time OS this behavior would be >>>guaranteed. >>>On MS Windows it is close enough. Linux probably does not >>>do >>>worse than Windows in this regard. >> **** >> I have no idea why you think this is true. This is not an >> assumption I would have >> considered valid until I talked to a linux expert (since >> you will be hosted on linux). Yet >> you make this assertion simply because you THINK it should >> work, as opposed to getting >> data that confirms that the OS actually works this way. >> **** > >An OS that unloads any part of an executing process when it >has no need for additional memory would be an idiotic OS. It >would be flat out stupid to do this. I am not convinced that >either Windows or Linux is this stupid. *** But we learned DECADES ago that "eager page-out" improves overall system performance, because it reduces startup transients when a process (or one of its threads) becomes schedulable. Again, the "working set" has to be paged in, and this is easier if there are available page frames in memory. So what you are calling "stupid" is in fact the DEMONSTRATED INTELLIGENT way to build operating systems, and we've known this since they early 1970s. This is why I have never known a system that did not page out an idle process IN ANITICIPATION OF NEED. Windows does it; I'm trying to find out if linux does it, but again, your opinion of how you THINK an operating system SHOULD work flies in the fact of reality. Just because you don't see a reason doesn't mean there isn't one, and one that has been established by long practice (the "actual practice" you fantasize about doesn't exist). Windows indeed works this way, and so have other operating systems, all based on BEST practice (not someone's fantasized "actual practice"). So I would not predicate actual behavior on fantasies, but learn the facts. ***** > >If this same OS unloads part of a process because the >process has been idle for a long time, AND needs more memory >to do some housekeeping task, then this case I could see a >reason that would not be stupid. **** It is the antiipation-of-need which in actual practice (not fantasized "actual practive", but ACTUAL practice) has been found to be the best solution for general purpose operating systems, to maximize overall performance. Until you have measured (there's that nasty phrase, implying the obtaining of facts) behavior, you have no reason to assume it corresponds to your fantasies. **** > >>> >>>> local Web server, and how it is the perfect answer to >>>> all >>>> your problems, when we know it >>>> isn't anything but another hack whose purpose is to >>>> compensate for bad initiali designs >>>> (CGI requiring a new process per request). You ignore >>>> core issues such as how you are >>>> going to port your app so it runs on a linux or Unix >>>> server, and how you are going to >>> >>>I am rewriting it to require very little OS specific code. >>>About the only thing that I will have to change is the way >>>that PNG files are read into memory. I will have to switch >>>to LibPNG. All of the MFC code is only development code >>>and >>>will not be part of production. >> **** >> PNG files would not be "read into memory"; instead, you >> will need to convert the input >> string (however it is provided, most likely via stdin) to >> a bitmap, and that requires a >> library that does not read the PNG data from a file but >> from a text stream. >> **** > >YES, or if I have to write is to disk and read it back in >again. Reading and writing 2K files will not kill me. > >>> >>>> rewrite it to be platform-independent and possibly even >>>> GUI-free, and your unrealistic >>> >>>The web service needs no GUI. It is all PNG in, and UTF-8 >>>out. >> **** >> OK, that's good. You *will* have to write the FastCGI >> interface and make sure it works >> multithreaded so it can handle multiple requests. > >I am going with Hectors idea of a tiny embedded webserver >built in to my code. There are several freeware one's one >of these looks good enogh for my needs: mongoose. > >> **** >>> >>>> expectations about how process memory is managed, and >>>> how >>>> every OS is magically going to >>>> behave in the way you wish, without any effort on your >>>> part. ANd you refuse to listen >>>> when people tell you that you don't understand either >>>> the >>>> technology or the intrinsic >>>> behavior of how an operating system works. >>>> >>>> You really need to do better research on these >>>> technologies before you start making such >>>> assertions, or ask questions and pay attention to the >>>> answers. I'm not even a Web server >>>> expert like Hector, and I can tell you are making >>>> nonsensical statements. Listen to him. >>>> He sounds like someone who does this for a living, and >>>> you >>>> should trust what he is telling >>>> you. >>>> joe >>> >>>All that I know for sure is that some of my fundamental >>>requirements are immutable. Some of Hector's advice seemed >>>to continue to ignore these requirements. I was >>>envisioning >>>the web service as a single application. This may not have >>>been the way that he was envisioning it. >> **** >> You can do it by writing a "front end" that exports a >> known port and implements your >> appliction-specific protocol to get the data to send to >> your "business logic" component. >> It doesn't solve the paging problem, or the 4GB problem. >> joe > >I think that I will be going with some sort of application >specific webserver such as mongoose. Hector tried to sell me >on his webserver, but, it was too costly and not flexible >enough for me right now. In the future when I scale up to >multiple servers this may change. > >> >> **** >>> >>>> >>>>> >>>>> > Its nonsense to think this is "answer" to >>>>>> your problem. In addition, if you worry about >>>>>> processing >>>>>> speed, native C/C++ is your choice, so all you are >>>>>> doing >>>>>> is writing a CGI application, a console application >>>>>> that >>>>>> reads standard input. >>>>>> >>>>>> If you want productivity, learn PHP. Simple, well >>>>>> supported. Then if you really find out that FASTCGI >>>>>> is >>>>>> needed for your slow OS and machine, PHP supports it. >>>>>> >>>>>> The point is your SOLUTION is not FASTCGI, it is the >>>>>> actual application that does the work. >>>>>> >>>>>> -- >>>>>> HLS >>>>> >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 18 Mar 2010 11:35
See below... On Wed, 17 Mar 2010 18:13:16 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:soh2q5dglai3e7440iltlj0324npra8ger(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:01:22 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:%23J$4XzXxKHA.4240(a)TK2MSFTNGP06.phx.gbl... >>>> Geoff wrote: >>>> >>>>> On Tue, 16 Mar 2010 19:59:35 -0400, Hector Santos >>>>> <sant9442(a)nospam.gmail.com> wrote: >>>>> >>>>>> Some other (windows based) web servers do support a >>>>>> ISAPI interface, off hand, a early version interface. >>>>>> Its simple a special DLL load that special hooks. I >>>>>> believe I seen Apache. I don't think it will work >>>>>> under >>>>>> a wienie OS. >>>>>> >>>>> >>>>> I did a little Wiki'ing of my own and apparently >>>>> mod_isapi is >>>>> Windows-Apache-only and doesn't support filters (which >>>>> is >>>>> what I >>>>> envisioned for this) so it's a dead end in this case. >>>> >>>> >>>> Yeah, today, for script map portability, PHP, is a >>>> viable >>>> option. It has FASTCGI support, which in my view, only >>>> was necessary for lower machine days. The alternative >>>> is >>>> to use an embedded language and PHP "EMBED" version >>>> offers >>>> this if you want to add it to your web server. I believe >>>> Apache has a module for this too. >>>> >>>> For Peter, I doubt this is the design issue. Handling >>>> the >>>> 4 GB of meta data files is the key thing he needs to >>>> work >>>> out. He erroneously thinks keep the EXE "resident" will >>>> solve this performance problem. He has no choice but to >>>> use memory map files and now that he said he need to >>>> load >>>> all them into a vector, he need a class that offers >>>> vector >>>> MMF virtualization. >>>> >>>> We have our own CFileMap class which is major part of >>>> our >>>> server large file database framework. I'm sure there >>>> other public classes out there. Here is an Old >>>> Microsoft >>>> CMemoryMappedFile MFC subclass based off CMemFile: >>>> >>>> http://support.microsoft.com/kb/142377 >>>> >>>> >>>> -- >>>> HLS >>> >>>Why can't I simply read in the files in the normal way? >>> >> **** >> Because you have noplace to put it! You can't use more >> than 2GB address space in a >> Windows app (I've sent a question off to my linux-expert >> friend to see how much address > >I have used 4000 MB of space under windows many times. It >seems like you may be continuing to insist upon factually >incorrect informatuion. *** Then you are using Win64. Win32 does not allow more than 2GB in normal operation and 3GB under special boot conditions, not normally encountered except in Enterprise Server. 4GB, not possible. Except in Win64. If you are not using WIn64, you are in errorr, and that's a fact. **** > >> space you get under linux; a question you should have >> asked somewhere along the line and >> known the answer to). Mapped files also exist in linux, >> but the details are not known to >> me. BUt there is no way in this universe you are going to >> be able to read 4FB of data >> into a WIn32 app running in any environment. Now, a >> 64-bit native app is a completely >> different question, and in that environment it is trivial. >> joe > >I don't want the conversion cost and the cost of 64-bit >pointers right now. I am already doing 4000+ MB, which is >close enough for the foreseeable future. **** This is not possible in Win32, and if you believe if, you have erroneous data. The upper 2GB is owned by the kernel, and the lower 2GB by the applications. The upper 2GB memory map is universal for every process (and invisible except to threads of that process that are running in the kernel). For Win64, the boundary is 8TB. I have no idea what you mean by the "cost of a 64-bit pointer". In a 64-bit address space, the "cost" of a 64-bit pointer is minuscle. Perhaps you are referring to the "cost" of loading one or using one, which are the same as the costs of loading a 32-bit pointer and using one (look at the bus widths and cache sizes of 64-bit machines!) joe **** > >> **** >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |