How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Joseph M. Newcomer on 18 Mar 2010 11:38 See below... On Wed, 17 Mar 2010 14:01:22 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:fr82q55im3pagn9lq813auvtmtj73dc315(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 22:22:30 -0400, Hector Santos >> <sant9442(a)nospam.gmail.com> wrote: >> >>> >>>Peter Olcott wrote: >>> >>>>> Once again, all you need is a standard C/C++ console >>>>> application that reads standard input to read your HTTP >>>>> transfer encoded data block. You either write the >>>>> multi-part MIME processor or you get a library or class >>>>> that does it for you. >>>> >>>> What handles transmission errors? >>> >>> >>>Socket drop events/detection. You have to be ready for >>>it. >> **** >> Any On... notification of CAsyncSocket has an error code >> ***** >>> >>>> Another thing, I want to >>>> immediately close the connection on the first packet if >>>> anything besides a 24-bit PNG file is detected. >>> >>> >>> shutdown(mySocketHandle,2) >>> flush receiver buffer loop >>> closesocket(mySocketHandle) >> ***** >> I take exception with his "first packet" concept; it shows >> he is totally clueless about >> TCP/IP protocol, which is *strictly* a stream protocol. I >> have no idea why he would think >> that "packet" is a relevant concept. I just answered this >> in detail. >> **** >>> >>>> My app MUST >>>> be written in C++. I was envisioning this as a single >>>> app. >>>> One that handles the URL-encoded input, and the same one >>>> to >>>> process this data. In any case the one that processes >>>> this >>>> data must remain resident. >>> >>> >>>What web server are you going to be using? >> **** >> It probably doesn't matter if he builds an appropriate >> FASTCGI interface for it. And it >> has to be in pure C++, not MFC, and it must not have a >> GUI. >> **** >>> >>>> I have no idea what you are talking about when referring >>>> to >>>> a memory map. >>> >>> >>>Well you should be aware of these things around here. >>>Other common >>>terms are Memory Map File (MMF) or File Map >>> >>>http://en.wikipedia.org/wiki/Memory-mapped_file >>>http://msdn.microsoft.com/en-us/library/aa366556(VS.85).aspx >>>http://msdn.microsoft.com/en-us/library/ms810613.aspx >>> >>> > It is not a 4.0 GB file, it is numerous files >>> >>>> making up a total of 4.0 GB. >>> >>> >>>So you open each one up as a MMF. If this is static meta >>>data, then >>>merged them into one. >> ***** >> He doesn't seem to grasp the fact that he only has 2GB >> (optimistically, 1GB) of contiguous >> virtual memory into which something can be mapped. >> Probably a lot less. He mistakes a >> 32-bit address space *capability* for a 32-bit address >> space *reality*, a notion I have >> tried to explain to him many times in the past as being >> wishful thinking, even if running >> a Win32 app marked /LARGEADDRESSAWARE on Win64 wherer >> there really is 4GB-128K of total >> virtual address space available. I guess his code, stack, >> and heap must all take 0 bytes. >> And there is no threading, so no thread stacks are >> required. > >Right now on my 8.0 GB 64-bit Windows 7 system it is >reporting 6051 MB of physical memory is available. >When I run my /LARGEADDRESSAWARE 32-bit process on this >machine it fails when it reaches 4.0 GB. >Now that I have reduced my memory requirements by 100-fold >without any reduction in performance this will be less of an >issue. **** Yes, going back to my comment, you are running on Win64, which allows 4GB of user space, which has to account for your heap, all your stacks if multithreaded, your code, and all your support code (various DLLs). I said you can't get 4GB on Win32. You can get it running a Win32 app on Win64. joe **** > >> joe >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 18 Mar 2010 11:59 "Stephen Myers" <""StephenMyers\"@discussions(a)microsoft.com"> wrote in message news:O7Fz2UqxKHA.6140(a)TK2MSFTNGP05.phx.gbl... > Peter Olcott wrote: >> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >> message >> news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>> message >>>> news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>> >>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>> message >>>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>> Peter Olcott wrote: >>>>>>> >>>>>>> >>>>>>>> Fastest speed is highest priority, secondary to >>>>>>>> this is >>>>>>>> portability, it looks like FastCGI provides both. >>>>>>>> Another very high priority is minimal development >>>>>>>> costs. >>>>>>>> As far as I can tell FastCGI is good here too. >>>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>>> methodology that was designed with EARLY needs to >>>>>>> speed >>>>>>> up >>>>>>> loading time. You won't have that problem today. >>>>>> You continue to ignore my repeatedly stated >>>>>> requirement of >>>>>> 4.0 GB of data (a deterministic finite automaton) >>>>>> that >>>>>> must >>>>>> be loaded before execution can begin. If you ignore >>>>>> my >>>>>> fundamental requirements you can't possibly provide >>>>>> good >>>>>> advice. >>>>> *** >>>>> And you continue to ignore our observations that there >>>>> are >>>>> no technologies that guarantee >>>>> this, and the timings of program behavior can be >>>>> swamped >>>>> by communication times (did I >>>>> tell you about the 70-second DNS resolution time? Did >>>>> you >>>>> know that you can expect delays >>>>> in integer number of seconds to locate your server?) >>>>> >>>> That is horrendous. I doubt if it is frequent, and >>>> local >>>> caching would hopefully prevent it from recurring. >>> **** >>> Depends. Did the end user configure for local DNS >>> caching? WHat was the cache retention >>> time set by the end user? Note, again, YOU HAVE NO >>> CONTROL OVER THESE PARAMETERS! >>> **** >>>>> You fasten on some technology that says it is "faster" >>>>> and >>>>> assume it is the answer to your >>>>> problem; you toss jargon around in ways that those of >>>>> us >>>>> who know the technology don't >>>>> even recognize, such as assertions how FastCGI allows >>>>> you >>>>> to have browse buttons in the >>>> It seems to me that most of you guys continue to simply >>>> ignore some of my most fundamental design requirements. >>>> App >>>> load time for me can be several minutes, app execution >>>> time >>>> of a large job is never more than 100 milliseconds. >>> ***** >>> No, we hear them, we just don't believe that FastCGI is >>> going to be the solution that will >>> solve every issue for you. Hector says that CGI has >>> been improved to where it has >>> comparable performance to FastCGI. I have no >>> information to confirm or deny that >>> statement. Again, you are not a server expert, so you >>> should pay attention to the server >>> expert. >>> **** >>>> If the app is the only process besides the OS, by >>>> limiting >>>> memory usage to no more than 75% of available RAM in >>>> actual >>>> practice both the app and its data are paged out almost >>>> not >>>> at all. On a real-time OS this behavior would be >>>> guaranteed. >>>> On MS Windows it is close enough. Linux probably does >>>> not do >>>> worse than Windows in this regard. >>> **** >>> I have no idea why you think this is true. This is not >>> an assumption I would have >>> considered valid until I talked to a linux expert (since >>> you will be hosted on linux). Yet >>> you make this assertion simply because you THINK it >>> should work, as opposed to getting >>> data that confirms that the OS actually works this way. >>> **** >> >> An OS that unloads any part of an executing process when >> it has no need for additional memory would be an idiotic >> OS. It would be flat out stupid to do this. I am not >> convinced that either Windows or Linux is this stupid. >> >> If this same OS unloads part of a process because the >> process has been idle for a long time, AND needs more >> memory to do some housekeeping task, then this case I >> could see a reason that would not be stupid. >> > Why should any OS keep all of your process in memory at > all? The concept of paged virtual memory allows the OS to > keep what's currently needed in memory. This allows 10 or > 20 or 500 processes to have the illusion that their 2GB is > resident whenever they're running. Consider a program > that puts up a logo from a resource at startup. Should > the OS keep that snippet (page) of memory in physical > memory on the off chance that it will be needed again? In my particular instance the dedicated server will only have the single purpose of running my OCR web application. If I have to I will turn virtual memory off. It must keep my process in memory even if I have to make changes to the OS to achieve this. The simplest way that I can force the data to remain resident, is to access every memory location whenever my process is otherwise idle. > >>>>> local Web server, and how it is the perfect answer to >>>>> all >>>>> your problems, when we know it >>>>> isn't anything but another hack whose purpose is to >>>>> compensate for bad initiali designs >>>>> (CGI requiring a new process per request). You ignore >>>>> core issues such as how you are >>>>> going to port your app so it runs on a linux or Unix >>>>> server, and how you are going to >>>> I am rewriting it to require very little OS specific >>>> code. >>>> About the only thing that I will have to change is the >>>> way >>>> that PNG files are read into memory. I will have to >>>> switch >>>> to LibPNG. All of the MFC code is only development code >>>> and >>>> will not be part of production. >>> **** >>> PNG files would not be "read into memory"; instead, you >>> will need to convert the input >>> string (however it is provided, most likely via stdin) >>> to a bitmap, and that requires a >>> library that does not read the PNG data from a file but >>> from a text stream. >>> **** >> >> YES, or if I have to write is to disk and read it back in >> again. Reading and writing 2K files will not kill me. >> >>>>> rewrite it to be platform-independent and possibly >>>>> even >>>>> GUI-free, and your unrealistic >>>> The web service needs no GUI. It is all PNG in, and >>>> UTF-8 >>>> out. >>> **** >>> OK, that's good. You *will* have to write the FastCGI >>> interface and make sure it works >>> multithreaded so it can handle multiple requests. >> >> I am going with Hectors idea of a tiny embedded webserver >> built in to my code. There are several freeware one's >> one of these looks good enogh for my needs: mongoose. >> >>> **** >>>>> expectations about how process memory is managed, and >>>>> how >>>>> every OS is magically going to >>>>> behave in the way you wish, without any effort on your >>>>> part. ANd you refuse to listen >>>>> when people tell you that you don't understand either >>>>> the >>>>> technology or the intrinsic >>>>> behavior of how an operating system works. >>>>> >>>>> You really need to do better research on these >>>>> technologies before you start making such >>>>> assertions, or ask questions and pay attention to the >>>>> answers. I'm not even a Web server >>>>> expert like Hector, and I can tell you are making >>>>> nonsensical statements. Listen to him. >>>>> He sounds like someone who does this for a living, and >>>>> you >>>>> should trust what he is telling >>>>> you. >>>>> joe >>>> All that I know for sure is that some of my fundamental >>>> requirements are immutable. Some of Hector's advice >>>> seemed >>>> to continue to ignore these requirements. I was >>>> envisioning >>>> the web service as a single application. This may not >>>> have >>>> been the way that he was envisioning it. >>> **** >>> You can do it by writing a "front end" that exports a >>> known port and implements your >>> appliction-specific protocol to get the data to send to >>> your "business logic" component. >>> It doesn't solve the paging problem, or the 4GB problem. >>> joe >> >> I think that I will be going with some sort of >> application specific webserver such as mongoose. Hector >> tried to sell me on his webserver, but, it was too costly >> and not flexible enough for me right now. In the future >> when I scale up to multiple servers this may change. >> >>> **** >>>>>>> Its nonsense to think this is "answer" to >>>>>>> your problem. In addition, if you worry about >>>>>>> processing >>>>>>> speed, native C/C++ is your choice, so all you are >>>>>>> doing >>>>>>> is writing a CGI application, a console application >>>>>>> that >>>>>>> reads standard input. >>>>>>> >>>>>>> If you want productivity, learn PHP. Simple, well >>>>>>> supported. Then if you really find out that FASTCGI >>>>>>> is >>>>>>> needed for your slow OS and machine, PHP supports >>>>>>> it. >>>>>>> >>>>>>> The point is your SOLUTION is not FASTCGI, it is the >>>>>>> actual application that does the work. >>>>>>> >>>>>>> -- >>>>>>> HLS >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>
From: Peter Olcott on 18 Mar 2010 12:03 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:u0h4q5l884o56c5ecd6hhnsb2fjci7j1vm(a)4ax.com... > See below... > On Wed, 17 Mar 2010 18:18:55 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:47i2q5tch6oigbaqllh9lk0942mp5pnkt0(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:54:10 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>message >>>>news:08i0q513cr5paa3m0mkvlmaoecd9vu7oqu(a)4ax.com... >>>>> See below... >>>>> On Tue, 16 Mar 2010 17:28:45 -0500, "Peter Olcott" >>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>> >>>>>> >>>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>>message >>>>>>news:ea%23kdFVxKHA.5576(a)TK2MSFTNGP05.phx.gbl... >>>>>>> Peter Olcott wrote: >>>>>>> >>>>>>>> >>>>>>>> I would be doing FastCGI in C++, if I do it. I am >>>>>>>> considering FastCGI because I need a completely >>>>>>>> portable >>>>>>>> way for a user to browse for a PNG file to send to >>>>>>>> my >>>>>>>> server. HTML has this built in, and it is connected >>>>>>>> to >>>>>>>> CGI. As far as a protocol goes, this would be >>>>>>>> trivial. >>>>>>>> The user sends a 24-bit PNG file, the server sends >>>>>>>> back >>>>>>>> UTF-8. That's the whole protocol. Sometimes the >>>>>>>> UTF-8 >>>>>>>> is >>>>>>>> the PNG file translated into text, and other times >>>>>>>> it >>>>>>>> is >>>>>>>> an error response. >>>>>>> >>>>>>> Doesn't sound you understand, or at least are >>>>>>> showing >>>>>>> you >>>>>>> don't, what FastCGI is. >>>>>>> >>>>>>> Forget fastcgi, what you need is a dedicated http >>>>>>> server >>>>>>> with the built-in procssing or a http server that >>>>>>> supports >>>>>>> spawning your cgi EXE application as a special URL. >>>>>> >>>>>>I can't take the several minutes of CGI process spawn >>>>>>time >>>>>>for my application, (it has to load gigabytes of data >>>>>>into >>>>>>memory before it begins) thus conventional CGI is >>>>>>infeasible. >>>>> *** >>>>> So, in other words, if I happen to be so unfrotunate >>>>> as >>>>> to >>>>> make a request on your server >>>>> and there IS no instance of the process, that in fact, >>>>> I >>>>> cannot expect 500ms reponse time >>>>> but instead can expect MINUTES of delay! >>>>> >>>>> NOte that all FASTCGI promises is that your process >>>>> does >>>>> not exist or require a restart >>>>> once it has been launched; it does not promise the >>>>> pages >>>>> will be "held in memory". >>>>> joe >>>> >>>>In actual practice operating systems typically do not >>>>page >>>>out code or data unless they need room for something >>>>else. >>>>This will be the only application on the dedicated >>>>server, >>>>thus little reason to be paged out. If necessary I will >>>>run >>>>a very tiny little background process that forces the >>>>app >>>>to >>>>do something once in a while. >>> *** >>> In acutal practice, they DO page out out code or data >>> "just in case", in fact, Windows has >>> done this for years. So don't cite fictitious "actual >>> practice" as your rationale. I'd >>> ask a linux expert what linux does before presuming I >>> knew >>> what it did. In the history of >>> operating systems, I've work on four (including WIndows) >>> that paged out idle process pages >>> on the general principle that if the process isn't being >>> used, it shouldn't be taking up >>> any space. So I'm curious where you learned aobut >>> "actual >>> practice" and why you believe >>> this must be true. Unix was one of the three systems I >>> used that did this, and VMS was >>> another IBM's TSS/360 was the fourth. so my "actual >>> practice" on real operating systems >>> says the opposite of your concept of "actual practice", >>> and I'm curious what OS you used >>> that worked this way. And why you think you can >>> generalize from one instance of one OS to >>> believe that all operating systems must follow this >>> pattern. >>> joe >> >>Like I already said (in a message that you have not gotten >>to yet) I can always force it to add up all of its pixels >>every once in a while. For example whenever it is >>otherwise >>idle. This would require read access to every memory >>location, thus keeping pages paged in. > **** > Only if you set the working set limits to not force it to > page its own pages out! > > (You do know about working set, right? I've explained it > in the past) > joe Why is any of this complexity actually necessary in actual practice? It would be absurd to be required these days when so much RAM is available. If I always have 4.0 GB more than I need either your whole point becomes moot, or OS design is atrocious. > >> >>> **** >>>> >>> >>>>> >>>>> ***** >>>>>> >>>>>>> >>>>>>> You will handle standard INPUT as the encoded data. >>>>>>> Simple. >>>>>>> >>>>>>> Example: >>>>>>> >>>>>>> int main(char argc, char *argv[]) >>>>>>> { >>>>>>> DoCGI(); >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> where DoCgi() is reads stdin and maybe environment >>>>>>> strings. >>>>>>> >>>>>>> int DoCGI() >>>>>>> { >>>>>>> // get environment string, if required >>>>>>> // read stdin as FORM POSTING encoding data >>>>>>> // process, generate response >>>>>>> >>>>>>> printf("Status: 200\n"); >>>>>>> printf("Content-Type: text/text\n\n"); >>>>>>> printf"UTF8:%s\n",szResponse); >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> For FastCGI, you would prepare your own socket >>>>>>> listening >>>>>>> server for incoming request from the web server >>>>>>> itself. >>>>>>> >>>>>>> -- >>>>>>> HLS >>>>>> >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 18 Mar 2010 12:05 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:m5h4q5du9bs9cev94k4etlsjv77f4o4r3s(a)4ax.com... > See below... > On Wed, 17 Mar 2010 14:40:38 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:u2a2q51uc73hcuqv16c4uld4s8957ud4hd(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:40:02 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>message >>>>news:r9h0q55c6oml5fhpkrcfs2jtbsof7rf1ld(a)4ax.com... >>>>> See below... >>>>> On Tue, 16 Mar 2010 20:50:58 -0500, "Peter Olcott" >>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>> >>>>>> >>>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>>message >>>>>>news:%23qdTxMXxKHA.4492(a)TK2MSFTNGP05.phx.gbl... >>>>>>> Peter Olcott wrote: >>>>>>> >>>>>>> >>>>>>>> You continue to ignore my repeatedly stated >>>>>>>> requirement >>>>>>>> of 4.0 GB of data (a deterministic finite >>>>>>>> automaton) >>>>>>>> that >>>>>>>> must be loaded before execution can begin. If you >>>>>>>> ignore >>>>>>>> my fundamental requirements you can't possibly >>>>>>>> provide >>>>>>>> good advice. >>>>>>> >>>>>>> >>>>>>> Heed my advice if you want to get somewhere. Its >>>>>>> coming >>>>>>> for FREE! >>>>>>> >>>>>>> If you keeping 4 GB loaded under a 24x7 loaded exe , >>>>>>> then >>>>>>> the rest of your system will suffer and you will get >>>>>>> paging delays during idle times. >>>>>>> >>>>>>> That calls for a memory map design requirement. It >>>>>>> has >>>>>>> nothing to do with FASTCGI unless youy think this >>>>>>> "FastCGI >>>>>>> Language" you wish to use provides a library for you >>>>>>> to >>>>>>> automatically do this for you - it does not. >>>>>>> Independent >>>>>>> concepts. >>>>>>> >>>>>>> -- >>>>>>> HLS >>>>>> >>>>>>Fundamentally FastCGI differs from CGI in that the app >>>>>>is >>>>>>held in memory rather than re-loaded every time. >>>>>>Although >>>>>>this feature may have limited alternatives, I see no >>>>>>reason >>>>>>to consider them. >>>>>> >>>>> **** >>>>> You obviously did not read the specs. NOWHERE does it >>>>> say >>>>> "the app is held in memory"; >>>>> specifically, >>>>> http://www.fastcgi.com/drupal/node/6?q=node/15 says >>>>> * FastCGI processes are persistent: after finishing a >>>>> request, they wait for a new >>>>> request instead of exiting. >>>> >>>>So if you really want to get nitpicky it is held in >>>>virtual >>>>memory (from the developer's point of view) that is >>>>mapped >>>>to RAM, if there is no reason to page it out. >>> **** >>> See what I mean? You are insisting that because YOU >>> don't >>> see a reason that it should be >>> paged out, the OPERATING SYSTEM will do what YOU think >>> it >>> ought to, and not page it out, >>> which flies in the face of reality, in which an >>> operating >>> system will page out pages of an >>> idle process. >> >>So if I must I keep invoking in by another process to >>force >>it to keep its pages. Every five minutes add up all of >>your >>pixels, forcing it to hit every memory location. > **** > Only if the working set size the OS maintains exceeds your > application footprint. > Otherwise, all you do is page-thrash your own app. > joe > You NEVER thrash if you always have far more physical memory than you need. If you do thrash when you always have far more physical memory than you need, the OS was designed by morons. > **** >> >>> **** >>>>Which >>>>essentially boils down to it is held in RAM. (Even >>>>though >>>>it >>>>can theoretically be paged out at any time, in actual >>>>practice this rarely occurs if you have enough RAM. >>>>Virtual >>>>Memory is an abstraction of RAM that can at times be >>>>ignored. >>> **** >>> Are you ABSOLUTELY CERTAIN that the host OS behaves >>> according to your dreams? It might >>> not, and it might not actually have tunable parameters >>> you >>> can control! THis is what I >>> mean about fact-finding; you have to talk to the experts >>> (in this case, a linux expert) to >>> understand what is going to REALLY happen to an idle >>> process, how working set is managed, >>> etc., and not rely upon your unfounded opinion of what >>> you >>> think SHOULD happen in the >>> best-of-all-possible-worlds scenario. Reality has this >>> really nasty habit of ignoring our >>> wishes. >>> joe >>> >>> **** >>>> >>>>> >>>>> >>>>> 1. The Web server creates FastCGI application >>>>> processes >>>>> to >>>>> handle requests. The processes >>>>> may be created at startup, or created on demand. >>>>> 2. The FastCGI program initializes itself, and waits >>>>> for >>>>> a >>>>> new connection from the Web >>>>> server. >>>>> 3. When a client request comes in, the Web server >>>>> opens >>>>> a >>>>> connection to the FastCGI >>>>> process. The server sends the CGI environment variable >>>>> information and standard input over >>>>> the connection. >>>> >>>>Not in the case of FastCGI, FastCGI does not use >>>>environment >>>>variables. >>> **** >>> Perhaps you misread the point 3, which clearly states >>> "sends the CGI environment variable >>> INFORMATION...over the connection" (emphasis added, in >>> case you missed it). >>> ***** >>>> >>>>> 4. The FastCGI process sends the standard output and >>>>> error >>>>> information back to the server >>>>> over the same connection. >>>>> . When the FastCGI process closes the connection, the >>>>> request is complete. The FastCGI >>>>> process then waits for another connection from the Web >>>>> server. >>>>> >>>>> NOWHERE in this spec does is make ANY claim that the >>>>> "app >>>>> is held in memory", only that a >>>> >>>>That is logically entailed by the statement that the app >>>>does not terminate at the end of the request. With >>>>enough >>>>RAM and few processes running one can virtually ignore >>>>virtual memory. It may not be this way in theory, but, I >>>>know that it is this way in practice, because in >>>>practice >>>>I >>>>achieve the performance that could not be otherwise >>>>achieved. >>> ***** >>> No, it is NOT "logically entailed" by the staement that >>> the app does not terminate; this >>> again shows your failure to understand how real >>> operating >>> systems work, which is how they >>> are designed to work, not how you wish they had been >>> designed to work. Windows, for >>> example, pages out the pages of an idle process, >>> particularly if they become LRU (but on >>> general principles, will page them out). Unix used to >>> work the same way. You have made >>> this incredible leap of faith based on what you WANT it >>> to >>> do, with no input from a linux >>> expert about how it ACTUALLY works. >>> ***** >>>> >>>>> new app instance is not launched. Why you would leap >>>>> to >>>>> the (erroneous) conclusion that >>>>> not starting a second app instance means the app is >>>>> "held >>>>> in memory" escapes me. What is >>>>> "held in memory" is up to the platform's specific >>>>> concepts >>>>> of address space management for >>>>> a process in the context of a certain physical memory >>>>> configuration, and it is likely that >>>>> some or all of the app's pages will be paged out to >>>>> make >>>>> room for other processes. This >>>>> is your typical "design by wishful thinking" approach >>>>> where yu make assertions about how >>>> >>>>I think that you may be getting just a little bit too >>>>disrespectful. I do however really appreciate the >>>>support >>>>that you provided on the quality of my patent. >>> **** >>> But we keep telling you where you are going wrong, and >>> you >>> keep insisting you must be >>> right, because your success depends on your assumptions >>> being right, even though most of >> >>I have seen my assumptions consistently be right, and I >>can >>force them to be right if needed. If the OS wants to page >>out an idle process then my process will automatically do >>busy work whenever it would otherwise be idle. If that >>does >>not work I will convert to a real-time OS. If that does >>not >>work I will write my own OS from scratch and make it work. >>Failure is not an option. >> >>> us know they are actually wrong, based on our own >>> experience. And when we tell you that >>> you are wrong, and why you are wrong, you come back and >>> tell us you *must* be right >>> because you *want* the behavior you hypothesize, even >>> though real systems don't work that >>> way! We are getting very frustrated with having to tell >>> you over and over again that your >>> assumptions are faulty, and it annoys us that you ignore >>> what we say and insist that the >>> operating system must behave the way you erroneously >>> state >>> it behaves. I can see Hector >>> getting frustrated for the same reason. I've been >>> through >>> this concerning memory >>> allocation and TCP/IP with you, and my patience is >>> running >>> thin. You are now repeating >>> the scenario with Web servers, where I am expert enough >>> to >>> know you are making nonsensical >>> statements and Hector actually is an expert and is >>> telling >>> you (a) you are wrong and (b) >>> how to do it right, and you keep making even more >>> confused >>> statements. >>> joe >>> >>> ***** >>>> >>>>> things work, which are wrong, and then proceed to >>>>> systematically ignore eveyone who tells >>>>> you "but it doesn't work that way". This discussion >>>>> is >>>>> shaping up just like the "physical >>>>> contiguous memory" discussion of a few months ago >>>>> where >>>>> you insisted that storage >>>>> allocators *had* to work in a way that, historically, >>>>> storage allocators have NEVER >>>>> worked, and that virtual memory had to work in a >>>>> fashion >>>>> that has never been recognized as >>>>> a way any operating system designer would implement >>>>> virtual memory. Now you are telling >>>>> us how FASTCGI works, which is, unfortunately, not the >>>>> way >>>>> it works at all. And that it >>>>> will solve problems it is incapable of (not to mention >>>>> irrelevant to) solving. >>>>> >>>>> So start paying attention to the answers. It will >>>>> save >>>>> you a lot of time and effort as >>>>> you attempt to make technologies do things they never >>>>> were >>>>> designed to do, or try to get >>>>> them to meet specifications they never pretended to >>>>> meet. >>>>> joe >>>>> ***** >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 18 Mar 2010 12:12
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:f9h4q5ltbdi70eot2n5me08gij5vg556bc(a)4ax.com... > See below... > On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>message >>>>news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>> >>>>>> >>>>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>>message >>>>>>news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>> Peter Olcott wrote: >>>>>>> >>>>>>> >>>>>>>> Fastest speed is highest priority, secondary to >>>>>>>> this >>>>>>>> is >>>>>>>> portability, it looks like FastCGI provides both. >>>>>>>> Another very high priority is minimal development >>>>>>>> costs. >>>>>>>> As far as I can tell FastCGI is good here too. >>>>>>> >>>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>>> methodology that was designed with EARLY needs to >>>>>>> speed >>>>>>> up >>>>>>> loading time. You won't have that problem today. >>>>>> >>>>>>You continue to ignore my repeatedly stated >>>>>>requirement >>>>>>of >>>>>>4.0 GB of data (a deterministic finite automaton) that >>>>>>must >>>>>>be loaded before execution can begin. If you ignore my >>>>>>fundamental requirements you can't possibly provide >>>>>>good >>>>>>advice. >>>>> *** >>>>> And you continue to ignore our observations that there >>>>> are >>>>> no technologies that guarantee >>>>> this, and the timings of program behavior can be >>>>> swamped >>>>> by communication times (did I >>>>> tell you about the 70-second DNS resolution time? Did >>>>> you >>>>> know that you can expect delays >>>>> in integer number of seconds to locate your server?) >>>>> >>>> >>>>That is horrendous. I doubt if it is frequent, and local >>>>caching would hopefully prevent it from recurring. >>> **** >>> Depends. Did the end user configure for local DNS >>> caching? WHat was the cache retention >>> time set by the end user? Note, again, YOU HAVE NO >>> CONTROL OVER THESE PARAMETERS! >>> **** >>>> >>>>> You fasten on some technology that says it is "faster" >>>>> and >>>>> assume it is the answer to your >>>>> problem; you toss jargon around in ways that those of >>>>> us >>>>> who know the technology don't >>>>> even recognize, such as assertions how FastCGI allows >>>>> you >>>>> to have browse buttons in the >>>> >>>>It seems to me that most of you guys continue to simply >>>>ignore some of my most fundamental design requirements. >>>>App >>>>load time for me can be several minutes, app execution >>>>time >>>>of a large job is never more than 100 milliseconds. >>> ***** >>> No, we hear them, we just don't believe that FastCGI is >>> going to be the solution that will >>> solve every issue for you. Hector says that CGI has >>> been >>> improved to where it has >>> comparable performance to FastCGI. I have no >>> information >>> to confirm or deny that >>> statement. Again, you are not a server expert, so you >>> should pay attention to the server >>> expert. >>> **** >>>> >>>>If the app is the only process besides the OS, by >>>>limiting >>>>memory usage to no more than 75% of available RAM in >>>>actual >>>>practice both the app and its data are paged out almost >>>>not >>>>at all. On a real-time OS this behavior would be >>>>guaranteed. >>>>On MS Windows it is close enough. Linux probably does >>>>not >>>>do >>>>worse than Windows in this regard. >>> **** >>> I have no idea why you think this is true. This is not >>> an >>> assumption I would have >>> considered valid until I talked to a linux expert (since >>> you will be hosted on linux). Yet >>> you make this assertion simply because you THINK it >>> should >>> work, as opposed to getting >>> data that confirms that the OS actually works this way. >>> **** >> >>An OS that unloads any part of an executing process when >>it >>has no need for additional memory would be an idiotic OS. >>It >>would be flat out stupid to do this. I am not convinced >>that >>either Windows or Linux is this stupid. > *** > But we learned DECADES ago that "eager page-out" improves > overall system performance, > because it reduces startup transients when a process (or > one of its threads) becomes > schedulable. Again, the "working set" has to be paged in, > and this is easier if there are > available page frames in memory. So what you are calling > "stupid" is in fact the > DEMONSTRATED INTELLIGENT way to build operating systems, > and we've known this since they > early 1970s. This is why I have never known a system that > did not page out an idle > process IN ANITICIPATION OF NEED. Windows does it; I'm > trying to find out if linux does > it, but again, your opinion of how you THINK an operating > system SHOULD work flies in the > fact of reality. Just because you don't see a reason > doesn't mean there isn't one, and > one that has been established by long practice (the > "actual practice" you fantasize about > doesn't exist). Windows indeed works this way, and so > have other operating systems, all > based on BEST practice (not someone's fantasized "actual > practice"). So I would not > predicate actual behavior on fantasies, but learn the > facts. > ***** I will simply do whatever it takes to achieve the functional result that I require. If there is always an extra 4.0 GB of RAM available, then your whole anticipation of need concept should become moot. >> >>If this same OS unloads part of a process because the >>process has been idle for a long time, AND needs more >>memory >>to do some housekeeping task, then this case I could see a >>reason that would not be stupid. > **** > It is the antiipation-of-need which in actual practice > (not fantasized "actual practive", > but ACTUAL practice) has been found to be the best > solution for general purpose operating > systems, to maximize overall performance. Until you have > measured (there's that nasty > phrase, implying the obtaining of facts) behavior, you > have no reason to assume it > corresponds to your fantasies. > **** >> >>>> >>>>> local Web server, and how it is the perfect answer to >>>>> all >>>>> your problems, when we know it >>>>> isn't anything but another hack whose purpose is to >>>>> compensate for bad initiali designs >>>>> (CGI requiring a new process per request). You ignore >>>>> core issues such as how you are >>>>> going to port your app so it runs on a linux or Unix >>>>> server, and how you are going to >>>> >>>>I am rewriting it to require very little OS specific >>>>code. >>>>About the only thing that I will have to change is the >>>>way >>>>that PNG files are read into memory. I will have to >>>>switch >>>>to LibPNG. All of the MFC code is only development code >>>>and >>>>will not be part of production. >>> **** >>> PNG files would not be "read into memory"; instead, you >>> will need to convert the input >>> string (however it is provided, most likely via stdin) >>> to >>> a bitmap, and that requires a >>> library that does not read the PNG data from a file but >>> from a text stream. >>> **** >> >>YES, or if I have to write is to disk and read it back in >>again. Reading and writing 2K files will not kill me. >> >>>> >>>>> rewrite it to be platform-independent and possibly >>>>> even >>>>> GUI-free, and your unrealistic >>>> >>>>The web service needs no GUI. It is all PNG in, and >>>>UTF-8 >>>>out. >>> **** >>> OK, that's good. You *will* have to write the FastCGI >>> interface and make sure it works >>> multithreaded so it can handle multiple requests. >> >>I am going with Hectors idea of a tiny embedded webserver >>built in to my code. There are several freeware one's one >>of these looks good enogh for my needs: mongoose. >> >>> **** >>>> >>>>> expectations about how process memory is managed, and >>>>> how >>>>> every OS is magically going to >>>>> behave in the way you wish, without any effort on your >>>>> part. ANd you refuse to listen >>>>> when people tell you that you don't understand either >>>>> the >>>>> technology or the intrinsic >>>>> behavior of how an operating system works. >>>>> >>>>> You really need to do better research on these >>>>> technologies before you start making such >>>>> assertions, or ask questions and pay attention to the >>>>> answers. I'm not even a Web server >>>>> expert like Hector, and I can tell you are making >>>>> nonsensical statements. Listen to him. >>>>> He sounds like someone who does this for a living, and >>>>> you >>>>> should trust what he is telling >>>>> you. >>>>> joe >>>> >>>>All that I know for sure is that some of my fundamental >>>>requirements are immutable. Some of Hector's advice >>>>seemed >>>>to continue to ignore these requirements. I was >>>>envisioning >>>>the web service as a single application. This may not >>>>have >>>>been the way that he was envisioning it. >>> **** >>> You can do it by writing a "front end" that exports a >>> known port and implements your >>> appliction-specific protocol to get the data to send to >>> your "business logic" component. >>> It doesn't solve the paging problem, or the 4GB problem. >>> joe >> >>I think that I will be going with some sort of application >>specific webserver such as mongoose. Hector tried to sell >>me >>on his webserver, but, it was too costly and not flexible >>enough for me right now. In the future when I scale up to >>multiple servers this may change. >> >>> >>> **** >>>> >>>>> >>>>>> >>>>>> > Its nonsense to think this is "answer" to >>>>>>> your problem. In addition, if you worry about >>>>>>> processing >>>>>>> speed, native C/C++ is your choice, so all you are >>>>>>> doing >>>>>>> is writing a CGI application, a console application >>>>>>> that >>>>>>> reads standard input. >>>>>>> >>>>>>> If you want productivity, learn PHP. Simple, well >>>>>>> supported. Then if you really find out that FASTCGI >>>>>>> is >>>>>>> needed for your slow OS and machine, PHP supports >>>>>>> it. >>>>>>> >>>>>>> The point is your SOLUTION is not FASTCGI, it is the >>>>>>> actual application that does the work. >>>>>>> >>>>>>> -- >>>>>>> HLS >>>>>> >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |