How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Hector Santos on 18 Mar 2010 12:26 Joe, Peter, is it just a habit, but you both have to quote the entire message just to embed a few lines of response in the middle of the message? It becomes a hunting game! Look for my replies below. Honestly, the NNTP Server would work more efficiently. :) Peter Olcott wrote: > "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in > message news:f9h4q5ltbdi70eot2n5me08gij5vg556bc(a)4ax.com... >> See below... >> On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>> message news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>>> See below... >>>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>> message >>>>> news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>> >>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>>> message >>>>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>>> Peter Olcott wrote: >>>>>>>> >>>>>>>> >>>>>>>>> Fastest speed is highest priority, secondary to >>>>>>>>> this >>>>>>>>> is >>>>>>>>> portability, it looks like FastCGI provides both. >>>>>>>>> Another very high priority is minimal development >>>>>>>>> costs. >>>>>>>>> As far as I can tell FastCGI is good here too. >>>>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>>>> methodology that was designed with EARLY needs to >>>>>>>> speed >>>>>>>> up >>>>>>>> loading time. You won't have that problem today. >>>>>>> You continue to ignore my repeatedly stated >>>>>>> requirement >>>>>>> of >>>>>>> 4.0 GB of data (a deterministic finite automaton) that >>>>>>> must >>>>>>> be loaded before execution can begin. If you ignore my >>>>>>> fundamental requirements you can't possibly provide >>>>>>> good >>>>>>> advice. >>>>>> *** >>>>>> And you continue to ignore our observations that there >>>>>> are >>>>>> no technologies that guarantee >>>>>> this, and the timings of program behavior can be >>>>>> swamped >>>>>> by communication times (did I >>>>>> tell you about the 70-second DNS resolution time? Did >>>>>> you >>>>>> know that you can expect delays >>>>>> in integer number of seconds to locate your server?) >>>>>> >>>>> That is horrendous. I doubt if it is frequent, and local >>>>> caching would hopefully prevent it from recurring. >>>> **** >>>> Depends. Did the end user configure for local DNS >>>> caching? WHat was the cache retention >>>> time set by the end user? Note, again, YOU HAVE NO >>>> CONTROL OVER THESE PARAMETERS! >>>> **** >>>>>> You fasten on some technology that says it is "faster" >>>>>> and >>>>>> assume it is the answer to your >>>>>> problem; you toss jargon around in ways that those of >>>>>> us >>>>>> who know the technology don't >>>>>> even recognize, such as assertions how FastCGI allows >>>>>> you >>>>>> to have browse buttons in the sure >>>>> It seems to me that most of you guys continue to simply >>>>> ignore some of my most fundamental design requirements. >>>>> App >>>>> load time for me can be several minutes, app execution >>>>> time >>>>> of a large job is never more than 100 milliseconds. >>>> ***** >>>> No, we hear them, we just don't believe that FastCGI is >>>> going to be the solution that will >>>> solve every issue for you. Hector says that CGI has >>>> been >>>> improved to where it has >>>> comparable performance to FastCGI. I have no >>>> information >>>> to confirm or deny that >>>> statement. Again, you are not a server expert, so you >>>> should pay attention to the server >>>> expert. >>>> **** >>>>> If the app is the only process besides the OS, by >>>>> limiting >>>>> memory usage to no more than 75% of available RAM in >>>>> actual >>>>> practice both the app and its data are paged out almost >>>>> not >>>>> at all. On a real-time OS this behavior would be >>>>> guaranteed. >>>>> On MS Windows it is close enough. Linux probably does >>>>> not >>>>> do >>>>> worse than Windows in this regard. >>>> **** >>>> I have no idea why you think this is true. This is not >>>> an >>>> assumption I would have >>>> considered valid until I talked to a linux expert (since >>>> you will be hosted on linux). Yet >>>> you make this assertion simply because you THINK it >>>> should >>>> work, as opposed to getting >>>> data that confirms that the OS actually works this way. >>>> **** whatever >>> An OS that unloads any part of an executing process when >>> it >>> has no need for additional memory would be an idiotic OS. >>> It >>> would be flat out stupid to do this. I am not convinced >>> that >>> either Windows or Linux is this stupid. >> *** >> But we learned DECADES ago that "eager page-out" improves >> overall system performance, >> because it reduces startup transients when a process (or >> one of its threads) becomes >> schedulable. Again, the "working set" has to be paged in, >> and this is easier if there are >> available page frames in memory. So what you are calling >> "stupid" is in fact the >> DEMONSTRATED INTELLIGENT way to build operating systems, >> and we've known this since they >> early 1970s. This is why I have never known a system that >> did not page out an idle >> process IN ANITICIPATION OF NEED. Windows does it; I'm >> trying to find out if linux does >> it, but again, your opinion of how you THINK an operating >> system SHOULD work flies in the >> fact of reality. Just because you don't see a reason >> doesn't mean there isn't one, and >> one that has been established by long practice (the >> "actual practice" you fantasize about >> doesn't exist). Windows indeed works this way, and so >> have other operating systems, all >> based on BEST practice (not someone's fantasized "actual >> practice"). So I would not >> predicate actual behavior on fantasies, but learn the >> facts. >> ***** > > I will simply do whatever it takes to achieve the functional > result that I require. If there is always an extra 4.0 GB of > RAM available, then your whole anticipation of need concept > should become moot. > >>> If this same OS unloads part of a process because the >>> process has been idle for a long time, AND needs more >>> memory >>> to do some housekeeping task, then this case I could see a >>> reason that would not be stupid. >> **** >> It is the antiipation-of-need which in actual practice >> (not fantasized "actual practive", >> but ACTUAL practice) has been found to be the best >> solution for general purpose operating >> systems, to maximize overall performance. Until you have >> measured (there's that nasty >> phrase, implying the obtaining of facts) behavior, you >> have no reason to assume it >> corresponds to your fantasies. >> **** >>>>>> local Web server, and how it is the perfect answer to >>>>>> all >>>>>> your problems, when we know it >>>>>> isn't anything but another hack whose purpose is to >>>>>> compensate for bad initiali designs >>>>>> (CGI requiring a new process per request). You ignore >>>>>> core issues such as how you are >>>>>> going to port your app so it runs on a linux or Unix >>>>>> server, and how you are going to >>>>> I am rewriting it to require very little OS specific >>>>> code. >>>>> About the only thing that I will have to change is the >>>>> way >>>>> that PNG files are read into memory. I will have to >>>>> switch >>>>> to LibPNG. All of the MFC code is only development code >>>>> and >>>>> will not be part of production. >>>> **** >>>> PNG files would not be "read into memory"; instead, you >>>> will need to convert the input >>>> string (however it is provided, most likely via stdin) >>>> to >>>> a bitmap, and that requires a >>>> library that does not read the PNG data from a file but >>>> from a text stream. >>>> **** >>> YES, or if I have to write is to disk and read it back in >>> again. Reading and writing 2K files will not kill me. >>> >>>>>> rewrite it to be platform-independent and possibly >>>>>> even >>>>>> GUI-free, and your unrealistic >>>>> The web service needs no GUI. It is all PNG in, and >>>>> UTF-8 >>>>> out. >>>> **** >>>> OK, that's good. You *will* have to write the FastCGI >>>> interface and make sure it works >>>> multithreaded so it can handle multiple requests. >>> I am going with Hectors idea of a tiny embedded webserver >>> built in to my code. There are several freeware one's one >>> of these looks good enogh for my needs: mongoose. >>> >>>> **** >>>>>> expectations about how process memory is managed, and >>>>>> how >>>>>> every OS is magically going to >>>>>> behave in the way you wish, without any effort on your >>>>>> part. ANd you refuse to listen >>>>>> when people tell you that you don't understand either >>>>>> the >>>>>> technology or the intrinsic >>>>>> behavior of how an operating system works. >>>>>> >>>>>> You really need to do better research on these >>>>>> technologies before you start making such >>>>>> assertions, or ask questions and pay attention to the >>>>>> answers. I'm not even a Web server >>>>>> expert like Hector, and I can tell you are making >>>>>> nonsensical statements. Listen to him. >>>>>> He sounds like someone who does this for a living, and >>>>>> you >>>>>> should trust what he is telling >>>>>> you. >>>>>> joe >>>>> All that I know for sure is that some of my fundamental >>>>> requirements are immutable. Some of Hector's advice >>>>> seemed >>>>> to continue to ignore these requirements. I was >>>>> envisioning >>>>> the web service as a single application. This may not >>>>> have >>>>> been the way that he was envisioning it. >>>> **** >>>> You can do it by writing a "front end" that exports a >>>> known port and implements your >>>> appliction-specific protocol to get the data to send to >>>> your "business logic" component. >>>> It doesn't solve the paging problem, or the 4GB problem. >>>> joe >>> I think that I will be going with some sort of application >>> specific webserver such as mongoose. Hector tried to sell >>> me >>> on his webserver, but, it was too costly and not flexible >>> enough for me right now. In the future when I scale up to >>> multiple servers this may change. >>> >>>> **** >>>>>>>> Its nonsense to think this is "answer" to >>>>>>>> your problem. In addition, if you worry about >>>>>>>> processing >>>>>>>> speed, native C/C++ is your choice, so all you are >>>>>>>> doing >>>>>>>> is writing a CGI application, a console application >>>>>>>> that >>>>>>>> reads standard input. >>>>>>>> >>>>>>>> If you want productivity, learn PHP. Simple, well >>>>>>>> supported. Then if you really find out that FASTCGI >>>>>>>> is >>>>>>>> needed for your slow OS and machine, PHP supports >>>>>>>> it. >>>>>>>> >>>>>>>> The point is your SOLUTION is not FASTCGI, it is the >>>>>>>> actual application that does the work. >>>>>>>> >>>>>>>> -- >>>>>>>> HLS >>>>>> Joseph M. Newcomer [MVP] >>>>>> email: newcomer(a)flounder.com >>>>>> Web: http://www.flounder.com >>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > > -- HLS
From: Peter Olcott on 18 Mar 2010 12:28 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:tph4q519di72be50p8tkgecton9v1bqdkv(a)4ax.com... > See below... > On Wed, 17 Mar 2010 18:13:16 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:soh2q5dglai3e7440iltlj0324npra8ger(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:01:22 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> >>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>message >>>>news:%23J$4XzXxKHA.4240(a)TK2MSFTNGP06.phx.gbl... >>>>> Geoff wrote: >>>>> >>>>>> On Tue, 16 Mar 2010 19:59:35 -0400, Hector Santos >>>>>> <sant9442(a)nospam.gmail.com> wrote: >>>>>> >>>>>>> Some other (windows based) web servers do support a >>>>>>> ISAPI interface, off hand, a early version >>>>>>> interface. >>>>>>> Its simple a special DLL load that special hooks. I >>>>>>> believe I seen Apache. I don't think it will work >>>>>>> under >>>>>>> a wienie OS. >>>>>>> >>>>>> >>>>>> I did a little Wiki'ing of my own and apparently >>>>>> mod_isapi is >>>>>> Windows-Apache-only and doesn't support filters >>>>>> (which >>>>>> is >>>>>> what I >>>>>> envisioned for this) so it's a dead end in this case. >>>>> >>>>> >>>>> Yeah, today, for script map portability, PHP, is a >>>>> viable >>>>> option. It has FASTCGI support, which in my view, >>>>> only >>>>> was necessary for lower machine days. The alternative >>>>> is >>>>> to use an embedded language and PHP "EMBED" version >>>>> offers >>>>> this if you want to add it to your web server. I >>>>> believe >>>>> Apache has a module for this too. >>>>> >>>>> For Peter, I doubt this is the design issue. Handling >>>>> the >>>>> 4 GB of meta data files is the key thing he needs to >>>>> work >>>>> out. He erroneously thinks keep the EXE "resident" >>>>> will >>>>> solve this performance problem. He has no choice but >>>>> to >>>>> use memory map files and now that he said he need to >>>>> load >>>>> all them into a vector, he need a class that offers >>>>> vector >>>>> MMF virtualization. >>>>> >>>>> We have our own CFileMap class which is major part of >>>>> our >>>>> server large file database framework. I'm sure there >>>>> other public classes out there. Here is an Old >>>>> Microsoft >>>>> CMemoryMappedFile MFC subclass based off CMemFile: >>>>> >>>>> http://support.microsoft.com/kb/142377 >>>>> >>>>> >>>>> -- >>>>> HLS >>>> >>>>Why can't I simply read in the files in the normal way? >>>> >>> **** >>> Because you have noplace to put it! You can't use more >>> than 2GB address space in a >>> Windows app (I've sent a question off to my linux-expert >>> friend to see how much address >> >>I have used 4000 MB of space under windows many times. It >>seems like you may be continuing to insist upon factually >>incorrect informatuion. > *** > Then you are using Win64. Win32 does not allow more than > 2GB in normal operation and 3GB > under special boot conditions, not normally encountered > except in Enterprise Server. 4GB, > not possible. Except in Win64. If you are not using > WIn64, you are in errorr, and that's > a fact. > **** Yes, Windows 7, 64 with 8.0 GB RAM >> > >>> space you get under linux; a question you should have >>> asked somewhere along the line and >>> known the answer to). Mapped files also exist in linux, >>> but the details are not known to >>> me. BUt there is no way in this universe you are going >>> to >>> be able to read 4FB of data >>> into a WIn32 app running in any environment. Now, a >>> 64-bit native app is a completely >>> different question, and in that environment it is >>> trivial. >>> joe >> >>I don't want the conversion cost and the cost of 64-bit >>pointers right now. I am already doing 4000+ MB, which is >>close enough for the foreseeable future. > **** > This is not possible in Win32, and if you believe if, you > have erroneous data. The upper > 2GB is owned by the kernel, and the lower 2GB by the > applications. The upper 2GB memory > map is universal for every process (and invisible except > to threads of that process that > are running in the kernel). For Win64, the boundary is > 8TB. Win64 > > I have no idea what you mean by the "cost of a 64-bit > pointer". In a 64-bit address > space, the "cost" of a 64-bit pointer is minuscle. > Perhaps you are referring to the > "cost" of loading one or using one, which are the same as > the costs of loading a 32-bit > pointer and using one (look at the bus widths and cache > sizes of 64-bit machines!) > joe I would imagine that std::vector pointers would double in size. I wrote a std::vector class once for the original Borland Turbo C++ 1.01. http://edn.embarcadero.com/article/21751 I wanted to do real development on my handheld HP 200LX. I had to write this std::vector without the benefit of templates, that was a little tricky. I forgot the internal details, perhaps only the std::vector object would increase in size, not the individual elements. If this is the case then 64-bit pointers would be cheap. Yes this is probably the case. > **** >> >>> **** >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 18 Mar 2010 12:32 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:44i4q51iamfnidn7cji8mvft16if8u6jdl(a)4ax.com... > See below... > On Wed, 17 Mar 2010 14:01:22 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:fr82q55im3pagn9lq813auvtmtj73dc315(a)4ax.com... >>> See below... >>> On Tue, 16 Mar 2010 22:22:30 -0400, Hector Santos >>> <sant9442(a)nospam.gmail.com> wrote: >>> >>>> >>>>Peter Olcott wrote: >>>> >>>>>> Once again, all you need is a standard C/C++ console >>>>>> application that reads standard input to read your >>>>>> HTTP >>>>>> transfer encoded data block. You either write the >>>>>> multi-part MIME processor or you get a library or >>>>>> class >>>>>> that does it for you. >>>>> >>>>> What handles transmission errors? >>>> >>>> >>>>Socket drop events/detection. You have to be ready for >>>>it. >>> **** >>> Any On... notification of CAsyncSocket has an error code >>> ***** >>>> >>>>> Another thing, I want to >>>>> immediately close the connection on the first packet >>>>> if >>>>> anything besides a 24-bit PNG file is detected. >>>> >>>> >>>> shutdown(mySocketHandle,2) >>>> flush receiver buffer loop >>>> closesocket(mySocketHandle) >>> ***** >>> I take exception with his "first packet" concept; it >>> shows >>> he is totally clueless about >>> TCP/IP protocol, which is *strictly* a stream protocol. >>> I >>> have no idea why he would think >>> that "packet" is a relevant concept. I just answered >>> this >>> in detail. >>> **** >>>> >>>>> My app MUST >>>>> be written in C++. I was envisioning this as a single >>>>> app. >>>>> One that handles the URL-encoded input, and the same >>>>> one >>>>> to >>>>> process this data. In any case the one that processes >>>>> this >>>>> data must remain resident. >>>> >>>> >>>>What web server are you going to be using? >>> **** >>> It probably doesn't matter if he builds an appropriate >>> FASTCGI interface for it. And it >>> has to be in pure C++, not MFC, and it must not have a >>> GUI. >>> **** >>>> >>>>> I have no idea what you are talking about when >>>>> referring >>>>> to >>>>> a memory map. >>>> >>>> >>>>Well you should be aware of these things around here. >>>>Other common >>>>terms are Memory Map File (MMF) or File Map >>>> >>>>http://en.wikipedia.org/wiki/Memory-mapped_file >>>>http://msdn.microsoft.com/en-us/library/aa366556(VS.85).aspx >>>>http://msdn.microsoft.com/en-us/library/ms810613.aspx >>>> >>>> > It is not a 4.0 GB file, it is numerous files >>>> >>>>> making up a total of 4.0 GB. >>>> >>>> >>>>So you open each one up as a MMF. If this is static >>>>meta >>>>data, then >>>>merged them into one. >>> ***** >>> He doesn't seem to grasp the fact that he only has 2GB >>> (optimistically, 1GB) of contiguous >>> virtual memory into which something can be mapped. >>> Probably a lot less. He mistakes a >>> 32-bit address space *capability* for a 32-bit address >>> space *reality*, a notion I have >>> tried to explain to him many times in the past as being >>> wishful thinking, even if running >>> a Win32 app marked /LARGEADDRESSAWARE on Win64 wherer >>> there really is 4GB-128K of total >>> virtual address space available. I guess his code, >>> stack, >>> and heap must all take 0 bytes. >>> And there is no threading, so no thread stacks are >>> required. >> >>Right now on my 8.0 GB 64-bit Windows 7 system it is >>reporting 6051 MB of physical memory is available. >>When I run my /LARGEADDRESSAWARE 32-bit process on this >>machine it fails when it reaches 4.0 GB. >>Now that I have reduced my memory requirements by 100-fold >>without any reduction in performance this will be less of >>an >>issue. > **** > Yes, going back to my comment, you are running on Win64, > which allows 4GB of user space, > which has to account for your heap, all your stacks if > multithreaded, your code, and all > your support code (various DLLs). I said you can't get > 4GB on Win32. You can get it > running a Win32 app on Win64. > joe > **** Yes and it also looks like converting to 64-bit will cost very little system resources. I was thinking that std::vector elements would increase in size by four bytes. With further consideration this would seem to not be the case. >> >>> joe >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 18 Mar 2010 12:40 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OLClPfrxKHA.404(a)TK2MSFTNGP02.phx.gbl... > Joe, Peter, is it just a habit, but you both have to quote > the entire message just to embed a few lines of response > in the middle of the message? It becomes a hunting game! > Look for my replies below. > (1) Its less trouble for me to do it this way. (2) When I look for replies I use page-down, not scroll-down, so finding the reply is very quick. (3) It retains the entire context of every reply, thus easily verifying who is at fault in communication errors. (4) Text takes very little room or bandwidth, so the cost of these resources is more than compensated by the savings in human labor. > Honestly, the NNTP Server would work more efficiently. :) > > Peter Olcott wrote: > >> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >> message >> news:f9h4q5ltbdi70eot2n5me08gij5vg556bc(a)4ax.com... >>> See below... >>> On Wed, 17 Mar 2010 15:04:40 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>> message >>>> news:43b2q51hukg9s1gofoqds7vodf0lcq4bhp(a)4ax.com... >>>>> See below... >>>>> On Tue, 16 Mar 2010 22:19:25 -0500, "Peter Olcott" >>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>> >>>>>> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>>>> message >>>>>> news:6ng0q5pbktq73efm697rl1tf18jv6dhknn(a)4ax.com... >>>>>>> On Tue, 16 Mar 2010 20:16:44 -0500, "Peter Olcott" >>>>>>> <NoSpam(a)OCR4Screen.com> wrote: >>>>>>> >>>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote >>>>>>>> in >>>>>>>> message >>>>>>>> news:ueIgi6WxKHA.1692(a)TK2MSFTNGP04.phx.gbl... >>>>>>>>> Peter Olcott wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Fastest speed is highest priority, secondary to >>>>>>>>>> this >>>>>>>>>> is >>>>>>>>>> portability, it looks like FastCGI provides both. >>>>>>>>>> Another very high priority is minimal development >>>>>>>>>> costs. >>>>>>>>>> As far as I can tell FastCGI is good here too. >>>>>>>>> Dude, FastCGI is not a "Language." Its an idea, a >>>>>>>>> methodology that was designed with EARLY needs to >>>>>>>>> speed >>>>>>>>> up >>>>>>>>> loading time. You won't have that problem today. >>>>>>>> You continue to ignore my repeatedly stated >>>>>>>> requirement >>>>>>>> of >>>>>>>> 4.0 GB of data (a deterministic finite automaton) >>>>>>>> that >>>>>>>> must >>>>>>>> be loaded before execution can begin. If you ignore >>>>>>>> my >>>>>>>> fundamental requirements you can't possibly provide >>>>>>>> good >>>>>>>> advice. >>>>>>> *** >>>>>>> And you continue to ignore our observations that >>>>>>> there >>>>>>> are >>>>>>> no technologies that guarantee >>>>>>> this, and the timings of program behavior can be >>>>>>> swamped >>>>>>> by communication times (did I >>>>>>> tell you about the 70-second DNS resolution time? >>>>>>> Did >>>>>>> you >>>>>>> know that you can expect delays >>>>>>> in integer number of seconds to locate your server?) >>>>>>> >>>>>> That is horrendous. I doubt if it is frequent, and >>>>>> local >>>>>> caching would hopefully prevent it from recurring. >>>>> **** >>>>> Depends. Did the end user configure for local DNS >>>>> caching? WHat was the cache retention >>>>> time set by the end user? Note, again, YOU HAVE NO >>>>> CONTROL OVER THESE PARAMETERS! >>>>> **** >>>>>>> You fasten on some technology that says it is >>>>>>> "faster" >>>>>>> and >>>>>>> assume it is the answer to your >>>>>>> problem; you toss jargon around in ways that those >>>>>>> of us >>>>>>> who know the technology don't >>>>>>> even recognize, such as assertions how FastCGI >>>>>>> allows >>>>>>> you >>>>>>> to have browse buttons in the > > sure > >>>>> It seems to me that most of you guys continue to > >>>>> simply > >>>>>> ignore some of my most fundamental design >>>>>> requirements. >>>>>> App >>>>>> load time for me can be several minutes, app >>>>>> execution >>>>>> time >>>>>> of a large job is never more than 100 milliseconds. >>>>> ***** >>>>> No, we hear them, we just don't believe that FastCGI >>>>> is >>>>> going to be the solution that will >>>>> solve every issue for you. Hector says that CGI has >>>>> been >>>>> improved to where it has >>>>> comparable performance to FastCGI. I have no >>>>> information >>>>> to confirm or deny that >>>>> statement. Again, you are not a server expert, so you >>>>> should pay attention to the server >>>>> expert. >>>>> **** >>>>>> If the app is the only process besides the OS, by >>>>>> limiting >>>>>> memory usage to no more than 75% of available RAM in >>>>>> actual >>>>>> practice both the app and its data are paged out >>>>>> almost >>>>>> not >>>>>> at all. On a real-time OS this behavior would be >>>>>> guaranteed. >>>>>> On MS Windows it is close enough. Linux probably does >>>>>> not >>>>>> do >>>>>> worse than Windows in this regard. >>>>> **** >>>>> I have no idea why you think this is true. This is >>>>> not an >>>>> assumption I would have >>>>> considered valid until I talked to a linux expert >>>>> (since >>>>> you will be hosted on linux). Yet >>>>> you make this assertion simply because you THINK it >>>>> should >>>>> work, as opposed to getting >>>>> data that confirms that the OS actually works this >>>>> way. >>>>> **** > > whatever > >>>> An OS that unloads any part of an executing process >>>> when it >>>> has no need for additional memory would be an idiotic >>>> OS. It >>>> would be flat out stupid to do this. I am not convinced >>>> that >>>> either Windows or Linux is this stupid. >>> *** >>> But we learned DECADES ago that "eager page-out" >>> improves overall system performance, >>> because it reduces startup transients when a process (or >>> one of its threads) becomes >>> schedulable. Again, the "working set" has to be paged >>> in, and this is easier if there are >>> available page frames in memory. So what you are >>> calling "stupid" is in fact the >>> DEMONSTRATED INTELLIGENT way to build operating systems, >>> and we've known this since they >>> early 1970s. This is why I have never known a system >>> that did not page out an idle >>> process IN ANITICIPATION OF NEED. Windows does it; I'm >>> trying to find out if linux does >>> it, but again, your opinion of how you THINK an >>> operating system SHOULD work flies in the >>> fact of reality. Just because you don't see a reason >>> doesn't mean there isn't one, and >>> one that has been established by long practice (the >>> "actual practice" you fantasize about >>> doesn't exist). Windows indeed works this way, and so >>> have other operating systems, all >>> based on BEST practice (not someone's fantasized "actual >>> practice"). So I would not >>> predicate actual behavior on fantasies, but learn the >>> facts. >>> ***** >> >> I will simply do whatever it takes to achieve the >> functional result that I require. If there is always an >> extra 4.0 GB of RAM available, then your whole >> anticipation of need concept should become moot. >> >>>> If this same OS unloads part of a process because the >>>> process has been idle for a long time, AND needs more >>>> memory >>>> to do some housekeeping task, then this case I could >>>> see a >>>> reason that would not be stupid. >>> **** >>> It is the antiipation-of-need which in actual practice >>> (not fantasized "actual practive", >>> but ACTUAL practice) has been found to be the best >>> solution for general purpose operating >>> systems, to maximize overall performance. Until you >>> have measured (there's that nasty >>> phrase, implying the obtaining of facts) behavior, you >>> have no reason to assume it >>> corresponds to your fantasies. >>> **** >>>>>>> local Web server, and how it is the perfect answer >>>>>>> to >>>>>>> all >>>>>>> your problems, when we know it >>>>>>> isn't anything but another hack whose purpose is to >>>>>>> compensate for bad initiali designs >>>>>>> (CGI requiring a new process per request). You >>>>>>> ignore >>>>>>> core issues such as how you are >>>>>>> going to port your app so it runs on a linux or Unix >>>>>>> server, and how you are going to >>>>>> I am rewriting it to require very little OS specific >>>>>> code. >>>>>> About the only thing that I will have to change is >>>>>> the way >>>>>> that PNG files are read into memory. I will have to >>>>>> switch >>>>>> to LibPNG. All of the MFC code is only development >>>>>> code >>>>>> and >>>>>> will not be part of production. >>>>> **** >>>>> PNG files would not be "read into memory"; instead, >>>>> you >>>>> will need to convert the input >>>>> string (however it is provided, most likely via stdin) >>>>> to >>>>> a bitmap, and that requires a >>>>> library that does not read the PNG data from a file >>>>> but >>>>> from a text stream. >>>>> **** >>>> YES, or if I have to write is to disk and read it back >>>> in >>>> again. Reading and writing 2K files will not kill me. >>>> >>>>>>> rewrite it to be platform-independent and possibly >>>>>>> even >>>>>>> GUI-free, and your unrealistic >>>>>> The web service needs no GUI. It is all PNG in, and >>>>>> UTF-8 >>>>>> out. >>>>> **** >>>>> OK, that's good. You *will* have to write the FastCGI >>>>> interface and make sure it works >>>>> multithreaded so it can handle multiple requests. >>>> I am going with Hectors idea of a tiny embedded >>>> webserver >>>> built in to my code. There are several freeware one's >>>> one >>>> of these looks good enogh for my needs: mongoose. >>>> >>>>> **** >>>>>>> expectations about how process memory is managed, >>>>>>> and >>>>>>> how >>>>>>> every OS is magically going to >>>>>>> behave in the way you wish, without any effort on >>>>>>> your >>>>>>> part. ANd you refuse to listen >>>>>>> when people tell you that you don't understand >>>>>>> either >>>>>>> the >>>>>>> technology or the intrinsic >>>>>>> behavior of how an operating system works. >>>>>>> >>>>>>> You really need to do better research on these >>>>>>> technologies before you start making such >>>>>>> assertions, or ask questions and pay attention to >>>>>>> the >>>>>>> answers. I'm not even a Web server >>>>>>> expert like Hector, and I can tell you are making >>>>>>> nonsensical statements. Listen to him. >>>>>>> He sounds like someone who does this for a living, >>>>>>> and >>>>>>> you >>>>>>> should trust what he is telling >>>>>>> you. >>>>>>> joe >>>>>> All that I know for sure is that some of my >>>>>> fundamental >>>>>> requirements are immutable. Some of Hector's advice >>>>>> seemed >>>>>> to continue to ignore these requirements. I was >>>>>> envisioning >>>>>> the web service as a single application. This may not >>>>>> have >>>>>> been the way that he was envisioning it. >>>>> **** >>>>> You can do it by writing a "front end" that exports a >>>>> known port and implements your >>>>> appliction-specific protocol to get the data to send >>>>> to >>>>> your "business logic" component. >>>>> It doesn't solve the paging problem, or the 4GB >>>>> problem. >>>>> joe >>>> I think that I will be going with some sort of >>>> application >>>> specific webserver such as mongoose. Hector tried to >>>> sell me >>>> on his webserver, but, it was too costly and not >>>> flexible >>>> enough for me right now. In the future when I scale up >>>> to >>>> multiple servers this may change. >>>> >>>>> **** >>>>>>>>> Its nonsense to think this is "answer" to >>>>>>>>> your problem. In addition, if you worry about >>>>>>>>> processing >>>>>>>>> speed, native C/C++ is your choice, so all you are >>>>>>>>> doing >>>>>>>>> is writing a CGI application, a console >>>>>>>>> application >>>>>>>>> that >>>>>>>>> reads standard input. >>>>>>>>> >>>>>>>>> If you want productivity, learn PHP. Simple, well >>>>>>>>> supported. Then if you really find out that >>>>>>>>> FASTCGI >>>>>>>>> is >>>>>>>>> needed for your slow OS and machine, PHP supports >>>>>>>>> it. >>>>>>>>> >>>>>>>>> The point is your SOLUTION is not FASTCGI, it is >>>>>>>>> the >>>>>>>>> actual application that does the work. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> HLS >>>>>>> Joseph M. Newcomer [MVP] >>>>>>> email: newcomer(a)flounder.com >>>>>>> Web: http://www.flounder.com >>>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> >> > > > > -- > HLS
From: Joseph M. Newcomer on 18 Mar 2010 13:18
See below... On Wed, 17 Mar 2010 14:21:36 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:fn42q59m8n702cpis6t409le60pdog0tft(a)4ax.com... >> See below... >> On Tue, 16 Mar 2010 21:06:35 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>message >>>news:eyHxkRXxKHA.2644(a)TK2MSFTNGP04.phx.gbl... >>>> Peter Olcott wrote: >>>> >>>>> >>>>> That is not simple enough for my users. I want a browse >>>>> button on a webpage that browses the local hard-drive. >>>> >>>> >>>> <INPUT type="File" name="filename" /> >>>> >>>>> It seems that the only completely portable way to do >>>>> this >>>>> is HTML and this HTML is hooked up to CGI. Maybe I >>>>> could >>>>> build something from scratch at the other end to handle >>>>> the input. I would guess that this would not provide >>>>> any >>>>> improvement over C++ FastCGI, and cost more development >>>>> time. >>>>> I am leaning toward Linux now, so a Microsoft thing >>>>> ported to Linux may not be as good as a native Linux >>>>> thing. >>>> >>>> Geez, you ain't going to get nothing done with this lost >>>> focus on "FastCGI". >>>> >>>> Once again, all you need is a standard C/C++ console >>>> application that reads standard input to read your HTTP >>>> transfer encoded data block. You either write the >>>> multi-part MIME processor or you get a library or class >>>> that does it for you. >>> >>>What handles transmission errors? Another thing, I want to >>>immediately close the connection on the first packet if >>>anything besides a 24-bit PNG file is detected. My app >>>MUST >>>be written in C++. I was envisioning this as a single app. >>>One that handles the URL-encoded input, and the same one >>>to >>>process this data. In any case the one that processes this >>>data must remain resident. >> **** >> You clearly missed any comprehsension of TCP/IP. TCP/IP >> is as reliable as a piece of wire >> between the two sites, and that essentially means it is >> 100% utterly reliable, or it has >> failed completely and there is no recovery (read about how >> TCP/IP retransmission works!) >> so there are *no* "transmission errors" at all, or you >> will get a notification that the >> protocol has failed utterly and there is NO recovery from >> that, so browsers deal with it! >> I have no idea what you are talking about when you talk >> about "clos[ing] the connection on >> the first packet" since, among other things, packets are >> invisible at the application >> level and you NEVER see them, you may or may not get all >> the bits you want in one Receive > >OK so all this is new to me, We were talking about sockets >which does deal with packets. **** No, we were talking about TCP/IP,w hich deals solely and exclusively in BYTE STREAMs, and packets are a low-level implementation detail NEVER seen by the application socket programmer! Noplace in any book on socket protramming does it tell you that TCP/IP delivers packets to you; they indicate the I/O is stream oriented e.g., Comer & Stevens, volume III, page 55, (one of the standard reference texts on TCP/IP socket programming that anyone who expects to do it should own), or Quinn & Shute page 66 (another fundamental book every socket programmer should own) explicitly explain all this, very carefully. Packets are a *transport* mechanism for TCP/IP, and their existence is completely gone by the time an application starts talking to a socket. In this case, any good introductory book on TCP/IP protocol (including things like buffer management piggybacking on ACK/NAK protocols, the sliding window protocol, etc. are discusssed in any goog introductory book on network programming) and the conversion to streams is also discussed. The role of blocking on send and recv are dsicussed, and they are not related to packets but to buffer management on the sender and receiver. > >> (it is a STREAM transmission, and therefore you simply >> CANNOT tell where packet boundaries >> are or anything else, and MUST assume that it will take 2 >> or more Receives to get what you >> need, and you have not identified what you mean by a >> "24-bit PNG file is detected". You > >It would be located in the first 1K bytes of the file >header. If I was dealing with packets, then I could examine >the first one and potentially reject the rest. **** Streams are what they are, and you cannot presume you can get ANY specific size on the receiver side! Maybe you can get 1K bytes, and maybe you get 986, or 852, or 711, or 254. There's no way to predict, control, or rely on any particular size being received. You have to assume there will be arbitrary and unpredictable breaks in the data, and what is received as a single recv is in no way correlated with the size of a send (for example, you "send" 8192 bytes, but your first recv gives 1056 bytes, and if so, this should not be seen as surprising; you may have to execute several recv calls to get that 8192 bytes, pehaps as many as a dozen, perhaps as few as 2. Or maybe even 1; unpredictable.) I have no idea why you have developed this fascination with "packets" since there is nothing in the streaming socket API (that is, TCPI/IP) that recognizes their existence. This is why it is worthwhile to do a little research before jumping to conclusions, e.g., that TCP/IP packets have any meaning at the application level. You should probably read my sample code for a multithreaded TCP/IP server from my MVP Tips site. joe **** > >> must be living in an alternate plane of existence wheere >> there is some magic that >> determines that because some magical bits appear at the >> front of a byte sequence that the >> rest of the sequence must necessarily conform to that >> purported reality, and that in spite >> of the fact that TCP/IP *guarantees* delivery of bits >> intact that you might accidentally >> see the wrong bits after the client makes a transmission >> (this will never happen; TCP/IP >> doesn't even allow for it!) which goes back to the advice >> I gave weeks ago, about reading >> an introductory text on how TCP/IP actually works. There >> will be no "transmission errors" > >This might now be moot because it looks like I will be >working at the higher level of HTTP, thus it seems that I >will not be directly dealing with the TCP/IP. Hector has >got me refocused on the webserver point of view. That seems >fruitful. There is a very good freeware webserver that I can >embed in my OCR processor, called mongoose. **** Note that you still have to address how to send files up from the client, and establish the protocol you will use, the encoding, etc., so you are well on your way using a custom server, having solved one (and only one) of the numerous problem that you will have to address. **** > >> that garbage the message; either the message is received >> correctly or a fake message has >> been sent, and if you have a fake message, you close the >> connection and that's the end of >> it. You go back and wait for the next connection. If you >> were using PHP and mime, you >> would reject the data if it were not the correct mime >> image, or, having read it, the >> conversion to an image encountered a problem. It would >> not be a transmission error; it >> would be the result of erroneous data having been sent, >> because you will receive >> everything that was sent, perfectly, or you will be >> notified that there has been an >> unrecoverable network error (as will the transmission >> side). This is inherent in the >> TCP/IP protocol. > >OK good. So this low level details of this now seem out of >the current scope of the project. *** Yes. They would have been ruled out long ago if you had taken the time to read some good books on TCP/IP and HTTP protocols. joe *** > >> >> So forget the concept of "first packet" or even "packets". >> They are nonsense at the >> TCP/IP application level. There is one, and only one, >> reality: the byte sequence. If you >> continue to believe in inappropriate (and erroneous) >> concepts like "packets" when using >> TCP/IP, you are DOOMED! They only have a reality at the >> lowest levels of the protocol, >> and at the application level, they have been erased >> entirely from your awareness. People >> who believe in anything other than the byte stream >> eventually have problems. > >OK great good to have that cleared up. > >> >> And please abandon your concepts of "resident", since you >> have demonstrated beyond any >> possible doubt that you simply do not understand how >> operating systems work. You have >> these strange wishful-thinking models that have no >> realization in any form of reality most >> of us are familiar with, and you assume that operating >> systems work in accordance to your >> dreams and not the way they are actually implemented. > >I think that the 1000-fold faster than OCR performance that >I achieve contradicts you on this. I can process an entire >display screen in 1/10 second on a machine 800% slower than >my current machine. This could not be achieved if my DFA was >not at least almost completely contained with actual >hardware memory. Also the 800% speed up is almost entirely >attributed to faster access to physical RAM. **** Have you tried the experiment after letting the program sit idle for 30 minutes? I think you will see a completely different behavior pattern if you exceed the working-set-paring theshold time. **** > >> **** >>> >>>> >>>> If you write a PHP script, it will do it for you. You >>>> can >>>> put this under a FASTCGI PHP setup under a certain port >>>> and then write a HTTP client do do your testing. >>>> >>>> However, I don't believe PHP naturally supports memory >>>> maps unless you find a public PHP class to expose the >>>> WIN32 memory map functions. >>>> >>>> The WIN32 API has all the memory mapping functions and >>>> you >>>> will need this to handle the 4GB file. >>>> >>>> Now, on the wienie side, I don't know what it offers for >>>> memory mapping. But I am sure it has something >>>> equivalent. >>>> >>>> -- >>>> HLS >>> >>>I have no idea what you are talking about when referring >>>to >>>a memory map. It is not a 4.0 GB file, it is numerous >>>files >>>making up a total of 4.0 GB. >> **** >> Note that you cannot read a 4GB file into any WIndows app, >> since you only have 2GB (or >> perhaps 3GB) of address space to use, and not all of that >> is available (your code, your >> heap, and your stack, or each thread's stack, occupy the >> same address space). Therefore, >> you will HAVE to use a memory-mapped file if you wish to >> access more data than you can >> read. He is only pointing out the only possible >> implementation that will let you access >> 4GB of data effiiciently. If you think you can access 4GB >> of data, you can't even do that >> in a Win32 app running in Win64 (which has a 4GB address >> space, but not all of it can be >> used for your data! See previous comment). If you can >> currently read it all in, then you >> don't have 4GB. >> joe > >I have seen windows report that my 32-bit app was using > >4000 MB of RAM. >8.0 GB of Ram 64-bit Windows 7, and /LARGEADDRWESSAWARE >This instance 4.0 GB is supposed to be the limit. > http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx#memory_limits > **** A 32-bit program cannot use more than 4GB of virtual memory because it cannot address more than 4GB of virtual memory. Any storage beyond this usage is probably used by the 64-bit system to hold paging tables and other process-related resources, and as such would count in the "memory footprint" of a 32-bit app running in a Win64 enviornment. But that space is not avialble to your program, because your Win32 program is limited to being able to name only 4GB of locations. joe **** > > >> >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |