How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Peter Olcott on 16 Mar 2010 23:49 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OPmpeAYxKHA.3304(a)TK2MSFTNGP06.phx.gbl... > Peter Olcott wrote: > > >>> For Peter, I doubt this is the design issue. Handling >>> the 4 GB of meta data files is the key thing he needs to >>> work out. He erroneously thinks keep the EXE "resident" >>> will solve this performance problem. He has no choice >>> but to use memory map files and now that he said he need >>> to load all them into a vector, he need a class that >>> offers vector MMF virtualization. >>> >>> We have our own CFileMap class which is major part of >>> our server large file database framework. I'm sure >>> there other public classes out there. Here is an Old >>> Microsoft CMemoryMappedFile MFC subclass based off >>> CMemFile: >>> >>> http://support.microsoft.com/kb/142377 > > >> Why can't I simply read in the files in the normal way? > > > To load all of the 4GB into memory? > > Try it. > > Unless you going completely 64 bit, you will be limited to > 32 bit boundaries, and 4GB is the limit. You won't be able > to load of it all even if you tried. I already know that, I am currently running 64-bit with 8.0 GB. I won't need 4.0 GB until I have the same configuration on my webserver. Until then I will probably only use 1.0 GB at most. > > So you open it like a file and use it as a "Vector" or > array but in mapped segments where only part of the file > is mapped to memory not all of it. Its called a Mapped > View. You can read and write to it like it as was pure > memory. The MMF cache memory manager will handle it. Yeah, I can't do that. That would reduce my performance by 100-fold. I don't use MFC or Win32 for this, I simple use a binary FILE. I want all of the data to be (as much as possible) resident in RAM at all times. Great surpluses of RAM have consistently provided this as measured by the performance that could not be otherwise achieved. In fact my recent 800% improvement is speed can be almost entirely attributed to faster access to memory. > > The purpose of the above MMF class is to allow you to > "read/write" in a near-like-normal way hiding all the > WIN32 details for you. > > Study it Peter. > > I already do that using FILE. I have never bothered with any non portable stuff unless it could not be otherwise achieved. > -- > HLS
From: Peter Olcott on 16 Mar 2010 23:54 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:08i0q513cr5paa3m0mkvlmaoecd9vu7oqu(a)4ax.com... > See below... > On Tue, 16 Mar 2010 17:28:45 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>message >>news:ea%23kdFVxKHA.5576(a)TK2MSFTNGP05.phx.gbl... >>> Peter Olcott wrote: >>> >>>> >>>> I would be doing FastCGI in C++, if I do it. I am >>>> considering FastCGI because I need a completely >>>> portable >>>> way for a user to browse for a PNG file to send to my >>>> server. HTML has this built in, and it is connected to >>>> CGI. As far as a protocol goes, this would be trivial. >>>> The user sends a 24-bit PNG file, the server sends back >>>> UTF-8. That's the whole protocol. Sometimes the UTF-8 >>>> is >>>> the PNG file translated into text, and other times it >>>> is >>>> an error response. >>> >>> Doesn't sound you understand, or at least are showing >>> you >>> don't, what FastCGI is. >>> >>> Forget fastcgi, what you need is a dedicated http server >>> with the built-in procssing or a http server that >>> supports >>> spawning your cgi EXE application as a special URL. >> >>I can't take the several minutes of CGI process spawn time >>for my application, (it has to load gigabytes of data into >>memory before it begins) thus conventional CGI is >>infeasible. > *** > So, in other words, if I happen to be so unfrotunate as to > make a request on your server > and there IS no instance of the process, that in fact, I > cannot expect 500ms reponse time > but instead can expect MINUTES of delay! > > NOte that all FASTCGI promises is that your process does > not exist or require a restart > once it has been launched; it does not promise the pages > will be "held in memory". > joe In actual practice operating systems typically do not page out code or data unless they need room for something else. This will be the only application on the dedicated server, thus little reason to be paged out. If necessary I will run a very tiny little background process that forces the app to do something once in a while. > > ***** >> >>> >>> You will handle standard INPUT as the encoded data. >>> Simple. >>> >>> Example: >>> >>> int main(char argc, char *argv[]) >>> { >>> DoCGI(); >>> return 0; >>> } >>> >>> where DoCgi() is reads stdin and maybe environment >>> strings. >>> >>> int DoCGI() >>> { >>> // get environment string, if required >>> // read stdin as FORM POSTING encoding data >>> // process, generate response >>> >>> printf("Status: 200\n"); >>> printf("Content-Type: text/text\n\n"); >>> printf"UTF8:%s\n",szResponse); >>> return 0; >>> } >>> >>> For FastCGI, you would prepare your own socket listening >>> server for incoming request from the web server itself. >>> >>> -- >>> HLS >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 16 Mar 2010 23:55 Peter Olcott wrote: > It seems to me that most of you guys continue to simply > ignore some of my most fundamental design requirements. App > load time for me can be several minutes, app execution time > of a large job is never more than 100 milliseconds. Thats because you didn't know about NT old technology called Memory Mapped Files. > The web service needs no GUI. It is all PNG in, and UTF-8 > out. A web service is just a HTTP protocol processor. What it does is based on the URI or specifically script-path: <scheme> "://" <server-name> ":" <server-port> <script-path> <extra-path> "?" <query-string> But the I/O is the same across the board. > All that I know for sure is that some of my fundamental > requirements are immutable. Some of Hector's advice seemed > to continue to ignore these requirements. Until you mentioned the 4GB I realized your layman thinking why you believe FastCGI is your answer - it isn't. You need to use Memory Mapped Files and until you realized this everything else is a moot point. > I was envisioning the web service as a single application. This > may not have been the way that he was envisioning it. Of course I was because I know YOU were not be too sophisticated but to do it any other way and this is why I said your reasons for FASTCGI were not incorrect. When you say "FASTCGI", you are talking about a "service" that the web server connects too. Not the client. I know you won't be able to do it because short of using PHP, you need to write a C/C++ client/server that does everything a CGI application will do - standard I/O. Again I ask, what web server will be using? If you are will be writing your own WEB SERVER (HTTP processor), you will be created a embedded solution - you won't need FastCGI. Now, as I mentioned, the old reason was slower machines and process startups. That isn't the case today. Today, FASTCGI is not as important and also most systems are going to the alternative - embedded solutions. So again, what web server? PS: Please quote the mail better, no need to quote the entire message for one or two lines. -- HLS
From: Peter Olcott on 17 Mar 2010 00:02 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:82j0q59uv9oml0nqtdh16421tclt4f3jm5(a)4ax.com... > See below... > On Tue, 16 Mar 2010 19:50:03 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>message >>news:e7e1pVWxKHA.948(a)TK2MSFTNGP05.phx.gbl... >>> Peter Olcott wrote: >>> >>>> >>>> I can't take the several minutes of CGI process spawn >>>> time for my application, (it has to load gigabytes of >>>> data into memory before it begins) thus conventional >>>> CGI >>>> is infeasible. >>> >>> >>> Peter, FASTCGI vs CGI, the different is in load time. >>> In >>> the past, that was an ISSUE, not today. You won't be >>> saving any time whatsoever with FASTCGI. >> >><sarcasm> >>Ah so I can load 4 GB of data from disk in less than a >>millisecond, great then I will go with regular CGI. >></sarcasm> > **** > Really? A 10,000 RPM hard drive holds s small number of K > bytes per track, and a track > takes 0.0001 seconds to pass under the read heads. That's > 0.1 ms per track; so under > minimum fragmentation you can read maybe 5 or 10 tracks > without a seek 9this is based > onhow many surfaces the drive has, which is typically > 5-10), and a seek usually takes > aboutg 7-8ms, so I find it less than believable that you > can load 4GB in 1ms; the disk > simply cannot deliver data at that rate! So you did not notice the <sarcasm> markers? > **** > \> >>Each transaction will only take a few milliseconds iff (if >>and only if) the 4 GB DFA is already loaded into memory. >>If >>I have to load it every time, then each transaction will >>take infeasibly long. > **** > You are once more assuming that you have a clue that > modern Web servers still have the > same performance problems they had a decade ago. Hector > says it ain't so, and since he is > closer to the technology than I am, I tend to believe him. > For example, does the phrase > "process pool" convey meaning to you? > joe I am speaking about the fundamental requirements of my patented OCR system. I envision the Web Service to be a single contiguous piece of compiled C++ code including my OCR system in this single piece. My patented process is very quick, only after it loads its DFA(s) from disk. > > **** >> >>> >>> If your transaction residence time is over 20-30 >>> seconds, >>> you got other problems with HTTP uploads - i.e. browser >>> uploads. >>> >>> You will need a special client to support a keep alive >>> concept otherwise the browser will think something >>> happen. >>> >>> You would also be do this via AJAX (jQuery with FORMS >>> pluggin) and work in a "progress bar" but unless you >>> Ping/Poll hit the server, you have will timeout issues. >>> >>> -- >>> HLS >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 17 Mar 2010 00:15
"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OJiTrWYxKHA.6140(a)TK2MSFTNGP05.phx.gbl... > Peter Olcott wrote: > > >> It seems to me that most of you guys continue to simply >> ignore some of my most fundamental design requirements. >> App load time for me can be several minutes, app >> execution time of a large job is never more than 100 >> milliseconds. > > > Thats because you didn't know about NT old technology > called Memory Mapped Files. > >> The web service needs no GUI. It is all PNG in, and UTF-8 >> out. > > > A web service is just a HTTP protocol processor. What it > does is based on the URI or specifically script-path: > > <scheme> "://" <server-name> ":" <server-port> > <script-path> <extra-path> "?" <query-string> > > But the I/O is the same across the board. > >> All that I know for sure is that some of my fundamental >> requirements are immutable. Some of Hector's advice >> seemed to continue to ignore these requirements. > > > Until you mentioned the 4GB I realized your layman > thinking why you believe FastCGI is your answer - it > isn't. You need to use Memory Mapped Files and until you > realized this everything else is a moot point. > >> I was envisioning the web service as a single >> application. This > > > may not have been the way that he was envisioning it. > > Of course I was because I know YOU were not be too > sophisticated but to do it any other way and this is why I > said your reasons for FASTCGI were not incorrect. > > When you say "FASTCGI", you are talking about a "service" > that the web server connects too. Not the client. I know > you won't be able to do it because short of using PHP, you > need to write a C/C++ client/server that does everything a > CGI application will do - standard I/O. > > Again I ask, what web server will be using? > > If you are will be writing your own WEB SERVER (HTTP > processor), you will be created a embedded solution - you > won't need FastCGI. > > Now, as I mentioned, the old reason was slower machines > and process startups. That isn't the case today. Today, > FASTCGI is not as important and also most systems are > going to the alternative - embedded solutions. > > So again, what web server? I will probably be using some version of Apache because I will probably be hosting this under one flavor of Linux or another. (1) My App must remain alive it can not terminate, it must keep at least most of its data in memory, even if I have to write another process to force this. (2) I will probably be using the HTML BrowseForFile button as the user interface, thus the PNG file will be coming as URL-encoded CGI format. (3) The processing of the URL encoded data back into a PNG data file must be reasonably fast. (4) The implementation of this system must take relatively little time, the least amount of time needed to meet the requirements. > > PS: Please quote the mail better, no need to quote the > entire message for one or two lines. > > -- > HLS |