Client Server and Throttling
in [MFC]
|
From: Joseph M. Newcomer on 5 Dec 2009 19:22 See below... On Sat, 05 Dec 2009 17:48:15 -0600, TomChapman <TomChapman12(a)gmail.com> wrote: >Joseph M. Newcomer wrote: >> See below... >> On Fri, 04 Dec 2009 22:49:52 -0600, TomChapman <TomChapman12(a)gmail.com> wrote: >> >>> I have written a client/server application set using TCP as the link. >>> Normally only a few packets are exchanged every minute. One thing the >>> client does is ask the server for a specific database record. The server >>> retrieves the record and sends it back to the client. In some odd cases >>> the client may all-of-a-sudden realize that it needs thousands of these >>> records. >>> >>> The obvious method is to send one request wait for the return data and >>> then send the next request. That would work. However... >>> >>> I'm thinking I'd get faster results for the client if I tried to stay >>> ahead by sending requests early so there were a few queued at the server >>> or in transit. >> **** >> Yes, this sounds like a good idea >> **** >>> I don't know the bast way to do this. How do I know when to send a new >>> request. An individual client does not know what load the server is >>> under. It may vary. >> **** >> Then the requests will simply wait at the server until it gets around to them. I don't >> see a problem here. >> **** >>> I'm thinking I need some kind of counter. What is the best approach >>> here? How should I tackle this throttling problem. >> **** >> You could limit the number of outstanding requests from any given client, but as you point >> out, that's a simple counter. But I'd be inclined to a more laissez-faire approach (free >> market) and just dump out all the requests; if the server is running slow, they take >> longer to process, but so what? You're going to need them anyway, so just queue them up >> and let the server deal with the issue. >> joe >> **** >>> >>> ---------- >>> Question 2: >>> >>> In many of my programs I seem to spend more time handling errors and >>> what-ifs then I do with nominal situation code. I'm use to thinking >>> about what-ifs. >>> >>> So in the case where I am sending thousands of request-for-data >>> messages. In the perfect world, the server would correctly respond and I >>> would receive 100% of the responses. But I always worry about what-ifs. >>> Specifically in this case, what if something goes wrong somewhere and I >>> never get a response. If I'm queuing multiple requests and using some >>> kind of counter, the counter might get out-of-wack if a packet >>> here-or-there was never responded to. How do I handle this situation? >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > >Just for my knowledge... > >What happens when two clients send data to the same server. I know each >connected client gets a separate derived CAsyncSocket class, each with a >separate OnReceive. I know that TCP guarantees in-order arrival at the >server. So I'm sure each client's data will be in-order. > >My question is: > >Say one client sends a thousand packets which will take the server some >time to process. While this is happening a second client then sends a >bunch of packets. > >Will my program process all of the packets from the first client first, >since they arrived first, or will the first client's OnReceive be >interspersed with OnReceive calls from the second instance for the >second client? **** In general, it is up to the server as to how it processes requests. A multithreaded or logically multithreaded [asynchronous] ("concurrent") server will handle the packets in some unspecified interlacing. An iterative server will handle the packets in FIFO order. So it depends on the architecture of your server. If I have two CAsyncSockets running, they will each signal an OnReceive when data has been received. Because of TCP flow control, there is no reason to suspect that, let's say 1000 requests from client A, have all arrived before any request for client B. Part of this depends on the details of the implementation of the IP stack; for example, are all buffers from a common pool or does each connection have a quota of its own? Overall, there are a lot of variables here, not all of which I have deep knowledge about. My expectation is that they will be interspersed. Note that if your server requires a fair amount of time to handle a request, it suggests that the requests should be handled by threads; whether the sockets are in the threads or the threads are simply handle as a queue of service threads for sockets in the main GUI thread is another dimension of the design. Short of actual experimentation, there's no good way to determine the behavior without having a good model of the client and the server. Back in the days when I was doing simulations of these systems (1968 or so) our biggest problems in doing accurate simulations of loads was the actual details of system behavior. In some cases, our models were far off because the details of the implementation were unknown to us. joe **** > >I could see good arguments in either way. One side of the coin would be >to process in the order received, which sounds logical. But on the other >side of the coin, it might be good to intersperse so that one client >doesn't monopolize the server. **** Now you are into the "QOS" (Quality of Service) domain. If you want guarantees about service, you have to implement your own policy mechanisms to achieve it. Which in general you cannot do unless you understand where the service bottlenecks are, which goes back to the issues of the architecture of your server, and the architecture of the TCP/IP stacks. I'd favor a multithreaded server here. If you have to "throttle" requests, it needs to be done at the server, because only the server knows how many clients and how many requests they have. For example, if client A has > 100 requests, the requests that come in > 100 go in with a lower priority, so you have to build a priority queue to handle the incoming requests. You can base it either on total pending requests, or requests/time with various priority-decay functions applied (> 100 requests in 1 minute starts shoving the requests down in priority, for example). But the clients have no way of knowing the server load, and shouldn't have to. If the server wishes to "throttle" a client, then you need a way for the server to say "hold back" (and I'm now talking policies above the TCP/IP quota management) to the client, and again this has to be implemented on the server, although, of course, the client then has to respond to such requests. The general solution is not trivial, and you would have to have "tuning" parameters that decided your policy, which would be set on the server. I've done such things in the past, but the answer is always "only the server can know", because only the server sees all the mix of requests. A client cannot a priori know that the server is "overloaded". 1000 requests if it is the only client active may pose no problem; 1000 requests if there are 1000 clients active gets into a whole new area. Since scaling is often nonlinear, you can't create a solution for 10 clients that works well if there are 10,000 clients (although you might successfully create one that has tunable parameters that allow scaling). joe **** Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 5 Dec 2009 22:20 See below... On Sat, 5 Dec 2009 09:45:04 -0800, "Alexander Grigoriev" <alegr(a)earthlink.net> wrote: >For what it's worth, even microsoft managed (of course) to get it wrong in >their server.sys. > >See, for example, KB 297019. When outlook PST files are located on a remote >server over a slow WAN link, and many users are accessing the files at once, >the server may run out of paged pool, or hang. **** It does seem strange. Paged pool is limited only by the size of the pagefile.sys file for paging, and that normally expands as needed. Also, it would be unusual for many users to access a single .pst file; the article talks about multiple users accessing multiple .pst files. It also appears to not be an Outlook problem, but an Exchange problem. Exchange is one of those mysterious programs whose purpose is unknown and unknowable; all the documentation I can find tells me (a) it is cool and (b) how to install and configure it, but nobody tells me why I would ever care about it. I went to several Exchange presentations at different PDCs, but nobody would tell me what Exchange did (although they could tell me in great detail what various switches did during the install). I've put it in the same category as SAP (did you ever see a SAP ad? It tells you that it will make your business run better, but never says what the product is or does!) The article does discuss running out of *non*-paged pool, which is somewhat more believable, but since the non-paged-pool limit is very high, it is unlikely it would happen in small sites. The article suggests that the problem arises only for "enterprise-wide" attempts at the solution, suggesting issues of scale. Note that it does not involve a single server, but instead involves using a file server using remote-file-access commands; the OP is suggesting using a different model entirely, a server *process*. This is quite different than the KB article describes. **** > >This tells that most likely, server.sys does NOT limit or throttle total >amount of data pending for transmission, thus, when it gets a lot of >requests at once, it will try to queue excessive amount of data, possibly >exceeding available memory. **** Note that server.sys is not going to be the program involved in the OP's problem, and that the exhaustion of non-paged pool is caused by the kernel, and in particular, kernel work items created in response to low-level file requests. The OP is not doing it this way, and therefore server.sys does not appear to be involved in the discussion at all. joe **** > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message >news:dhrkh5p6tqr1fae6sqrm0rcetdra025r17(a)4ax.com... >> See below... >> On Fri, 04 Dec 2009 22:49:52 -0600, TomChapman <TomChapman12(a)gmail.com> >> wrote: >> >>>I have written a client/server application set using TCP as the link. >>>Normally only a few packets are exchanged every minute. One thing the >>>client does is ask the server for a specific database record. The server >>>retrieves the record and sends it back to the client. In some odd cases >>>the client may all-of-a-sudden realize that it needs thousands of these >>>records. >>> >>>The obvious method is to send one request wait for the return data and >>>then send the next request. That would work. However... >>> >>>I'm thinking I'd get faster results for the client if I tried to stay >>>ahead by sending requests early so there were a few queued at the server >>>or in transit. >> **** >> Yes, this sounds like a good idea >> **** >>> >>>I don't know the bast way to do this. How do I know when to send a new >>>request. An individual client does not know what load the server is >>>under. It may vary. >> **** >> Then the requests will simply wait at the server until it gets around to >> them. I don't >> see a problem here. >> **** >>> >>>I'm thinking I need some kind of counter. What is the best approach >>>here? How should I tackle this throttling problem. >> **** >> You could limit the number of outstanding requests from any given client, >> but as you point >> out, that's a simple counter. But I'd be inclined to a more laissez-faire >> approach (free >> market) and just dump out all the requests; if the server is running slow, >> they take >> longer to process, but so what? You're going to need them anyway, so just >> queue them up >> and let the server deal with the issue. >> joe >> **** >>> >>> >>>---------- >>>Question 2: >>> >>>In many of my programs I seem to spend more time handling errors and >>>what-ifs then I do with nominal situation code. I'm use to thinking >>>about what-ifs. >>> >>>So in the case where I am sending thousands of request-for-data >>>messages. In the perfect world, the server would correctly respond and I >>>would receive 100% of the responses. But I always worry about what-ifs. >>>Specifically in this case, what if something goes wrong somewhere and I >>>never get a response. If I'm queuing multiple requests and using some >>>kind of counter, the counter might get out-of-wack if a packet >>>here-or-there was never responded to. How do I handle this situation? >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Alexander Grigoriev on 6 Dec 2009 00:32
Kernel paged pool is limited by the reserved area of address space, though Win2008 gets rid of an explicit limit. X64 doesn't suffer from it pretty much. My point was that with large load, one has to be careful how (when) the resources are allocated for the requests. If a system receives large number of requests and has to keep their resources (allocated buffers) for long time (because of slow uplink), it can easily cause denial of service condition. The described problem was really neither Outlook's nor Exchange's. It just appeared when many users were accessing their non-local PST files on a single fileserver over a slow WAN. "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:508mh5l9apqpgk4bvqctv67rrr297muffk(a)4ax.com... > See below... > On Sat, 5 Dec 2009 09:45:04 -0800, "Alexander Grigoriev" > <alegr(a)earthlink.net> wrote: > >>For what it's worth, even microsoft managed (of course) to get it wrong in >>their server.sys. >> >>See, for example, KB 297019. When outlook PST files are located on a >>remote >>server over a slow WAN link, and many users are accessing the files at >>once, >>the server may run out of paged pool, or hang. > **** > It does seem strange. Paged pool is limited only by the size of the > pagefile.sys file for > paging, and that normally expands as needed. > > Also, it would be unusual for many users to access a single .pst file; the > article talks > about multiple users accessing multiple .pst files. > > It also appears to not be an Outlook problem, but an Exchange problem. > Exchange is one of > those mysterious programs whose purpose is unknown and unknowable; all the > documentation I > can find tells me (a) it is cool and (b) how to install and configure it, > but nobody tells > me why I would ever care about it. I went to several Exchange > presentations at different > PDCs, but nobody would tell me what Exchange did (although they could tell > me in great > detail what various switches did during the install). I've put it in the > same category as > SAP (did you ever see a SAP ad? It tells you that it will make your > business run better, > but never says what the product is or does!) > > The article does discuss running out of *non*-paged pool, which is > somewhat more > believable, but since the non-paged-pool limit is very high, it is > unlikely it would > happen in small sites. The article suggests that the problem arises only > for > "enterprise-wide" attempts at the solution, suggesting issues of scale. > Note that it does > not involve a single server, but instead involves using a file server > using > remote-file-access commands; the OP is suggesting using a different model > entirely, a > server *process*. This is quite different than the KB article describes. > **** > >> >>This tells that most likely, server.sys does NOT limit or throttle total >>amount of data pending for transmission, thus, when it gets a lot of >>requests at once, it will try to queue excessive amount of data, possibly >>exceeding available memory. > **** > Note that server.sys is not going to be the program involved in the OP's > problem, and that > the exhaustion of non-paged pool is caused by the kernel, and in > particular, kernel work > items created in response to low-level file requests. The OP is not doing > it this way, > and therefore server.sys does not appear to be involved in the discussion > at all. > joe > **** >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message >>news:dhrkh5p6tqr1fae6sqrm0rcetdra025r17(a)4ax.com... >>> See below... >>> On Fri, 04 Dec 2009 22:49:52 -0600, TomChapman <TomChapman12(a)gmail.com> >>> wrote: >>> >>>>I have written a client/server application set using TCP as the link. >>>>Normally only a few packets are exchanged every minute. One thing the >>>>client does is ask the server for a specific database record. The server >>>>retrieves the record and sends it back to the client. In some odd cases >>>>the client may all-of-a-sudden realize that it needs thousands of these >>>>records. >>>> >>>>The obvious method is to send one request wait for the return data and >>>>then send the next request. That would work. However... >>>> >>>>I'm thinking I'd get faster results for the client if I tried to stay >>>>ahead by sending requests early so there were a few queued at the server >>>>or in transit. >>> **** >>> Yes, this sounds like a good idea >>> **** >>>> >>>>I don't know the bast way to do this. How do I know when to send a new >>>>request. An individual client does not know what load the server is >>>>under. It may vary. >>> **** >>> Then the requests will simply wait at the server until it gets around to >>> them. I don't >>> see a problem here. >>> **** >>>> >>>>I'm thinking I need some kind of counter. What is the best approach >>>>here? How should I tackle this throttling problem. >>> **** >>> You could limit the number of outstanding requests from any given >>> client, >>> but as you point >>> out, that's a simple counter. But I'd be inclined to a more >>> laissez-faire >>> approach (free >>> market) and just dump out all the requests; if the server is running >>> slow, >>> they take >>> longer to process, but so what? You're going to need them anyway, so >>> just >>> queue them up >>> and let the server deal with the issue. >>> joe >>> **** >>>> >>>> >>>>---------- >>>>Question 2: >>>> >>>>In many of my programs I seem to spend more time handling errors and >>>>what-ifs then I do with nominal situation code. I'm use to thinking >>>>about what-ifs. >>>> >>>>So in the case where I am sending thousands of request-for-data >>>>messages. In the perfect world, the server would correctly respond and I >>>>would receive 100% of the responses. But I always worry about what-ifs. >>>>Specifically in this case, what if something goes wrong somewhere and I >>>>never get a response. If I'm queuing multiple requests and using some >>>>kind of counter, the counter might get out-of-wack if a packet >>>>here-or-there was never responded to. How do I handle this situation? >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |