How hard is socket programming?
in [MFC]
|
Prev: Runtime Error does not specify program???
Next: windows 7 - imitating touch by sending wm_gesture ?
From: Peter Olcott on 20 Mar 2010 00:11 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:2bg8q5p3k3pf1fdgnsvm5fv566jins4kuf(a)4ax.com... > See below... > On Fri, 19 Mar 2010 19:21:17 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>message >>news:OdDHnT7xKHA.5480(a)TK2MSFTNGP06.phx.gbl... >>> Peter Olcott Asked: >>> >>>> If I can not reject a file at the HTTP level, then I >>>> have >>>> to work at a lower level. In the ideal case I can >>>> receive >>>> the file size before any of the rest of the file is >>>> sent, >>>> and reject is with only a few bytes of wasted >>>> bandwidth. >>> >>> >>> You will get the size (Content-Length:) of the HTTP >>> request body from the HTTP request header block. >>> >>> You can issue an error at that point after receiving the >>> header and before receiving the body. This is >>> described >>> in HTTP 1.1 standard >> >>This also includes all lower levels (TCP/IP, et cetera) of >>any data transmitted over HTTP? >> >>I imagine that there could be an underlying buffer that >>another lower level protocol uses, such that when HTTP >>sees >>the first 20 bytes, this lower level protocol has already >>eaten up 100K of my bandwidth quota. > **** > You are completley missing the point here, once again > getting hung up on irrelevant > concepts such as "buffers" and "packets". I repeat, you > have NO CONTROL over any of this, > which is purely magic. Your app gets a stream. YOu have > NO IDEA how much of that stream I have to take control of this to prevent denial of service attacks, even if I have to go to a much lower level than HTTP. From speaking with Hector, it looks like the lowest reasonable level is sockets. The next higher level that seems to make the most sense is HTTP. > is sitting in buffers in your server already; it might be > 276 bytes, and it might be 100K > bytes, and it is nothing you need to worry about. That's > what shutdown() is for: it says I want to prevent malicious users from eating up my service providers bandwidth allocation. I would like to do this in an optimal way. I do not yet understand the technology quite well enough to determine what this way would be. > "kill off any incoming buffered data I haven't received > yet and don't bother receiving any > more" and "make sure any outgoing buffers are sent NOW!" > (each of these is indicated by a > 1-bit flag that forms the parameter value of shutdown(). > You are obsessing over > irrelevancies here. And go read about TCP/IP buffer > management and packet flow, and then > realize that NONE of these aspects of TCP/IP are remotely > visible to you as a TCP/IP > programmer, even if you are ignoring HTTP level and > wrirting raw socket code! For HTTP, > you are typically going to get the whole thing that was > sent, before the protocol bothers > to inform you that something has arrived; the "something" > is an atomic entity. It is > going to feed it as a stream to stdin, and you will have > NO IDEA what packet magic, MTU > boundaries, etc. are involved in getting that stream to > your app. So just go do it, and > stop worrying about cases that are not going to be any > problem in practice. It is simply > MANDATORY that your server will validate every detail of > the file format, including > illegal PNG encodings; anything else is a minor > implementation detail, and should be below > your radar. I was thinking that someone might be sending me multiple GB files in a tight loop to eat up my bandwidth budget. It just occurred to me that this would also cost them bandwidth too. > > Yes, bad transmissions use up bandwidth. Technically, we > refer to this using the phrase > "life is hard". Meaning, tough, suck it up and live with > the fact.Until there is > convincing proof that this is a serious issue, it is not > worth worrying about. Note that > you can create a "blacklist" of IP addresses that you > refuse to accept connections from, > and if you do this, realize that these addresses (used by > crackers who are trying to break > your program) are probably spoofed addresses of > potentially legitimate users. Life is > hard.. You can "age" the blacklist so repeated trials from > the same script kiddie will > probably be rejected right away, but in case they spoofed > a legitimate address, it will > become legal after N hours, for some N of your choice > (N=24, N=24*k for k a number of > days, are good initial ideas for aging parameters). Sounds like a good idea. > joe > **** > joe > **** >> >>> RFC 2068, section 8.2 Message Transmission Requirements: >>> >>> http://www.ietf.org/rfc/rfc2068.txt >>> >>> This is a feature of a HTTP 1.1 server, not HTTP 1.0 >>> server which generally requires the entire payload to be >>> received first. >>> >>> You HTTP 1.1 web server needs to do this very carefully, >>> otherwise it can cause resends by the clients. >>> >>>> After I determine that the file is not too large I then >>>> get however many minimal bytes are required to >>>> determine >>>> the file type, and then reject the rest of the file if >>>> it >>>> is not 24-bit PNG. >>> >>> >>> Only HTTP 1.1 clients will gracefully support a >>> mid-stream >>> reject by the web server. Otherwise, resends can occur. >>> >>> In other words, if the client is using HTTP 1.0, you >>> will >>> see that in the first line of the HTTP request header >>> block, you could either reject the usage of this client >>> or >>> ignore the fact the user will see irregular >>> "disconnected" >>> error pages. >>> >>> -- >>> HLS >> >>Since HTTP 1.1 has been around for 14 years I may simply >>reject all HTTP 1.0 calls and request the user update to a >>newer browser. >> >>Or I could force the user to use some sort of java applet >>that sends the data using the HTTP 1.1 format. This client >>side code could also verify the size and type of the file >>(specifically 24-bit PNG) before anything is sent. It >>could >>also provide a file search dialogbox that only looks for >>PNG >>files. With this scenario I would no longer be limited to >>HTTP, I could devise my own protocol that could strip off >>some of the extra HTTP baggage. >> >>Would I be back to sockets again if I did this? Would >>this >>be programming at the TCP/IP level or some other level? >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 20 Mar 2010 00:16 Peter Olcott wrote: > I just want to understand how the handle the most malicious > user in the most robust way. If I can block an IP address > without costing me any bandwidth, then I think I have one > significant aspect of blocking the most malicious user. As joe puts it, you are "obsessing" again :) Its very simple with the server model: s = socket() - create a socket handle bind(s) - associate an local IP with socket s handle listen(s) - starting to listen to connection accept(s,c) - wait for clients, new socket c for connect CHECK PEER IP ADDRESS OF C IN FILTER LIST, IF FOUND, CLOSE SOCKET GOTO BACK TO ACCEPT recv(c) - receiver whatever send(c) - send whatever shutdown(c) - tell remote I'm about to close closesocket(c) - close socket handle go back to accept -- HLS
From: Hector Santos on 20 Mar 2010 00:20 Peter Olcott wrote: > I want to prevent malicious users from eating up my service > providers bandwidth allocation. I would like to do this in > an optimal way. Are you kidding me? Don't you have in your business plan to get your own WEB SERVER and not use GoDaddy? -- HLS
From: Joseph M. Newcomer on 20 Mar 2010 00:26 See below... On Fri, 19 Mar 2010 21:45:41 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%23OJgTK9xKHA.5940(a)TK2MSFTNGP02.phx.gbl... >> Peter, >> >> When it comes to the internet, you can only expect >> compliance with good guys. You can't stop bad-guys. They >> won't listen to you, your rules or anyone elses. The only >> thing you can do is issue a disconnect (close the socket, >> "hangup the telephone") and monitor the IP for redundant >> bad behavior, then block the IP. >> >> -- >> HLS > >If I configure my (java applet based) protocol as I have >explained in another message, I might even be able to >prevent the bad guys from impacting me very much. If the >first six bytes are not what I expect, I disconnect. The >signed (thus unmodifiable) java applet enforces the >proprietary protocol, and no other protocol can produce the >correct first six bytes. (It might take a few more bytes). *** Not at all clear! If I wanted to take your site out, I'd use your applet, use a packet ssniffer to see what it sent, and reverse-engineer something based on that. Note that I can read you Java byte codes, JavaScript code. etc. so you will have no secrets from me. I agee with what Hector is saying, ten times over: the Bad Guys are going to mess with you, and all robustenss engineering has to be in the protocol and in the server. If you use HTTP, you have to validate the alleged PNG file bits ANYWAY. There's nothing the client side can do to increase your server robustness. Seriously, unless you are going for VPNs or SSL or its successors (SSL Is rather dated), there's nothing that is going to happen on the client side that is going to make the server side easier to do. Eveyone lies: especially incoming data. joe **** > >I am guessing (and I think that you said) I can also reject >abusive IP addresses. Hopefully this requires little >overhead. Possibly I can have the hosting provider reject >them for me, thus costing zero of my bandwidth budget. **** If I want to take you out, I will not use the same IP address twice in a row. Look up the concept of "IP spoofing". Old technology, most attack scripts support it intrinsically. YOu keep assuming the world is "well behaved" according to your fantasies, well, it ain't so, and putting any server out on the INternet is a huge invitation that says "hit me!" and they will. WIth sucker punches. We have ways to avoid this. Not trusting anyone, watching your back at all times, trust no one, all work well. Heavy armor isn' a bad idea (we call them firewalls). but sitting there and saying "nobody will do anything bad to me" isn't viable today. Crackers are professionals, often funded by major governments (most attacks originate in two servers in mainland China and one in Bulgaria, and the attackers are not script kiddies or twelve-year-olds witth their first modem. They are organied crime and/or state-sponsored cyberterrorists. And they will come after your site. ***** > >> >> Peter Olcott wrote: >> >>> OK great is this before or after my bandwidth quota has >>> been hit with the full data load? >>> >>> In other words could a lower level protocol such as >>> TCP/IP send a large portion of the file before my HTTP >>> gets a chance to reject it because it is "denial of >>> service attack" size? >> >> > >> >>> Alternatively could I somehow structure my code so that I >>> only get just the bytes indicating the size and reject >>> all other data before eating up any more of my bandwidth >>> quota? >>> >>> example I only see these bytes---->"size(1000000000)" and >>> reject the file before any additional bytes hit my >>> bandwidth quota. >>> >>> NOTE the only valid data sent to my web application will >>> be a single 24-bit PNG file that can vary in size up to a >>> predetermined limit of something like 10K. >> >> > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 20 Mar 2010 00:30
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:kqh8q5h7k00484tpkjle30vsjqi25fbo83(a)4ax.com... > See be;ow... > On Fri, 19 Mar 2010 17:20:55 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>message >>news:%23s1kB76xKHA.4240(a)TK2MSFTNGP06.phx.gbl... >>> Peter Olcott wrote: >>> >>>> It looks like I am going to be using HTTP as the >>>> protocol. I just bought two books on it. I am >>>> estimating >>>> that the TCP/IP is mostly invisible at the HTTP layer. >>>> I >>>> am using the HTTP protocol because it seems that the >>>> HTML >>>> element that I am using, sends the data using this >>>> protocol. >>>> <input name="userfile" type="file" accept="image/png" >>>> /> >>>> >>>> Using HTTP is it possible to reject a file that is the >>>> wrong format before the entire file is sent? >>> >>> >>> The HTTP client will send the type in the HTTP request >>> BODY block: >>> >>> Content-type: image/png >>> >>>> Using HTTP is it possible to reject a file that is too >>>> large before very much of this file is sent? >>> >>> >>> The HTTP request will have a HTTP request header: >>> >>> Content-length: >>> >>> But that length is the entire HTTP body following the >>> request header block, i.e. you can send two or more >>> uploads. You can use this length for a rough value, but >>> to get the actual image size you need to parse the HTTP >>> BODY to get the type and size. >>> >>> Note: Not all browsers will send a size in the body >>> block, >>> like if its only 1 image. Therefore you use the top >>> HTTP >>> request block Content-Length: header. >>> >>> >>> -- >>> HLS >> >>OK great is this before or after my bandwidth quota has >>been >>hit with the full data load? > **** > You are obsessing again. If you want to use HTTP, you > have to live with its limitations. > If you want to implement your own protocol, you can abort > a partially-completed > transaction just by closing the connection when you decide > that the data is invalid. This > means you have to not just "accept" the data AS IF it is > valid PNG encoding, but you have > to VERIFY that it makes sense at every step. > > But as long as you keep obsessing about details, you will > make no progress. Either you > accept that HTTP can do the job (and that you will > sometimes get an upload that isn't > valid) or you have to implement your own port# and > protocol spec, and this allows you to > abort at any time, but note that SOME amount of data has > already been received. Suck it Yes this is my second level design. I want to be able to handle malicious attacks in the most robust way. For example the first few bytes of data would be encrypted using a public key/ private key pair, and if it is not authenticate close the connection. This would be implemented (on the client side) using a signed java applet. > up and accept that this is reality. It happens. Every Web > server in the world has to deal > with this, and most of them deal with it by ignoring the > problem because they don't really > care in the slightest. If you care, you pay a price: you > have to write more complex > server code and you have to write complex validation code. > This is simply reality. > joe > **** >> >>In other words could a lower level protocol such as TCP/IP >>send a large portion of the file before my HTTP gets a >>chance to reject it because it is "denial of service >>attack" >>size? > *** > Sure. Live with the idea. You aren't going to be able to > change this behavior! > **** >> >>Alternatively could I somehow structure my code so that I >>only get just the bytes indicating the size and reject all >>other data before eating up any more of my bandwidth >>quota? > **** > If someone wants to screw you, they will send you bogus > packets until they saturate your > server or your bandwidht quota. We all them "script > kiddies" and they are a real problem, > but you can't make them go away by changing your protocol. > They'll get you one way or > another. I want to make this as close to impossible as I can. I envision the possibility of (for example) a competitor doing this to shut me down. I want to implement the most cost-effective way to reduce this risk. Maybe the main answer is to require all users to have an account, even the free trial users. This account can only be set up from a valid email address. Some form of bot protection would also be used. That may prove to be an entirely sufficient answer. I want to have these things completely thought through before I begin. It is much cheaper to do this now than it will be when I have thousands of users depending on my service. > **** >> >>example I only see these bytes---->"size(1000000000)" and >>reject the file before any additional bytes hit my >>bandwidth >>quota. > **** > If you write your own protocol handler--as I did for my > multithreaded socket example--then > you can do anything you want within certain buffer > boundaries; you might have 8K or 15K or > 64K data already received that the shutdown() will > discard, tough. that's life. Deal > with it. > **** >> >>NOTE the only valid data sent to my web application will >>be >>a single 24-bit PNG file that can vary in size up to a >>predetermined limit of something like 10K. > *** > Then you are truly obsessing over a problem not worthy of > concern. Just build it. > joe > **** >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |