CURL cannot connect to URL - IP address - after successful connection
in [General]
|
Prev: [RCU] [bug of pecl pam ] I try to implement the changepassword plugin
Next: CURL cannot connect to URL - IP address - aftersuccessful connection
From: "ioannes on 25 Apr 2010 09:17 I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. The target URL is an ip address, not a named url, so maybe it has something to do with DNS. I am on a shared server. Any ideas on why this happens? John
From: "Tommy Pham" on 26 Apr 2010 04:58 > -----Original Message----- > From: ioannes(a)btinternet.com [mailto:ioannes(a)btinternet.com] > Sent: Sunday, April 25, 2010 6:18 AM > To: php-general(a)lists.php.net > Subject: [PHP] CURL cannot connect to URL - IP address - after successful > connection > > I can return a target page - once, but then on refresh within a few hours the > script curl_error is that it cannot connect to the host and return is empty. > The target URL is an ip address, not a named url, so maybe it has something > to do with DNS. I am on a shared server. Any ideas on why this happens? > > John > Just to eliminate all possibilities, are you to open the same URL/URI in the web pages repeatedly? Also, what happens when you fake the user agent in the web browser? The target site may have some anti bot mechanism in place to reduce stress/load on the server(s). Regards, Tommy
From: "Tommy Pham" on 26 Apr 2010 05:14 > -----Original Message----- > From: Tommy Pham [mailto:tommyhp2(a)gmail.com] > Sent: Monday, April 26, 2010 1:59 AM > To: 'php-general(a)lists.php.net' > Subject: RE: [PHP] CURL cannot connect to URL - IP address - after successful > connection > > > -----Original Message----- > > From: ioannes(a)btinternet.com [mailto:ioannes(a)btinternet.com] > > Sent: Sunday, April 25, 2010 6:18 AM > > To: php-general(a)lists.php.net > > Subject: [PHP] CURL cannot connect to URL - IP address - after > > successful connection > > > > I can return a target page - once, but then on refresh within a few > > hours the script curl_error is that it cannot connect to the host and return is > empty. > > The target URL is an ip address, not a named url, so maybe it has > > something to do with DNS. I am on a shared server. Any ideas on why this > happens? > > > > John > > > > Just to eliminate all possibilities, are you to open the same URL/URI in the > web pages repeatedly? Also, what happens when you fake the user agent in > the web browser? The target site may have some anti bot mechanism in > place to reduce stress/load on the server(s). > > Regards, > Tommy One more thing, check it with cookies enabled/disabled in the web browser too.
From: "Tommy Pham" on 26 Apr 2010 12:13 > -----Original Message----- > From: ioannes(a)btinternet.com [mailto:ioannes(a)btinternet.com] > Sent: Sunday, April 25, 2010 10:44 PM > To: ash(a)ashleysheridan.co.uk; tommyhp2(a)gmail.com >> Tommy Pham > Subject: Re: [PHP] CURL cannot connect to URL - IP address - after successful > connection > > The answer I got from support desk on my shared server: 'You are trying to > curl to a datapipe server, if it is rejecting the server name and port, you will > need to take that up with them.' > > John I assume that you did full testing with the browser as I suggested? If everything works, one other thing to keep in mind is that the target also may implement reverse DNS lookup in their anti-bot. One good way to test that is to remote in via SSH (if on Linux/Unix) to test with wget. Otherwise, I'm pretty sure that target site have some anti-bot mechanisms in place. Microseconds of analyzing valid 'user' requests is better than processing 2-3 seconds and sending the response which will consume bandwidth. What you could also try is setting different user-agents for every request and use cookies in cURL should the target site have an anti-bot mechanism. Regards, Tommy
From: "Tommy Pham" on 26 Apr 2010 19:57
> -----Original Message----- > From: ioannes(a)btinternet.com [mailto:ioannes(a)btinternet.com] > Sent: Monday, April 26, 2010 7:10 AM > To: Tommy Pham > Subject: Re: [PHP] CURL cannot connect to URL - IP address - after successful > connection > > > > On 2010/04/27 1:13, Tommy Pham wrote: > >> -----Original Message----- > > > I assume that you did full testing with the browser as I suggested? > > If everything works, one other thing to keep in mind is that the > > target also may implement reverse DNS lookup in their anti-bot. One > > good way to test that is to remote in via SSH (if on Linux/Unix) to test with > wget. > > Otherwise, I'm pretty sure that target site have some anti-bot > > mechanisms in place. Microseconds of analyzing valid 'user' requests > > is better than processing 2-3 seconds and sending the response which > > will consume bandwidth. What you could also try is setting different > > user-agents for every request and use cookies in cURL should the > > target site have an anti-bot mechanism. > > > > Regards, > > Tommy > > > > > Yes, I think I have tested with/without cookies on the browser, trying > different user agents (code emailed previously using array and rand) and > cookies are used in script/not used. And it works on subsequent requests? > > I cannot work out how to use Putty/ssh/public private keys etc..wget... > Learning how to use that is easier than learning to code PHP, IMO. > I read about some problems with curl setting the port and a required patch > on the server. > > John If cURL requires some kind of patch as you say, then it wouldn't have work in the first place. Perhaps it's better to post your (obfuscated personal data) code. Or try on your local PC on your local web server to eliminate possibility of proxies, anti-bots, etc... problems to test that your code works as intended and not cURL problem as you say. I didn't have problems using cURL before. But then my targeted sites were very big companies and didn't care about bots much. Regards, Tommy PS: Always reply to the list so others in the future can benefit unless it's something personal. |