I am trying to separate out the different types of errors that can occur when using Python's sockets module to perform internet requests so that I can handle them appropriately. Particularly, the case when no internet connection is available, which I would like to handle so that I can check and wait for availability.
The requests I am dealing with are via URLs, at the core of which is socket's getaddrinfo() function, which, primarily, resolves a hostname to an IP. For remote hosts this involves a DNS query, which obviously does not work without the internet, and hence is the first point of failure if no internet connection is available.
However, it seems that getaddrinfo() raises the same exception for both "no internet" (i.e. DNS query send failed or timed out) and a non-existent domain (i.e. internet is available and DNS query/answer completed) - and as a result I cannot detect the no internet condition that I need.
For example, running the following non-existent domain with an internet connection:
socket.getaddrinfo("www.thsidomaindoesntexist12309657.com", 80)
The resulting exception is:
socket.gaierror: [Errno 11001] getaddrinfo failed
Where errno 11001 corresponds to socket.EAI_NONAME. This is the expected behaviour, since the domain actually does not exist.
Yet when I try the following existing domain with no internet connection (network adaptor disabled):
socket.getaddrinfo("www.google.com", 80)
I get exactly the same exception as before, apparently indicating that the domain does not exist (even though we can't know because we never got a DNS response).
How can I detect no internet connection when using sockets and getaddrinfo()? Is it impossible, or is this a bug I am experiencing, or something else?
How can I detect no internet connection when using sockets and getaddrinfo()
You cannot get the information from getaddrinfo. This functions is an abstract interface to name resolution which could be done against a local DNS server but is also often done against a DNS server in the local network. The function provides no information why it failed to resolve a name since it often has no idea itself why the resolving failed.
Apart from that even if the DNS lookup succeeds there is no guarantee that there is actually a working internet connection. It might be that the result of the DNS lookup is taken from cache or it might be that one could actually resolve the name but that the actual connection to the target (like a web server) is blocked by some firewall. It might also be that one gets an IP address but that it is the wrong one as is often the case with captive portals.
The common way to find out if there is a working internet connection is to contact a known good server with a specific request and check if the response is as expected.
Related
I have build a node js code for an API server. Part of one feature is that when it starts, it should be able to know its own IP, despite the type of setup of the server where it is running.
The classic scenario is not that hard (I think). There are several options, like using the os module and find the ip or the external interface. I am sure there are other ways and some might be better, but this is the way I have been doing it so far. Feel free to add alternatives as informative as possible.
There is this case that I stumbled on. In one case, the web server was running on a google cloud instance. This instance has two IPs, one internal and one external. What I want is the external IP. However, when I use the method above, the actual external IP is not part of the object returned. The internal IP is declared as being considered as non-internal. Even when I run different commands from within the server command line, the only IP returned is the one that is actually internal and cannot be used to access the node server.
From what I understand, the instance itself is not aware of it's external IP. There might be a dns (I think) that redirects requests made to the external IP towards the correct instance.
While reading in the internet I read that problems getting the server's correct external IP might also rise when using load balancing or proxies.
The solution I thought about is to have the node js code make a request towards a service that I will build. This service will treat the node js servers as clients, and will return their external IPs. From experiments that I have done, the req object contains among others the information of the client's IP. So I should check first req.connection.remoteAddress and then the first element of req.headers['x-forwarded-for']. Ideally the server would make a request towards itself, but
I know there are external API like https://api.ipify.org?format=json that do just that - return the actual IP. But I would very much like to have the node js servers independent of services I cannot control.
However, I really am hoping that there are better solutions out there than making a request from the server which returns the server IP.
However, I really am hoping that there are better solutions out there
than making a request from the server which returns the server IP.
It is not really possible, you always rely on some kind of external observer / external request.
While reading in the internet I read that problems getting the
server's correct external IP might also rise when using load balancing
or proxies.
This is because not in all scenarios your own device is able to be self-aware of its external ip. There might be sitting behind some network, that means external address assigned to devices that forwards the WAN to it. (example : router) so when you try to obtain external ip from the devices interface itself, you end up obtaining an ip but inside the scope of the routers LAN and not the one used for external requests .
So if you really want to
Have a method to use in all scenarios
Not rely on 3rd party services
Only Solution :
Build your own ip echo service (you maintain and can use for future projects).
I am working in a Linux Debian 7 distribution.
I programmed a very simple procedure to test onc-rpc.
The 'remote' call works fine when I call it this way:
test_client localhost
However, when I invoke it by calling a remote server IP:
test_client 202.170.91.155
I get an error message:
202.170.91.155: RPC: Port mapper failure - RPC: Unable to receive
I am just learning and I know almost nothing about portmapping in Linux.
The IP I was trying to call for the service is the IP address of the same machine. I speculate the rpcbind daemon does the necessary mapping for the server when you register a new service, but I really have no clue.
Which steps should I take in order to fix this error?
In the end, the issue has nothing to do with programming.
This kind of error may appear when the rpcbind daemon (In charge on giving information about the port where the requested service is listening) does not give a response to the calling machine.
This can happen due to errors in NAT or firewalling.
So, a first attempt to diagnose the problema, as with anything regarding networking, could involve verifying the connection is fine by making ping to every interface involved, from closest to furthest, making sure each jump works fine.
For instance, in the issue decribed above: Pinging the supposed internal server ip address (it failed) helped to discover that the machine was using an IP other than the intended (the one mapped to receive request to port 111).
Stopping and restarting eth0 configuration with ifdown/ifup correctly updated ipv4 info, effectively making reachable the rpc service.
We use a Visual Studio Online-hosted build server to automate our build process. As part of this I'm looking into adding unit and integration tests into this process.
These tests require access to our SQL Azure DBs (2 of them, both on the same server), which in turn requires access through the DB server's firewall.
I have a PowerShell script which uses New-AzureRmSqlServerFirewallRule to add IP addresses to the DB server, and these firewall rules are successfully showing up in the Azure portal.
Specifically, the script adds firewall rules for:
All IPv4 addresses* on the build server (as returned by Get-NetIPAddress)
Build server's external IP address (as returned by https://api.ipify.org)
In conjunction, it appears that the pre-defined AllowAllAzureIPs and AllowAllWindowsAzureIps rules are automatically added.
However, the tests subsequently fail with the exception:
System.Data.SqlClient.SqlException:
System.Data.SqlClient.SqlException: A network-related or
instance-specific error occurred while establishing a connection to
SQL Server. The server was not found or was not accessible. Verify
that the instance name is correct and that SQL Server is configured to
allow remote connections. (provider: Named Pipes Provider, error: 40 -
Could not open a connection to SQL Server)
I'm unsure why the build server is unable to reach the DB server - could it be that the host of the test processes is using yet a different IP address?
Update
As has been pointed out, the exception message mentions "Named Pipes Provider" which suggests that the DB connection is using a named pipe instead of an IP/TCP connection. To test this I changed the local app.config to contain an unknown/random/inaccessible IP and ran the tests locally (they otherwise run successfully locally): I received exactly the same exception message mentioning "Named Pipes Provider". Perhaps at some level the ReliableSqlConnection class resolves to a named pipe but my point is that I can induce this very same exception by changing to an unknown or inaccessible IP address in my DB connection string.
Furthermore, the DB connection string starts with tcp: which, as per this blog post, explicitly tells the connection to use TCP/IP and not named pipes.
I have also modified the firewall rule to permit all IP addresses (0.0.0.0 to 255.255.255.255) but the same exception is still thrown. This suggests that the SQL Azure firewall rule is not the cause of the 'blockage'.
My suspicion therefore turns to network access being blocked (though a whitelist is probably present to permit the build server to reach the code repository). I added a very simple PowerShell script to the start of the build process:
Test-Connection "172.217.18.100" #resolves to www.google.com
This results in
Testing connection to computer '172.217.18.100' failed: Error due to lack of resources
Have the build servers disabled ping/ICMP or is all outgoing traffic blocked?
* The script only considers IPv4 addresses because I haven't had any success in passing IPv6 addresses to New-AzureRmSqlServerFirewallRule.
We finally solved the issue. The problem had nothing to do with Firewalls. The issue was that the app.config files in our unit test didn't go through the transformation step that our web.config files did. So all the settings were from our local development and therefore wrong.
More about this here:
Connect to external services inside Visual Studio Online build/test task
What connection string are you using? Your error seems to indicate that this is not truly a firewall issue, but rather a connection is being attempted to a server that doesn't exist.
My * incorrect * hypothesis right now is that your connection string contains only the server name, without .database.windows.net suffix which causes the client driver to look for server on local network. The error presented appears to not be a firewall related issue.
( Edited to reflect author feedback. )
If you're connecting over TCP, then why is your error message saying Named Pipes?
[...]
(provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
I'd look into this paradox first.
The firewall test is very simple, allow 0.0.0.0 to 255.255.255.255 or 0.0.0.0/0 and re-test. My money is on the same error message.
I have a misconfigured heroku website. It shows error 104 (Read Error: Connection reset by peer) upon typing its URL and hitting enter. But subsequently refreshing the URL a couple of times makes the URL load correctly (some kind of fallback kicks in? - not that I knowingly configured any). The URL is http://damadam.in/ (it's a naked domain).
I bought this domain from godaddy. In Godaddy's control panel where I have the DNS Zone file, the host www points to damadam.herokuapp.com (under CName). http://damadam.in is set to forward to http://www.damadam.in. Lastly, in my heroku control panel both http://damadam.in and http://www.damadam.in have damadam.herokuapp.com as the DNS target (could this last configuration be the problem)?
Can someone help me properly set this thing up?
This is not a http response code, but rather an error number indicating something was wrong with the connection.
"Connection reset by peer" means that, on the route from your computer to the final destination, a node decided to forcefully stop and reset the connection. On a configuration level I don't think you will be able to do much about this. If there was some kind of DNS misconfiguration, you would not see a read error, but a DNS Error instead.
Make sure that your local network is stable (e.g. connect to your modem with an ethernet cable, rather than through wifi). If this connection is stable, try again at a later date. Connections between nodes can break, and in some cases not all traffic might be able to reach the intended destination. If behaviour persists through a greater length of time, contact your host, in this case Godaddy, and ask them to look into this problem. It might be just a faulty piece of equipment
I'm writing a piece of P2P software, which requires a direct connection to the Internet. It is decentralized, so there is no always-on server that it can contact with a request for the server to attempt to connect back to it in order to observe if the connection attempt arrives.
Is there a way to test the connection for firewall status?
I'm thinking in my dream land where wishes were horses, there would be some sort of 3rd-party, public, already existent servers to whom I could send some sort of simple command, and they would send a special ping back. Then I could simply listen to see if that arrives and know whether I'm behind a firewall.
Even if such a thing does not exist, are there any alternative routes available?
Nantucket - does your service listen on UDP or TCP?
For UDP - what you are sort of describing is something the STUN protocol was designed for. It matches your definition of "some sort of simple command, and they would send a special ping back"
STUN is a very "ping like" (UDP) protocol for a server to echo back to a client what IP and port it sees the client as. The client can then use the response from the server and compare the result with what it thinks its locally enumerated IP address is. If the server's response matches the locally enumerated IP address, the client host can self determinte that it is directly connected to the Internet. Otherwise, the client must assume it is behind a NAT - but for the majority of routers, you have just created a port mapping that can be used for other P2P connection scenarios.
Further, you can you use the RESPONSE-PORT attribute in the STUN binding request for the server to respond back to a different port. This will effectively allow you to detect if you are firewalled or not.
TCP - this gets a little tricky. STUN can partially be used to determine if you are behind a NAT. Or simply making an http request to whatismyip.com and parsing the result to see if there's a NAT. But it gets tricky, as there's no service on the internet that I know of that will test a TCP connection back to you.
With all the above in mind, the vast majority of broadband users are likely behind a NAT that also acts as a firewall. Either given by their ISP or their own wireless router device. And even if they are not, most operating systems have some sort of minimal firewall to block unsolicited traffic. So it's very limiting to have a P2P client out there than can only work on direct connections.
With that said, on Windows (and likely others), you can program your app's install package can register with the Windows firewall so your it is not blocked. But if you aren't targeting Windows, you may have to ask the user to manually fix his firewall software.
Oh shameless plug. You can use this open source STUN server and client library which supports all of the semantics described above. Follow up with me offline if you need access to a stun service.
You might find this article useful
http://msdn.microsoft.com/en-us/library/aa364726%28v=VS.85%29.aspx
I would start with each os and ask if firewall services are turned on. Secondly, I would attempt the socket connections and determine from the error codes if connections are being reset or timeout. I'm only familiar with winsock coding, so I can't really say much for Linux or mac os.