Failure on local socket bind when wifi drops - node.js

We are getting this strange issue on a raspberry pi.
We run a service on a socket that should work for both local and remote clients via wifi.
The trouble is that stopping the remote network also stops connections from local clients.
Our python server sets up a socket like this:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_DONTROUTE, 1)
s.settimeout(2)
s.bind(("", 8888))
while True:
try:
conn, addr = s.accept()
except socket.timeout:
print("Socket timeout on s.accept(), continuing")
continue
#do stuff
We have a local node client running a loop like this every second or so (and actually sending data):
// every second
socket.connect("localhost", "8888" );
socket.on('connect', function() { /* do stuff */ });
socket.on('error', function(ex) { });
Everything runs fine until we cut wifi.
We server side times-out on s.accept and we see the error message in our logs.
I think that the socket is bound to listen on 0.0.0.0 but somehow does not fail over to 127.0.0.1 or some sort of strange routing situation occurs.
netstat -an | grep 8888 gives
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8888 127.0.0.1:52794 TIME_WAIT
tcp 0 0 127.0.0.1:8888 127.0.0.1:52724 TIME_WAIT
tcp 0 0 127.0.0.1:8888 127.0.0.1:52740 TIME_WAIT
tcp 0 0 127.0.0.1:8888 127.0.0.1:52778 TIME_WAIT
netstart -rn gives
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.1.1 0.0.0.0 UG 304 0 0 wlan0
192.168.1.0 0.0.0.0 255.255.255.0 U 304 0 0 wlan0
I'm guessing that we just need a localhost route?
The local connections establish again when the wifi comes back up. So I don't there is some permanent dropping of the bind in the python socket.
the hosts line in /etc/nsswitch.conf gives
hosts: files mdns4_minimal [NOTFOUND=return] dns
We monitored ping to localhost during the test and it continues to function fine.
We also monitored netstat to see that port stays LISTENING on 0.0.0.0 Perhaps this is the issue?

Easiest Solution
It looks like you should avoid any naming by using "127.0.0.1" as described in our comment discussion.
In more detail:
According to the source and the docs (after nodejs first tests for an ip,) it checks if you've provided a lookup function as an option to connect, if not, it does its own "dns.lookup" call as the default. Despite the name, this function is actually trying to use system naming but might be subtly different, for example it may try to prefer ipv6.
To debug further you could try to make a more direct test case with dns.lookup and compare things like the output of getent ahosts|ahostsv4|ahostsv6 localhost against your different systems and when the wifi is down, as well as comparing other configuration like the gai.conf to try to determine if system naming is a bit different on this system or being given slightly different requests.

Related

Netplan ipv6 "No route to host"

I've clearly misconfigured my ipv6 listener addresses. Question 1: I need someone to spot the error. And a bonus Question 2: is there any risk to removing the "/sbin/ip address add" command below?
This is on a ubuntu 18.04 virtual machine.
Here's a curl command, showing that ipv6 routing is not set up correctly:
curl -vvv -L "http://[2600:1303:d000:1::17c3:4571]"
* Trying 2600:1303:d000:1::17c3:4571...
* TCP_NODELAY set
* Immediate connect fail for 2600:1303:d000:1::17c3:4571: No route to host
* Closing connection 0
curl: (7) Couldn't connect to server
Previously, when the program started, it issued a native "/sbin/ip address add". That turned out to be problematic because when the network interface was reset, the route would be lost.
So, I added /etc/netplan because configuring the ipv6 routing survives a network restart. However, I didn't remove the "ip addr add" on startup. Currently, two commands are used to setup routing: netplan and "/sbin/ip address add". Here's the order of commands executed:
1. Manual one-time command of "/etc/netplan apply" which didn't return any errors.
2. "ip addr add" every time the server starts. So, at least once since the manual netplan command.
Here's the ipv6 addresses it's listening on:
netstat -anp | grep redir | grep LISTEN
tcp6 0 0 2600:1303:d000:1::17:80 :::* LISTEN 3187/my-service
tcp6 0 0 2600:1303:d000:1::17:80 :::* LISTEN 3187/my-service
tcp6 0 0 2600:1303:d000:1::17:80 :::* LISTEN 3187/my-service
tcp6 0 0 2600:1303:d000:1::17:80 :::* LISTEN 3187/my-service
tcp6 0 0 2600:1303:d000:1::17:80 :::* LISTEN 3187/my-service
Here's what I want it to be listening on:
[2600:1303:c000:1::15d4:456f]:80"
[2600:1303:d000:1::17c3:4570]:80"
[2600:1303:d000:1::17c3:4571]:80"
[2600:1303:d000:1::17c3:4572]:80"
Here's the "ip address add" command. Notice that the I'm using "/24" and I wonder if that's the problem. Or perhaps using both "ip addr add" and netplan isn't working as intended. This is a legacy application and because my networking skills in this area aren't strong, I wasn't sure if it would be safe to remove the "ip addr add". I didn't think it would hurt to keep the "ip addr add" command, but perhaps I was wrong.
out, e = exec.Command("/sbin/ip", "-6", "addr", "add", ip + "/24", "dev", "eth0").CombinedOutput()
Here's my netplan config file. Note that this is only part of the file; for security reasons I'm not including the mac address, name servers or gateway. However, they are correct because my ipv4 addresses work. Also, I haven't posted the real ipv6 addresses for security reasons as well.
root#ubuntu:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
ethernets:
eth0:
addresses: [ '2600:1303:d000:1::17c3:456e/24', '2600:1303:d000:1::17c3:4570/24', '2600:1303:d000:1::17c3:4572/24', '2600:1303:d000:1::17c3:4571/24', '2600:1303:d000:1::17c3:456f/24' ]
Thanks in Advance ;)
There were a couple of problems:
There was no gateway6.
The bit mask should have been "/64" instead of "/24"
Netplan wouldn't delete the old routes. I had to do following steps:
1. ip -6 addr del 2600:1303:d000:1::17c3:456f/24 dev eth0
2. netplan apply

How do I change a listening rule with netcat?

I have a server that has a rule like so when I check netstat -tulpn:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -
This has been all well and good as the mysql database we've use has only needed local access, but now I want to remote connect to it too. I need my netstat -tulpn to read like so:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN -
How do I update this? I've been looking through nc but everything seems to only talk about setting up listening for the first time. Whenever I try to assign a new rule I get a message like this:
(base) ct#do-not-touch:/$ sudo nc -l 0.0.0.0 3306
nc: Address already in use
How do update the 127.0.0.1:3306 rule to be for 0.0.0.0:3306? Am I somehow totally off base here?
You can't* have two processes listening on the same IP and port. Since 0.0.0.0 means listen on all IPs, you can't bind to it if there's anything bound to any IP on that port. Stop the existing process listening on 127.0.0.1:3306, and then you'll be able to start one listening on 0.0.0.0:3306.
*: Technically, a flag called SO_REUSEPORT does exist, and forking after binding lets you do it as well, but neither is useful in your situation.
Ok so if you run into this in the future, you have kill the process that is using whatever port but first you must know that port.
sudo lsof -t -i:3306
Then kill whatever id.

Linux port blocked - This site can't be reached, refused to connect

I set my linux as an access point, and then run simple web-server that print "hello world" at port 3000.
and connect it with my smart phone successfully.
in linux terminal, http://localhost:3000 works well.
But in smart phone,
If I access to http://172.24.1.105:3000, can't connect to it. (172.24... is ap's ip)
the chrome's error message is
This site can't be reached. 172.24.1.105 refused to connect
I searched Google (https://serverfault.com/questions/725262/what-causes-the-connection-refused-message) and I suspicious linux's firewall.
pi#raspberrypi:~/prj/ap_server $ sudo tcpdump -n icmp
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
... when I access to port 3000,
15:07:13.102750 IP 192.168.0.3 > 168.126.63.2: ICMP 192.168.0.3 udp port 42531 unreachable, length 386
the log is above. so I couldn't reach ap's webserver.
so I wonder two things...
1. How can I disable to its port block?
2. in tcpdump log, I access to port 3000 actually, why the log print port 42531?
Plus)
even I type sudo service iptables stop, the problem is not solved
sudo netstat -ntlp | grep 3000 logs:
**tcp6 0 0 :::3000 :::* LISTEN 1999/nodejs**
+I followed this tutorial-> https://frillip.com/using-your-raspberry-pi-3-as-a-wifi-access-point-with-hostapd/ .
and there is ipv4 setting.
If you want to run it on your mobile it will work on Live IP (externel) address
if it is working fine on local address (localhost) and not on live IP then
enable routing from your router
and allow that specific port it will work fine.
I found the issue.
my dhcp set was
interface=wlan0 # Use interface wlan0
listen-address=172.24.1.1 # Explicitly specify the address to listen on
bind-interfaces # Bind to the interface to make sure we aren't sending things elsewhere
server=8.8.8.8 # Forward DNS requests to Google DNS
domain-needed # Don't forward short names
bogus-priv # Never forward addresses in the non-routed address spaces.
dhcp-range=172.24.1.50,172.24.1.150,12h # Assign IP addresses between 172.24.1.50 and 172.24.1.150 with a 12 hour lease time
like above.
I tried to connect the external ip(172.24.1.105) that I can see on mobile continuously but got failed. but when I tried with 172.24.1.1, then success.
I don't know why. maybe there is accurate ip address and something in mobile is temporal.
See similar topic at Node JS not listening to port 1337 on server
Your web server is not listening remote address.

Listening to EVERY port on a machine

For testing purposes I want to build a server that listens for TCP connections to every port (or at least on most of the ports) on a certain interface (e.g., eth0). The server is accessed via SSH to eth1, so no problem there. (I do not care about UDP or other protocols)
I want to do a special kind of middlebox detection/analysis, therefore I need to be able to fully establish a connection. Having an HTTP connection would be the best, because the "client" could be implemented as JS in the Browser.
I started with a simple jetty server, but had to realize that jetty needs to spawn at least on thread per port it is listening to. This leads to problems when I want to listen to several thousand ports. Or is there a way around that?
My next try was to use iptables:
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp -j DNAT --to-destination 127.0.0.1:8080`
It seemed to work. It allows to connect on every port and the traffic gets routed to the local port 8080 where jetty listens. But now I no longer know which port was used by the client. Because jetty thinks the connection was established via port 8080. Is there a way to determine the real incomming port from jetty? I could send the port as part of the HTTP request, but if the client tries to contact port 1234 .. and a middlebox redirects this to port 5678 .. I am unable to know what port was used.
I also tried userland solutions like socat. The problem was even worse than before. Because now jetty also saw the remote IP as being 127.0.0.1.
Or, is there another way to achieve this?
Oh and btw: I have full control of the machine. So I could change the kernel or whatever is needed. Right now I use Ubuntu 14.04 LTS, but if a solution needs something else I could go with that.
NB: This is a Python solution, because I know Python, but you can accomplish the same thing in any language the exposes the underlying C library getsockopt call.
If you replace your DNAT rule with a REDIRECT rule, you can then
use getsockopt with the SO_ORIGINAL_DST option to retrieve the
original address of a REDIRECT-ed connection.
Consider the following code:
#!/usr/bin/python
import socket
import struct
SO_ORIGINAL_DST = 80
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0', 2000))
s.listen(10)
while True:
csock, caddr = s.accept()
orig_dst = csock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
orig_port = struct.unpack('>H', orig_dst[2:4])
orig_addr = socket.inet_ntoa(orig_dst[4:8])
print 'connection from', caddr
print 'connection to', (orig_addr, orig_port)
print
If I have an iptables rule that looks like:
# iptables -t nat -A PREROUTING -p tcp --dport 1500:1600 \
-j REDIRECT --to-port 2000
And while the above Python code is running I connect from another
host to my_ip_address:1500, I see:
connection from ('192.168.1.20', 35790)
connection to ('192.168.1.75', (1500,))
And if I connect to port 1550 I see:
connection from ('192.168.1.20', 42054)
connection to ('192.168.1.75', (1550,))
Which I think is exactly what you were asking for. Note that to my knowledge this will only work for TCP connections; there are other solutions (possibly involving the TPROXY iptables target) that may work with UDP connections as well.

Unable to get to node.js externally

I've looked everywhere for an answer on this, but haven't had any luck.
I've installed node.js on my server. I've created the standard "Hello World" example like:
var http = require('http');
http.createServer(function (request, response) {
response.writeHead(200, {'Content-Type': 'text/plain'});
response.end('Hello World\n');
}).listen(8080, "0.0.0.0");
console.log('Server running at http://0.0.0.0:8080/');
After running the script on the server:
node app.js
I can connect internal to port 808 and see the Hello World message, but when I try to connect to port 8080 my server externally I get a "Can't connect to server" error. I've also tried this in my listen function:
etc..
}).listen(8080, "204.xxx.xxx.xxx");
(with my real external IP address) and haven't had any luck.
I've tried to accept connections on 8080 by adding this to iptables:
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
but still have hit a wall. When I run netstat I get:
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -
which I think tells me that port 8080 is listening for connections.
So — what am I doing wrong here?
You are most likely behind network address translation (NAT).
If you're using a normal home internet connection and you have a gateway router, you can have multiple devices using your home's internet connection (connected via Ethernet or Wifi), no?
But you only have one IP address.
To accomplish this, the router lets you connect out - but doesn't let any connections initiated from the outside back in (simplification for relevance - read up if you want more information).
You're going to have to look at configuring port forwarding - you want external port 8080 to forward to your computer's internal IP address.

Resources