So, I've got a CentOS6 host where some process has cached an old DNS server, and I'm trying to find out what process that is. (Let's say I can't go blindly rebooting servers and I need to physically confirm what program making requests to this old DNS server).
I've been looking at the outgoing DNS traffic from said host to the old DNS server, and it's originating from some short-lived ephemeral port that opens and closes immediately after, and I'm trying to figure out what process requested it.
example:
$ tcpdump -i eth0 port 53 | grep "> old.dns.host"
19:26:33.442632 IP my.host.46133 > old.dns.host.domain: 56541+ PTR? 49.6.123.10.in-addr.arpa. (42)
my.host is still reaching out to the old DNS server old.dns.host, over the ephemeral port 46133. But when checking netstat, that port is gone. Even watching netstat and using -c don't pick anything up.
So in general -- if I have a port that opens only for a few milliseconds for the duration of a request, how do I find out what process opened it?
Related
I have a node.js server running on an EC2 instance. I can connect to my index.html page by visiting 12.345.678.900:2112 (not the actual site, just an example). Later on, I open a socket connection that is listening on port 3113. When I'm on the machine itself, I can access the webpage through the localhost (localhost:3113/foo). When I try to access it outside the local machine through 12.345.678.900:3113/foo, I can't access it. The socket connection is successful and the server is listening on 3113 when I run netstat, but it just doesn't allow me to access the page. I have a feeling my ports may be blocked, but from what I tell it shouldn't be. I don't have this problem when I run this server on my local machine however.
Incoming Connections
Solution:
Since curl was telling me port 3113 wasn't open, but it wasn't being blocked on AWS I looked through my code and realized that I opened port 3113 but then closed it soon after, which meant that while I was getting a log statement telling me the port was open since it was closed almost immediately I wasn't able to access the port.
When you setup TCP listening port, you had to specify IP address and port number. If your server is listening on 127.0.0.1:3113, then it won't be accessible from its external IP address. You can also specify the special 0.0.0.0 as the IP address, which indicates that you want to listen on that port in "all" IP addresses (for some definition of "all").
If your machine have multiple network interfaces (e.g. multiple network cards, or virtualised network interface), then it's possible to have the same ip address actually being on different networks and you'll have to specify the specific network interface when you listen to them. But if you did set up something unusual like that, you probably already know about these networking stuffs already. Note that if you have Docker or virtual machines, they may also creates virtual network interface for each containers/VM.
On AWS/EC2, your machine may also have internal IP address that's only accessible from your VPC and an Elastic IP which can be accessed directly from the outside world. You didn't say which "external" IP address you were using.
Finally, you may have firewall installed on the server. ipfw or iptable may have rules blocking external listening port.
I made a nginx server at 192.168.1.106 and I can access it from the host computer but it doesn't show when I access it on a computer in the same network. But I can ssh into the server from a different computer using the same address. I checked my firewall to make sure it isn't blocking anything.
How do I access the server form a different computer?
> iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
The server_namedocs directive is used to identify virtual hosts, they're not used to set the binding.
netstat tells you that nginx listens on 0.0.0.0:80 which means that it will accept connections from any IP.
If you want to change the IP nginx binds on, you have to change the listendocs rule.
So, if you want to set nginx to bind to localhost, you'd change that to:
listen 127.0.0.1:80;
In this way, requests that are not coming from localhost are discarded (they don't even hit nginx).
I will like to host OwnCloud on my own machine, and access it via localhost (I don't know yet if this is possible, but if it's not, then the question remains valid, this is just to give you a better idea of the situation), however, I'm not completely sure that I'll be the only one being able to access it.
From looking at other similar questions, for example here ("How can I access my localhost server from other computers?"), I found that there is some configuration you have to do first if you want others to be able to access your localhost, and they even have to be in the same network.
However, in this other question ("How can I access my localhost from my Android device?"), I find that you can do it easily without any configuration.
Then the problem is, how to make sure no one, except my machine, can access the localhost?
If you have Apache, put Listen 127.0.0.1:80 on the configuration file and restart Apache.
Nginx is almost the same. Put listen 127.0.0.1:80; on the config file and restart Nginx.
After restarting your webserver, issue a netstat -an A | find "80" on Windows, or netstat -patune | grep 80 on Linux and see if the service is running only on 127.0.0.1. On my Linux I see something like this (notice 127.0.0.1:80):
tcp 0 0 127.0.0.1:80 0.0.0.0:* LISTEN 0 9711 -
And only programs running on the same computer can access it now. You phone cannot, your router cannot, nobody outside your computer can even know that the port 80 is listening. Even knowing the IP, the port, and belonging to the same network is not possible to connect to it.
I have 2 centOS-7 machines say machine A,B.
Machine A have a server listening on port 80. When i run curl-loader(for load testing purpose) with 2000 request per second in the same machine, all the request is hitting the server, i checked 'ss -s' command and open TCP sockets is more than 2000.
But my problem is when i run curl-loader in machine B and try to hit server A only few requests are reaching server, remaining all are dropping out.
in machine B 'ss -s' command returns 2K+ value. but in machine A 'ss -s' command will return only value 25-30. Remaining all request are returned are TCP-CONNECT error
#arkascha is probably right. Some throttling is happening. I would only add that the throttling could be in the network, in the server OR in the client. Any of them could be throttling the number of connections.
It's better to test it without any squid or high level software. You can just do:
(On server) $ echo hello | nc -l -k 10000
(On client) $ for ((i=0; i<2000; i++)); do nc :10000 & done
This will create a "mini" TCP server on the server machine on port 10000. Anyone that connects to that server will receive the message "hello".
Then from the client, you can create 2000 (almost simultaneous) TCP connections to the server on port 10000.
If the connections are successfully established, then the throttling is probably at the "proxy" level. If the connections appear throttled, then the throttling is likely at the firewall or network level.
First let me explain my scenario. We have an application using TCP that gets hanged frequently due to CLOSE_WAIT connections. From the netstat I can trace the remote host for which the CLOSE_WAIT happens. But I want to know the elapsed time (time of occurrence of CLOSE_WAIT on a particular port). If I know the exact time the CLOSE_WAIT happens, I can analyze the logs corresponding to the time stamp to find any possible reason for the same.
I know I can run netstat at regular intervals. By this way also, I can get the exact time window of CLOSE_WAIT connections.
Is there any simpler way to get what I need using netstat or any other commands ?
You could watch the traffic directly with tcpdump. If you know the remote IP and/or port you can narrow it down to just that traffic.
tcpdump -i eth0 src 192.168.1.1 and port 80