First let me explain my scenario. We have an application using TCP that gets hanged frequently due to CLOSE_WAIT connections. From the netstat I can trace the remote host for which the CLOSE_WAIT happens. But I want to know the elapsed time (time of occurrence of CLOSE_WAIT on a particular port). If I know the exact time the CLOSE_WAIT happens, I can analyze the logs corresponding to the time stamp to find any possible reason for the same.
I know I can run netstat at regular intervals. By this way also, I can get the exact time window of CLOSE_WAIT connections.
Is there any simpler way to get what I need using netstat or any other commands ?
You could watch the traffic directly with tcpdump. If you know the remote IP and/or port you can narrow it down to just that traffic.
tcpdump -i eth0 src 192.168.1.1 and port 80
Related
netstat lists all the connections (incoming and outgoing)
how do I filter out just the incoming connections
I have tried the netstat command but it lists all the connections but j need only the incoming connections
Once sockets are created, there isn't really such thing as inbound and outbound, as connections go both ways. What you really care about are connections you started, vs connections others started. To do that, you have to monitor for new connections, and log them as they are created.
tcpdump is a great tool for this. There are tons of guides on the internet, but a command to get you started would be tcpdump -Qin ....
With netstat you may identify the state of the socket, but in many cases there are no states in raw mode and usually no states used in UDP and UDPLite. You may try to display the listening state for incoming connections by running netstat with the following argument:
netstat --listening
However, as far as I understood from your question it is better to use the tcpdump tool as mentioned in other comments.
So, I've got a CentOS6 host where some process has cached an old DNS server, and I'm trying to find out what process that is. (Let's say I can't go blindly rebooting servers and I need to physically confirm what program making requests to this old DNS server).
I've been looking at the outgoing DNS traffic from said host to the old DNS server, and it's originating from some short-lived ephemeral port that opens and closes immediately after, and I'm trying to figure out what process requested it.
example:
$ tcpdump -i eth0 port 53 | grep "> old.dns.host"
19:26:33.442632 IP my.host.46133 > old.dns.host.domain: 56541+ PTR? 49.6.123.10.in-addr.arpa. (42)
my.host is still reaching out to the old DNS server old.dns.host, over the ephemeral port 46133. But when checking netstat, that port is gone. Even watching netstat and using -c don't pick anything up.
So in general -- if I have a port that opens only for a few milliseconds for the duration of a request, how do I find out what process opened it?
What is the best way to tell if traffic is reaching a socket?
netstat -i is convenient for checking counters in/out of an interface, but how to tell if that traffic is reaching a socket?
Thanks
You can use netstat to check the count of ESTABLISHED connection to the port number that the socket is using to listen. Alternatively tcpdump can show you packets flowing in and out of that port.
arangod is running for some time without any problems, but at some point no more connections can be made.
aranogsh then shows the following error message:
Error message 'Could not connect to 'tcp://127.0.0.1:8529' 'connect() failed with #99 - Cannot assign requested address''
In the log file arangod still writes more trace information.
After restarting aranogd it is running without problems again, until the problem suddenly reoccurs.
Why is this happening?
Since this question was sort of answered by time, I'll use this answer to elaborate howto dig into such a situation and to get a valuable analysis on which operating system parameters to look. I'll base this on linux targets.
First we need to find out whats currently going on using the netstat tool as a root user (we care for tcp ports only):
netstat -alnpt
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
...
tcp 0 0 0.0.0.0:8529 0.0.0.0:* LISTEN 3478/arangod
tcp 0 0 127.0.0.1:45218 127.0.0.1:8529 ESTABLISHED 6902/arangosh
tcp 1 0 127.0.0.1:46985 127.0.0.1:8529 CLOSE_WAIT 485/arangosh
We see an overview of the 3 possible value groups:
LISTEN: These are daemon processes offering tcp services to remote ends, in this case the arangod process with its server socket. It binds port 8529 on all available ipv4 addresses of the system (0.0.0.0) and accepts connections from any remote location (0.0.0.0:*)
ESTABLISHED: this is an active tcp connection in this case between arangosh and arangod; Arangosh has its client port (45218) in the higher range connecting arangod on port 8529.
CLOSE_WAIT: this is a connection in termination state. Its normal to have them. The TCP stack of the operating system keeps them around for a while to have a knowledge where to sort in stray TCP-packages that may have been sent, but did not arive on time.
As you see TCP ports are 16 bits unsigned integers, ranging from 0 to 65535. Server sockets start from the lower end, and most operating systems require processes to be running as root to bind ports below 1024. Client sockets start from the upper end and range down to a specified limit on the client. Since multiple clients can connect one server, while the server port range seems narrow, its usually the client side ports that wear out. If the client frequently closes and reopens the connection you may see many sockets in CLOSE_WAIT state, as many discussions across the net hint, these are the symptoms of your system eventually running out of resources. In general the solution to this problem is to to re-use existing connections through the keepalive feature.
As the solaris ndd command explains thoroughly which parameters it may modify with which consequences in the solaris kernel, the terms explained there are rather generic to tcp sockets, and may be found on many other operating systems in other ways; in linux - which we focus on here - through the /proc/sys/net-filesystem.
Some valuable switches there are:
ipv4/ip_local_port_range This is the range for the local sockets. You can try to narrow it, and use arangob --keep-alive false to explore whats happening if your system runs out of these.
time wait (often shorted to tw) is the section that controls what the TCP-Stack should do with already closed sockets in CLOSE_WAIT state. The Linux kernel can do a trick here - it can instantly re-use connections in that state for new connections. Vincent Bernat explains very nicely which screws to turn and what the differnt parameters in the kernel mean.
So once you decided to change some of your values in /proc so your host better scales to the given situation, you need to make them reboot persistant - since /proc is volatile and won't remember values across reboots.
Most linux systems therefore offer the /etc/sysctl.[d|conf] file; It maps slashes in the proc filesystem to dots, so /proc/sys/net/ipv4/tcp_tw_reuse will translate into net.ipv4.tcp_tw_reuse.
My machine is continuously making udp dns traffic request. What I need to know is the PID of the process generating this traffic.
The normal way in TCP connection is to use netstat/lsof and get the process associated at the pid.
Is UDP the connection is stateless, so, when I call netastat/lsof I can see it only if the UDP socket is opened and it's sending traffic.
I have tried with lsof -i UDP and with netstat -anpue but I can't be able to find which process is doing that request because I need to call lsof/netstat exactly when the udp traffic is sent, if I call lsof/netstat before/after the udp datagram is sent is impossible to view the opened UDP socket.
call netstat/lsof exactly when 3/4 udp packet is sent is IMPOSSIBLE.
how I can identify the infamous process?
I have already inspected the traffic to try to identify the sent PID from the content of the packet, but is not possible to identify it from the content of the traffic.
anyone can help me?
I'm root on this machine FEDORA 12 Linux noise.company.lan 2.6.32.16-141.fc12.x86_64 #1 SMP Wed Jul 7 04:49:59 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
EDIT:
I have asked the same question on superuser platform.
More related place to this kind of question OF COURSE!
Click here, You can find the right answer in the right place
netstat -anp |grep -i udp
The process is at the last column
Are these UDP DNS requests going to the name servers your Fedora box is configured to use? Just opening Firefox and going to http://www.google.com will generate a stream of UDP packets as name resolution happens. Run tcpdump port 53 in a terminal window, open Firefox and go to some website, you'll see what I mean.
Linux Socket Monitor can track changes to network sockets. It should alert you when new socket is opened.