I have several questions regarding a Ubuntu machine which I am working on.
First of all here is the output of max open files:
$ cat /proc/sys/fs/file-max
1024000
However when checking how many are currently open:
$ lsof | wc -l
2002129
How is this number so much higer than the max?
Second, I had a look at what these open files were. 99% of them are these enteries:
ruby2.7 2749468 2750970 worker-14 fluentd 62u IPv4 1356781066 0t0 TCP {URL REDACTED}:5044->ip-10-153-7-71.eu-central-1.compute.internal:13637 (CLOSE_WAIT)
There is a mix of CLOSE_WAIT and ESTABLISHED connections. But when I check with netstat or ss it does not show that many connections:
ss -a | wc -l
1447
What else can I do to investigate why fluentd is rejecting connections and the machine is killing connections which cause the log delivery to fail?
As a final example, this one is a little crude. Listing fd's in /proc doesnt show as many as how many lsof shows:
ls -l /proc/*/fd | wc -l
1109
In my custom linux image using netstat 1.42 (2001-04-15) from the package net-tools 1.60 I am seeing different behaviors for different linux versions..
On Linux 5.4.28:
# netstat -tnp | grep ESTABLISHED
tcp 0 0 <local_ip>:22 <client_ip>:14321 ESTABLISHED 29342/1
tcp 0 40 <local_ip>:22 <client_ip>:22470 ESTABLISHED 28443/0
On Linux 5.10.65:
# netstat -tnp | grep ESTABLISHED
tcp6 0 52 <local_ip>:22 <client_ip>:22470 ESTABLISHED 1/init
tcp6 0 52 <local_ip>:22 <client_ip>:33679 ESTABLISHED 1/init
The thing to note here is that on Linux 5.10, netstat shows PID as 1 and process as init for all ssh sessions. But in older linux, I am seeing correct PIDs for all ssh sessions. I need to know the client_ip and the PID for every ssh session so I can kill the session for whichever client I want.
From online searches, I could not figure out any fixes.
One alternative that I can see till now is using who to get this output:
# who
root pts/1 00:00 Jan 12 07:16:19 <client_ip>
root pts/2 00:03 Jan 12 08:21:07 <client_ip>
grep this with client_ip then use ps and grep that with the second column, like ps | grep pts/1 for first client.
Is there anything I can do to netstat to fix this though? Because netstat also give the client port which is useful for me.
I have a Linux machine at the company I work, and we know some web services are running on port 8111. I need to find out what is running check why the code is failing.
I do a netstat -at and I can see someone listening on the port:
tcp6 0 0 [::]:8111 [::]:* LISTEN
then I do a sudo lsof -i:8111 and I get the proc that is doing the listening:
dockerd 64285 root 266u IPv6 197189396 0t0 TCP *:8111 (LISTEN)
It's the docker daemon, so I do a docker ps -a but I don't see in the list a container that is listening on that port.
The person who was responsible for this abandoned the company and left no documentation, and just quit from one day to the next, and we are unable to get in touch with him.
Update:
To get all Ids, Image names with Ports from running instances try:
docker ps | tail -n +1 | awk '{print $1}' | xargs -n1 docker inspect | grep -E '"Id"|"Image"|"Ports"'
How to create a TCP connection using nc which listens to multiple hosts?
nc -l -p 12345
Simultaneous connections are not possible with netcat. You should use something like ucspi-tcp's tcpserver tool or leverage xinetd since you're on Linux.
See: https://superuser.com/questions/232747/netcat-as-a-multithread-server
Consecutive connections could be handled through a shell script that restarts netcat after it finishes.
ncat can do it.
E.g. ncat --broker --listen -p 12345 will distribute all incoming messages to all other clients (think of it as a hub).
I recommend socat as nc alternative.
For OP's problem, socat - TCP-LISTEN:12345,fork,reuseaddr can do the job.
-k
Forces nc to stay listening for another connection after its current connection is completed. It is an error to use this option without the -l option.
using nc it is not possible to open parallel connections to same port, however you can trick nc to open multiple connections to same port.
To understand this, lets say you start listening on 4444 port using $ nc -l -p 4444 -v. Now, if you check output of $ netstat -anp | grep 4444 you will get its state as LISTEN and in here its pid is 3410.
tcp 0 0 0.0.0.0:4444 0.0.0.0:* LISTEN 3410/nc
Now, after it gets connected to client, lets say you run $ nc localhost 4444 -v, its state will get changed into ESTABLISHED. Now, try running $ netstat -anp | grep 4444 you will get its state as ESTABLISHED, see for same pid 3410, and a client process with pid 3435
tcp 0 0 127.0.0.1:46678 127.0.0.1:4444 ESTABLISHED 3435/nc
tcp 0 0 127.0.0.1:4444 127.0.0.1:46678 ESTABLISHED 3410/nc
Please note that there is no available listening port, so you can't have another client process. However if you run again $ nc -l -p 4444 -v you can have a listening port and can have multiple client process.
see netstat -anp | grep 4444 output after you start listening to same port.
tcp 0 0 0.0.0.0:4444 0.0.0.0:* LISTEN 3476/nc
tcp 0 0 127.0.0.1:46678 127.0.0.1:4444 ESTABLISHED 3435/nc
tcp 0 0 127.0.0.1:4444 127.0.0.1:46678 ESTABLISHED 3410/nc
see netstat -anp | grep 4444 output after you attach new client to same port.
tcp 0 0 127.0.0.1:4444 127.0.0.1:46694 ESTABLISHED 3476/nc
tcp 0 0 127.0.0.1:46678 127.0.0.1:4444 ESTABLISHED 3435/nc
tcp 0 0 127.0.0.1:4444 127.0.0.1:46678 ESTABLISHED 3410/nc
tcp 0 0 127.0.0.1:46694 127.0.0.1:4444 ESTABLISHED 3483/nc
You can say connections behavior is like:
SERVER_PROCESS_1 <---> CLIENT_PROCESS_1
SERVER_PROCESS_2 <---> CLIENT_PROCESS_2
so, you can write some script to simulate this behavior, or use this bash script to modify.
#!/usr/bin/bash
lport="4444"
i=0;
while [ true ]; do
echo "opening socket $(( i++ ))";
if [[ "$(ss sport = :$lport -l -H | wc -l)" -eq 0 ]]; then
nc -l -vv -p $lport &
#do something else to process or attach different command to each diff server process
fi;
if [[ "$(ss sport = :$lport -l -H | wc -l)" -ne 0 ]]; then
watch -n 0.1 -g "ss sport = :$lport -l -H" > /dev/null;
fi;
if [[ i -eq 10 ]]; then
break;
fi;
done;
in here every time client consume a connection this script will start new listen socket.
This behavior is however can be changed in ncat (here, using -k)as you can analyze the with below example:
server is started using $ ncat -l -p 4444 -v -4 -k and 3 clients are started using $ ncat -4 localhost 4444. Now output for $ netstat -anp | grep 4444 is:
tcp 0 0 0.0.0.0:4444 0.0.0.0:* LISTEN 3596/ncat
tcp 0 0 127.0.0.1:4444 127.0.0.1:46726 ESTABLISHED 3596/ncat
tcp 0 0 127.0.0.1:46726 127.0.0.1:4444 ESTABLISHED 3602/ncat
tcp 0 0 127.0.0.1:46722 127.0.0.1:4444 ESTABLISHED 3597/ncat
tcp 0 0 127.0.0.1:4444 127.0.0.1:46724 ESTABLISHED 3596/ncat
tcp 0 0 127.0.0.1:4444 127.0.0.1:46722 ESTABLISHED 3596/ncat
tcp 0 0 127.0.0.1:46724 127.0.0.1:4444 ESTABLISHED 3601/ncat
Every time new client connect, server fork its process to attach to client, so each server process is using same pid. So output of server in this way is shared to every attached clients, however each client can send individual message to server.
You can say connections behavior is like:
SERVER_PROCESS_1 <---> CLIENT_PROCESS_1
SERVER_PROCESS_1 <---> CLIENT_PROCESS_2
SERVER_PROCESS_1 <---> CLIENT_PROCESS_3
without -k, ncat will behave same as nc.
Benefits or loses can be defined on how they are to be needed.
For this example, i used nc or nc.traditional (v1.10-41.1+b1), and ncat (7.80).
This is an incomplete answer, because I haven't got it working. Arguably more of a question, in fact. Maybe someone else can finish it off.
First of all, it seems there are different versions of netcat. I'm on Ubuntu, so I've probably got the version that came with Ubuntu. When I nc -h, it says this:
OpenBSD netcat (Debian patchlevel 1.187-1ubuntu0.1)
When I run man nc, it says this:
-F Pass the first connected socket using sendmsg(2) to stdout and exit. This
is useful in conjunction with -X to have nc perform connection setup with
a proxy but then leave the rest of the connection to another program (e.g.
ssh(1) using the ssh_config(5) ProxyUseFdpass option).
It seems to me that this means that, instead of doing the usual thing with stdin and stdout, it just prints something to stdout. That something could then be used by another process to do the actual connection to the client.
Unfortunately, -F has no effect that I can see. So maybe I'm doing it wrong. Or maybe there's some secret pipe somewhere that I have to listen to, or a supplementary argument they forgot to document. Or maybe I happen to have a broken build of netcat, and it works for everyone else who's on Ubuntu.
In combination with the -k option (or, failing that, a while-true loop), this would allow many different clients to have separate connections. Suppose you have an executable called handle_connection, which takes as arguments an in file descriptor from a client and an out file descriptor to the client, and spawns a subprocess which communicates with the client. Then the server script might look like this:
nc -lkF $host $port | while read in out ; do
handle_connection $in $out ;
done
ncat can do it, but the correct command with ncat is:
ncat --keep-open --listen -p 12345
This will accept multiple connections at the same time.
You can then send the data with multiple clients. e.g. open in two or more terminals, and try typing there:
nc localhost 12345
See the output below:
tcp 0 0 192.168.2.222:35774 192.168.3.200:9090 TIME_WAIT
On Linux I want to find out which program connects to 192.168.3.200:9090, how can I achieve this?
I think
# lsof -Pnl +M -i4 // Listens on all IPV4 Ports
or
# netstat -ntp
are the commands you are searching for.
Then try to use fgrep to find the line. (in your case: fgrep '192.168.3.200:9090')
--> netstat -ntp | fgrep '192.168.3.200:9090'
you can do netstat -anp.
You'll need to sudo or be root to see the process information.
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 665/sshd
netstat -avnp | egrep '^tcp' | fgrep '192.168.3.200:9090'