I am basically trying to write a pstree-like command except that it should follow processes across machines.
What I mean is that if I run this :
$ ssh $node sleep 1000
Then the command should display something like this :
ssh $node -- ($node) sleep 1000
And if I'm running :
$ ssh $node ssh $node sleep 1000
ssh $node---($node) ssh $node---($node) sleep 1000
And so on ...
My question is this : How can I map one ssh session on one machine to a spawned process on another machine ?
Local parent-child processes are not a problem, but how can I figure out which ssh command on one node that triggered another process on another node.
linux 2.6.18
only openSSH for "remote" stuff. Running OpenSSH_4.3p2 currently.
SSH access to all nodes of course (key based auth) so ps and netstat are available from all nodes.
Linux-only "hacks" are fine, does not need to be portable though that would be an added bonus of course.
The user will always be the same and my command/script is running as that user. That user is not root.
Does not have to be fast, only accurate.
The spontaneous solution would be to write a pstree clone, that triggers on the command string "ssh", figures out the source-port and then goes to the remote machine in question and figures out which one of sshd's children that was spawned by this particular command.
But maybe there's a more clever way ? :P
Actually, I think your spontaneous solution is the right way to do it: use netstat to get the source-port and look for it on the remote machine. You might have trouble using "netstat -p" without being root - I tried it on two machines, one which was happy to show me my own processes and one which wasn't.
As well as ssh clients, you might extend this to look for other clients that use ssh connections, like rsync or Mercurial. Just be careful not to trace your program's own connection recursively!
A quick experiment with netstat and pstree shows that the idea is sound:
me#mymachine:~$ netstat -p
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 mymachine.example:43681 remote.example.com:ssh ESTABLISHED 27044/ssh
tcp 0 0 mymachine.example:39228 remote.example.com:ssh ESTABLISHED 14499/ssh
tcp 0 0 mymachine.example:45814 remote.example.com:ssh ESTABLISHED 20899/ssh
me#mymachine:~$ ssh remote netstat -p | grep mymachine.example:43681
tcp 0 0 remote.example.com:ssh mymachine.example:43681 ESTABLISHED 10361/1
me#mymachine:~$ ssh remote pstree -a 10361
sshd
`-grep -n -e wotsit -i -R /local/home/me/somewhere /dev/null
I'd be interested to see the result, because it would be very useful to me!
Related
I want to be able to SSH to a number linux devices at once, behind different NATs. I can't configure the network that they are on. However, I'm having trouble getting ssh to go over an existing connection.
I have full control over both my client and the devices. Here's the process so far:
On my client, I first run
socat TCP-LISTEN:5001,pktinfo,fork EXEC:./create_socket.sh,fdin=3,fdout=4,nofork
Contents of ./create_socket.sh:
ssh -N -M -S "~/sockets/${SOCAT_PEERADDR}" -o "ProxyCommand=socat - FD:3!!FD:4" "root#${SOCAT_PEERADDR}"
On the device, I'm running
socat TCP:my_host:4321 TCP:localhost:22
However, nothing comes in or out of FD:3!!FD:4, I assume because the ProxyCommand is a subprocess. I've also tried setting fdin=3,fdout=3 and changing ./create_socket.sh to:
ssh -N -M -S "~/sockets/${SOCAT_PEERADDR}" -o "ProxyUseFdpass=yes" -o "ProxyCommand=echo 3" "root#${host}"
This prints an error:
mm_receive_fd: no message header
proxy dialer did not pass back a connection
I believe this is because the fd should be sent in some way using sendmsg, but the fd doesn't originate from the subprocess anyways. I'd like to make it as simple as possible, and this feels close to workable.
You want to turn the client/server model on its head and make a generic server to spawn a client on-demand and in-response-to an incoming unauthenticated TCP connection from across a network boundary, and then tell that newly-spawned client to use that unauthenticated TCP session. I think that may have security considerations that you haven't thought of. If a malicious person spams connections to your computer, your computer will spawn a lot of SSH instances to connect back and these processes can take up a lot of local system resources while authenticating. You're effectively trying to set up SSH to automatically connect to an untrusted (unverified) remote-initiated machine across a network boundary. I can't stress how dangerous that could be for your client computer. Using the wrong options could expose any credentials you have or even give a malicious person full access to your machine.
It's also worth noting that the scenario you're asking to do, building a tunnel between multiple devices to multiplex additional connections across an untrusted network boundary, is exactly the purpose of VPN software. Yes, SSH can build tunnels. VPN software can build tunnels better. The concept would be that you'd run a VPN server on your client machine. The VPN server will create a new (virtual) network interface which represents only your devices. The devices would connect to the VPN server and be assigned an IP address. Then, from the client machine, you'd just initiate SSH to the device's VPN address and it will be routed over the virtual network interface and arrive at the device and be handled by its SSH daemon server. Then you don't need to muck around with socat or SSH options for port forwarding. And you'd get all the tooling and tutorials that exist around VPNs. I strongly encourage you to look at VPN software.
If you really want to use SSH, then I strongly encourage you to learn about securing SSH servers. You've stated that the devices are across network boundaries (NAT) and that your client system is unprotected. I'm not going to stop you from shooting yourself in the foot but it would be very easy to spectacularly do so in the situation you've stated. If you're in a work setting, you should talk to your system administrators to discuss firewall rules, bastion hosts, stuff like that.
Yes, you can do what you've stated. I strongly advise caution though. I advise it strongly enough that I won't suggest anything which would work with that as stated. I will suggest a variant with the same concepts but more authentication.
First, you've effectively set up your own SSH bounce server but without any of the common tooling compatible with SSH servers. So that's the first thing I'd fix: use SSH server software to authenticate incoming tunnel requests by using ssh client software to initiate the connection from the device instead of socat. ssh already has plenty of capabilities to create tunnels in both directions and you get authentication bundled with it (with socat, there's no authentication). The devices should be able to authenticate using encryption keys (ssh calls these identities). You'll need to connect once manually from the device to verify and authorize the remote encryption key fingerprint. You'll also need to copy the public key file (NOT the private key file) to your client machine and add it to your authorized_keys files. You can ask for help on that separately if you need it.
A second issue is that you appear to be using fd3 and fd4. I don't know why you're doing that. If anything, you should be using fd0 and fd1 since these are stdin and stdout, respectively. But you don't even need to do that if you're using socat to initiate a connection. Just use - where stdin and stdout are meant. It should be completely compatible with -o ProxyCommand without specifying any file descriptors. There's an example at the end of this answer.
The invocation from the device side might look like this (put it into a script file):
IDENTITY=/home/WavesAtParticles/.ssh/tunnel.id_rsa # on device
REMOTE_SOCKET=/home/WavesAtParticles/.ssh/$(hostname).sock # on client
REMOTEUSER=WavesAtParticles # on client
REMOTEHOST=remotehost # client hostname or IP address accessible from device
while true
do
echo "$(date -Is) connecting"
#
# Set up your SSH tunnel. Check stderr for known issues.
ssh \
-i "${IDENTITY}" \
-R "${REMOTE_SOCKET}:127.0.0.1:22" \
-o ExitOnForwardFailure=yes \
-o PasswordAuthentication=no \
-o IdentitiesOnly=yes \
-l "${REMOTEUSER}" \
"${REMOTEHOST}" \
"sleep inf" \
2> >(
read -r line
if echo "${line}" | grep -q "Error: remote port forwarding failed"
then
ssh \
-i "${IDENTITY}" \
-o PasswordAuthentication=no \
-o IdentitiesOnly=yes \
-l "${REMOTEUSER}" \
"${REMOTEHOST}" \
"rm ${REMOTE_SOCKET}" \
2>/dev/null # convince me this is wrong
echo "$(date -Is) removed stale socket"
fi
#
# Re-print stderr to the terminal
>&2 echo "${line}" # the stderr line we checked
>&2 cat - # and any unused stderr messages
)
echo "disconnected"
sleep 30
done
Remember, copying and pasting is bad in terms of shell scripts. At a minimum, I recommend you read man ssh and man ssh_config, and to check the script against shellcheck.net. The intent of the script is:
In a loop, have your device (re)connect to your client to maintain your tunnel.
If the connection drops or fails, then reconnect every 30 seconds.
Run ssh with the following parameters:
-i "${IDENTITY}": specify a private key to use for authentication.
-R "${REMOTE_SOCKET}:127.0.0.1:22": specify a connection request forwarder which accept connections on the Remote side /home/WavesAtParticles/$(hostname).sock then forward them to the local side by connecting to 127.0.0.1:22.
-o ExitOnForwardFailure=yes: if the remote side fails to set up the connection forwarder, then the local side should emit an error and die (and we check for this error in a subshell).
-o PasswordAuthentication=no: do not fall back to a password request, particularly since the local user isn't here to type it in
-o IdentitiesOnly=yes: do not use any default identity nor any identity offered by any local agent. Use only the one specified by -i.
-l "${REMOTEUSER}": log in as the specified user.
remotehost, eg your client machine that you want a device to connect to.
Sleep forever
If the connection failed because of a stale socket, then work around the issue by:
Log in separately
Delete the (stale) socket
Print today's date indicating when it was deleted
Loop again
There's an option which is intended to make this error-handling redundant: StreamLocalBindUnlink. However the option does not correctly work and has a bug open for years. I imagine that's because there really aren't many people who use ssh to forward over unix domain sockets. It's annoying but not difficult to workaround.
Using a unix domain socket should limit connectivity to whoever can reach the socket file (which should be only you and root if it's placed in your ${HOME}/.ssh directory and the directory has correct permissions). I don't know if that's important for your case or not.
On the other hand you can also simplify this a lot if you're willing to open a TCP port on 127.0.0.1 for each device. But then any other user on the same system can also connect. You should specifically listen on 127.0.0.1 which would then only accept connections from the same host to prevent external machines from reaching the forwarding port. You'd change the ${REMOTE_SOCKET} variable to, for example, 127.0.0.1:4567 to listen on port 4567 and only accept local connections. So you'd lose the named socket capability and permit any other user on the client machine to connect to your device, but gain a much simpler tunnel script (because you can remove the whole bit about parsing stderr to remove a stale socket file).
As long as your device is online (can reach your workstation's incoming port) and is running that script, and the authentication is valid, then the tunnel should also be online or coming-online. It will take some time to recover after a loss (and restore) of network connectivity, though. You can tune that with ConnectTimeout, TCPKeepAlive, and ServerAliveInterval options and the sleep 30 part of the loop. You could run it in a tmux session to keep it going even when you don't have a login session running. You could also run it as a system service on the device to bring it online even after recovering from a power failure.
Then from your client, you can connect in reverse:
ssh -o ProxyCommand='socat - unix-connect:/home/WavesAtParticles/remotehost.sock' -l WavesAtParticles .
In this invocation, you'll start ssh. It will then set up the proxycommand using socat. It will take its stdin/stdout and relay it through a connected AF_UNIX socket at the path provided. You'll need to update the path for the remote host you expect. But there's no need to specify file descriptors at all.
If ssh complains:
2019/08/26 18:09:52 socat[29914] E connect(5, AF=1 "/home/WavesAtParticles/remotehost.sock", 7): Connection refused
ssh_exchange_identification: Connection closed by remote host
then the tunnel is currently down and you should investigate the remotehost device's connectivity.
If you use the remote forwarding option with a TCP port listening instead of a unix domain socket, then the client-through-tunnel-to-remote invocation becomes even easier: ssh -p 4567 WavesAtParticles#localhost.
Again, you're trying to invert the client/server model and I don't think that's a very good idea to do with SSH.
I’m going to try this today:
http://localhost.run/
It seems like what you are looking for.
Not to answer your question but helpful for people who may not know:
Ngrok is the easiest way I’ve found. they do webservers as well as tcp connections. I’d recommend installing it through homebrew.
https://ngrok.com/product
$ ngrok http 5000
In the terminal for http, 5000 being the port of your application.
$ ngrok tcp 5000
In the terminal for tcp.
It’s free for testing(random changing domains).
For tcp connections remove “http://“ from the web address to get the IP address. Sorry I can’t remember. I think the client ports to 80 and I believe you can change that by adding port 5001 or something, google it to double check
I encounted a problem today:
When I started HDP docker container, an error occured:
listen tcp 0.0.0.0:8086: bind: address already in use
According to error message, I know that port 8086 was already in use, so I tried some commands to determine which program was using port 8086.
lsof -i:8086
lsof -i tcp:8086
lsof | grep 8086
But all of commands above make no outputs!
I felt really confused about that, after some searching on google, I tried another command:
netstat -pna | grep 8086
I got correct output from this command.
I know some differences between lsof and netstat, but I really do not know why I cannot get any output from lsof -i:8086?.
Here are some differences between two commands I searched from google:
netstat(net statistic) is connection based,it shows NW connections (udp/tcp ports), routing tables, interface, multi-cast membership, etc.
lsof(list of open files) is application based, this is kind of like netstat + ps, there you can see all accessed ports, NW connections, etc.
but lsof includes stuff like my local emacs window terminal session (tty dev/pts/n) which is not part of netstat
I faced a similar issue today. The solution was to run the lsof command with sudo privileges.
sudo lsof -i:8086
should print the desired output.
LSOF: List of Open Files. It lists all the open files belonging to all active processes.
Examples:
sudo lsof -n -i
sudo lsof -n -i4
sudo lsof -n -i :80
-n inhibits the conversion of network numbers to host names for network files. Inhibiting conversion may make lsof run faster. It is also useful when host
lookup is not working properly
-i selects the listing of files any of whose Internet address matches the address specified in i. If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files. If -i4 or -i6 is specified with no following address, only files of the indicated IP version, IPv4 or IPv6, are displayed.
NETSTAT: It is a tool to get the network statistics. By default, netstat displays a list of open sockets. If you don't specify any
address families, then the active sockets of all configured address
families will be printed.
Displays the kernel routing tables:
netstat -r
Display all listening and established connection for both TCP and UDP with PID data:
netstat -plunt
Additionally, You have another command line tool to use which is SS.
SS: It is used to dump socket statistics. It allows showing information similar to netstat. It can display more TCP and state
information than other tools.
-plunt gives data for the TCP and UDP connections which are established and listening with process information:
sudo ss -plunt
You should be root to get proper answers to your lsof questions. Your command is fine, assuming something really is listening on that port.
As you already mentioned, lsof is a very useful command which is used to list files opened by a specific process, while netstat is a tool for monitoring network connections.
You should be able to find the PID of the process listening on port 8086 with netstat:
netstat -tunlp |grep :8086
and then use lsof to list the files used by the process:
lsof -p PID
How can you find out which process is listening upon which port on Windows and Linux?
Are there some Applications explicitly monitoring?
Some great tools for this are made by Sysinternals, now owned by Microsoft.
The one you want is Tcpview and it will show you the ports and which application has them opened, as well as the PID and other nice things. Tcpview is windows based but they have a command line version as well. All these tools are free.
This is the link Microsoft's sysinternals downloads
Both Windows and Linux has the netstat-command built-in, although they are used differently.
On Windows: netstat -a -b (lists both listening and connected ports)
On Linux: netstat -l -p (lists only listening ports)
On windows 7, you can use
netstat -b -a
netstat /?
-b Displays the executable involved in creating each connection or
listening port. In some cases well-known executables host
multiple independent components, and in these cases the
sequence of components involved in creating the connection
or listening port is displayed. In this case the executable
name is in [] at the bottom, on top is the component it called,
and so forth until TCP/IP was reached. Note that this option
can be time-consuming and will fail unless you have sufficient
permissions.
-o Displays the owning process ID associated with each connection.
On Linux use, -p needs root privileges.
#netstat -p
#netstat -h
-p, --programs display PID/Program name for sockets
Not sure that stackoverflow is the right place for this question, maybe http://www.superuser.com would be a better choice.
Although from the top of my head:
Linux has lsof and netstat commands that will provide this information.
Windows has ProcessExplorer that should give this information.
In Linux you can use the ss command to dump the socket information. It gives information about active port numbers in the client side also. More details can be found here
http://linux.die.net/man/8/ss
I'm trying to use fuser to find the pids of processes I own which have certain TCP ports open.
In the fuser man page it says:
... The most common time this problem occurs is when looking for TCP or UDP sockets when running fuser as a non-root user. In this case fuser will report no access. ...
However, on my Ubuntu box, fuser does report sockets open for processes that I own, e.g.:
perl -MIO::Socket 'IO::Socket::INET->new(Listen => 10, LocalPort => 3000)' &
fuser -n tcp 3000
Question: how are things set up to allow this to happen? Is it a kernel config option?
Thanks!
Note: the question is: how are some linux distros configured so that fuser will report processes owning sockets when fuser is run as a normal user? One one Ubuntu distro "fuser -n tcp 3000" will report a process if I own the process, yet on another linux distro (I think Centos) it won't report the process even if I own it.
fuser goes through the /proc file system (proc(5)) working through the /proc/[pid]/fd/ directory and checking the file descriptors. Processes owned by you have corresponding /proc entries again owned by you. This allows you to check your processes, but not others.
One very useful tool to see what given program is doing is strace(1). For example, you can see what system calls, and with what arguments, are done by the fuser:
~$ strace fuser -n tcp 3000
I am running CentOS 5 with csf firewall. I'm running a program that can't connect to another server (using some port that is blocked by csf I presume). Where is the log file for 'ports'?
Netstat is the command to use to get ports and network activity. To diagonise server processes I usually use:
netstat -tln
This yields port numbers in tcp mode listening. To identify associated processes you can also use -p to grab the pid. Here is the IANA ports list.
I found my answer right after searching a few more threads.
# tail -f /var/log/messages
Shows the UDP message but not the port.... Hmm....