When running autossh without the "-f" option, everything works fine.
Adding the "-f" option indeed sends autossh to the background, but after the ssh tunneling established correctly the autossh itself exit, leaving the ssh connection without monitor.
Here is the command I'm running: autossh -f -M 20000 -N -L 0.0.0.0:5601:10.10.0.8:5601 10.10.0.8
Anyone knows what can cause this problem? alternatively - anyone knows how can I debug autossh when using the "-f"? (is there any log file produce when using AUTOSSH_DEBUG=1)?
(I'm running on Ubuntu 14.04)
Thanks,
Shay
Seeing as no one has a better suggestion... Try running autossh under a watchdog like daemontools. With this method autossh runs as a foreground child of a supervise daemon (so, no -f switch). You can start and stop it with the svc command, and you can log all of its output with multilog.
This method has proven sufficiently reliable for me, on a fleet of production boxes.
On macOS I had a problem where autossh -M 0 -R 1234:localhost:22 worked but adding -f to make autossh run in background would log the following and autossh would die instantly:
2018/04/10 12:00:06 autossh[67839]: ssh exited with status 0; autossh exiting
Adding -N ("Do not execute a remote command.") fixed the issue:
autossh -f -M 0 -N -R 1234:localhost:22
Seeing you already had -N in the command this is probably unrelated but possibly helpful to others.
Related
I need a bash script which kills als SOCAT-Proccesses and restarts them again. I managed it via crontab after a reboot, but this produces too much downtime to reboot again after there are too many SOCAT-Proccesses.
I used
#!/bin/sh
killall socat &
sleep 3s
socat UDP4-LISTEN:PORT,fork,su=nobody UDP6:[IPV6]:PORT & disown
socat TCP4-LISTEN:PORT2,fork,su=nobody TCP6:[IPV6]:PORT2 & disown
exit
Now I have the problem that the script
does not run completely in background
often stops after executing the first SOCAT-Command.
I also tried nohup, but it also does not run completely in background.
What exactly should instigate this script to run?
Do you want to run it periodically, manually or when socat fails?
We certainly should understand why socat is failing in the first place.
I think it would be a good idea to output some logging from your script and kill socat with SIGKILL (-9).
A starting point here:
$ cat /usr/local/bin/restart_ip_addr_fam_bridge.sh
#!/bin/sh
IPV6="::1"
PORT=10001
PORT2=10002
while [ true ]; do
killall -9 socat 2> /dev/null
socat -T3600 UDP4-LISTEN:$PORT,reuseaddr,fork,su=nobody UDP6:[$IPV6]:$PORT &
socat TCP4-LISTEN:$PORT2,reuseaddr,fork,su=nobody TCP6:[$IPV6]:$PORT2 &
# Wait for a request to run or you could wait for a fixed time here
while [ ! -f /tmp/req_restart_ip_addr_fam_bridge ]; do
sleep 3
done
rm -f /tmp/req_restart_ip_addr_fam_bridge
printf "%s: Restarting IP address family bridge\n" "$(date '+%D %T')" >> /tmp/restart_ip_addr_fam_bridge.log
chmod 666 /tmp/restart_ip_addr_fam_bridge.log
# Avoid busy wait
sleep 5
done
exit
You can run it at startup, for example, adding this line to /etc/rc.local:
/usr/local/bin/restart_ip_addr_fam_bridge.sh &
And request it to restart your socat bridges with:
$ touch /tmp/req_restart_ip_addr_fam_bridge
See the running log with:
$ cat /tmp/restart_ip_addr_fam_bridge.log
08/20/22 15:14:43: Restarting IP address family bridge
Test your socat bridges:
$ nc -6 -l ::1 10002 | $ nc -4 127.0.0.1 10002
Typed from IPv4 TCP client | Typed from IPv4 TCP client
Be careful restart_ip_addr_fam_bridge.sh here is running as root from rc.local script.
Probably this is not very desirable/safe depending on your application.
EDIT: Added timeout for socat UDP bridge as recommended by #dest-unreach.
I ran an ssh command doing the following: ssh user#remote "my command &". Now the process seems to be running in the background, but I cannot find it, and I want to end it. I've used netstat, but cannot find the process.
Didn't you expect it to run in the background? Thats what the & does. You can use ps af to show all of the processes running under your username. You can then kill it by PID.
Thanks everybody. I found the process doing ps aux. For some reason, the port that it was using wasn't being display in netstat.
I suggest some methods
sudo killall ssh
It may not be the best method to use this method, it is better to filter first and then close it
or
ps -o pid,cmd | grep ssh
kill -QUIT (pid)
To stop a program, send the QUIT signal.
I logged in a virtual machine via ssh and I tried to run a script in background, the script is shown below:
#!/bin/bash
APP_NAME=`basename $0`
CFG_FILE=$1
. $CFG_FILE #just some variables
CMD=$2
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
CUR_LOG_DIR=$LOGS_RUNNING
echo $$ > $PID_FILE
#Main script code
#This script shall be called using the following syntax
# $ nohup script_name output_dir &
TIMESTAMP=`date +"%Y%m%d%H%M%S"`
CAP_INTERFACE="eth0"
/usr/sbin/tcpdump -nei $CAP_INTERFACE -s 65535 -w file_result
rm $PID_FILE
The result should be tcpdump running in background, redirecting the command result to file_result.
The script is called with:
nohup $SCRIPT_NAME $CFG_FILE start &
And It is stopped calling the STOP_SCRIPT:
##STOP_SCRIPT
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
if [ -f $PID_FILE ]
then
PID=`cat $PID_FILE`
# send SIGTERM to kill all children of $PID
pkill -TERM -P $PID
fi
When I check the file_result, after running the stop script, It is empty.
What is happening? How can I solve it?
I found this link: https://it.toolbox.com/question/launching-tcpdump-processes-in-background-using-ssh-060614
The author seems to have faced a similar issue. They debate about race conditions, but I didn't understand completely.
I'm not sure what you're trying to accomplish by having the startup script itself continue to run, but here's an approach that I think accomplishes what you're trying to do, namely start tcpdump and have it continue to run immune to hangups via nohup. I've simplified things a bit for illustrative purposes - feel free to add any variables back as you see fit, such as the nohup.out output directory, TIMESTAMP, etc.
Script #1: tcpdump_start.sh
#!/bin/sh
rm -f nohup.out
nohup /usr/sbin/tcpdump -ni eth0 -s 65535 -w file_result.pcap &
# Write tcpdump's PID to a file
echo $! > /var/run/tcpdump.pid
Script #2: tcpdump_stop.sh
#!/bin/sh
if [ -f /var/run/tcpdump.pid ]
then
kill `cat /var/run/tcpdump.pid`
echo tcpdump `cat /var/run/tcpdump.pid` killed.
rm -f /var/run/tcpdump.pid
else
echo tcpdump not running.
fi
To start tcpdump, just run tcpdump_start.sh.
To stop the tcpdump instance started with tcpdump_start.sh, just run tcpdump_stop.sh.
The captured packets will be written to the file_result.pcap file, and yes, it's a pcap file, not a text file, so it helps to name it with the proper file extension. The tcpdump statistics will be written to the nohup.out file when tcpdump is terminated.
I too had faced problems when running tcpdump over an SSH session.
In my case, I was running
sudo nohup tcpdump -w {pcap_dump_file} {filter} > /dev/null 2>&1 &
Where, running this command over Paramiko SSH session as a background process was the problem.
To get around this, I used screen utility of Linux.
screen is an easy to use tool for long-running of processes as a service.
Might be an old post, but this is also relevant. I couldn;t understand why no file was being created only to realise that the file might not be created until a certain amount of data had been captured.
https://github.com/the-tcpdump-group/tcpdump/issues/485
I am writing a Ruby wrapper around Docker and nsenter. One of the command my tool provides is to start a Bash shell within a container. Currently, I am doing it like this:
payload = "sudo nsenter --target #{pid(container_name)} --mount --uts --ipc --net --pid -- env #{env} /bin/bash -i -l;"
Kernel.exec(payload)
In Ruby, Kernel#exec relies on the exec(2) syscall, hence there is no fork.
One issue is that the container sometime dies prematurely which effectively kills my newly created Bash prompt. I then get back the prompt originally used to run my Ruby tool, but I cannot see what I am typing anymore, the tty seems broken and running reset effectively solves the issue.
I'd like to conditionally run reset if the program I exec-ed crashes. I found that the following works well:
$ ./myrubytool || reset
Except I'd like to avoid forcing people using my tool to append || reset every time.
I have tried the following:
payload = "(sudo nsenter --target #{pid(container_name)} --mount --uts --ipc --net --pid -- env #{env} /bin/bash -i -l) || reset;"
But this surprisingly puts reset in the background (i.e. I can run reset by entering fg). One benefit is that the tty is working properly, but it's not really ideal.
Would you have any idea to solve this issue?
If terminal echo has been disabled in a terminal, then you can run the command stty echo to re-enable the terminal echo. (Conversely, stty -echo disables terminal echo, and stty -a displays all terminal settings.)
This is safe to run even if terminal echo is already enabled, so if you want to play it safe, you can do something like ./myrubytool ; stty echo which will re-enable terminal echo if it is disabled regardless of the exit status of your Ruby program. You can put this in a shell script if you want to.
It might be that there is a way to execute a command when the Ruby program exits (often referred to as a "trap"), but I'm not familiar enough with Ruby to know whether such capabilities exist.
However, if you are creating a script for general use, you probably should look into more robust techniques and not rely on workarounds.
How about this? It should do exactly what you want.
It runs the command in a separate process, waits on it, and if, when it finishes, the return value is not 0, it runs the command reset.
payload = "sudo nsenter --target #{pid(container_name)} --mount --uts --ipc --net --pid -- env #{env} /bin/bash -i -l;"
fork { Kernel.exec(payload) }
pid, status = Process.wait2
unless status.exitstatus == 0
system("reset")
end
EDIT
If all you want to do is turn echo back on, change the system("reset") line to system("stty echo").
My server deployment script triggers a long-running process through SSH, like so:
ssh host 'install.sh'
Since my internet connection at home is not the best, I can sometimes be disconnected while the install.sh is running. (This is easily simulated by closing the terminal window.) I would really like for the install.sh script to keep running in those cases, so that I don't end up with interrupted apt-get processes and similar nuisances.
The reason why install.sh gets killed seems to be that stdout and stderr are closed when the SSH session is yanked, so writing to them fails. (It's not an issue of SIGHUP, by the way -- using nohup makes no difference.) If I put touch ~/1 && echo this fails && touch ~/2 into install.sh, only ~/1 is created.
So running ssh host 'install.sh &> install.out' solves the problem, but then I lose any "live" progress and error output.
So my question is: What's an easy/idiomatic way to run a process through SSH so that it doesn't crash if SSH dies, but so that I can still see the output as it runs?
Solutions I have tried:
When I run things manually, I use screen for cases like this, but I don't think it will be of much help here because I need to run install.sh automatically from a shell script. Screen seems to be made for interactive use (it complains "Must be connected to a terminal.").
Using install.sh 2>&1 | tee install.out didn't help either (silly of me to think it might).
You can redirect stdout/stderr into install.out and then tail -f it. The following snippet actually works:
touch install.out && # so tail does not bark (race condition)
(install.sh < /dev/null &> install.out &
tail --pid "$!" -F install.out)
But surely there must a less awkward way to do the same thing?
Try using screen:
screen ./install.sh
If your ssh session gets interrupted, you can simply reattach to the session via another ssh connection:
screen -x
You can provide a terminal to your ssh session using the -t switch:
ssh -t server screen ./install.sh
install.sh 2>&1 | tee install.out
if the only issue is not getting stderr. You didn't say exactly why the tee wasn't acceptable. You may need the other nohup/stdin tweaks.