I mindlessly use nohup ssh for issuing a remote ssh command without worrying about the accident hangup. Now I'm starting to think about it and it is not pretty clear.
What I'm wondering is that why just doing "ssh remote sleep 100 &" stops the job after few seconds? For instance,
$ ssh remote sleep 100 &
[1] 13358
$
[1]+ Stopped ssh remote sleep 100
By what reason is this job stopped? Could you explain the internals of this job control?
If you want the remote command to keep working until it's finished (and not depend on the ssh connection with the remote host): you could use screen (or tmux) .
connect using ssh to the remote host
once connected: screen to start a screen session (a kind of "virtual terminal", that will keep running until you close it, instead of depending on your own connection to it)
you can then detach from screen (ctrl-a d) and re-attach to it later (from another machine, etc) : just ssh again, "screen -l" to list screens, and "screen -r" to re-attach to one. Read about screen on the net.
The reason your job is stopped is not linked to the command, but to the internal of job handling. Some (good) infos can be found on http://www.linusakesson.net/programming/tty/ (search for background if you don't read the whole thing. But read the whole thing ^^). In a nutshell ... writing to a TTY from a background job will cause a SIGTTOU to suspend the entire process group (maybe your ssh asked for a password? or it displays something when connecting?)
The advantage of screen over running on the remote host usign "nohup" are numerous. The main one is that if you try to re-connect to a nohup program (ex: vi) it can't (easily) be done... especially if it is multi-line. But when you re-attach to a screen session, you see the (virtual) terminal as if you never left it (ie, it's updated if the command added things on the screen, and it still have rows/columns, etc).
You can also work at several person on the same terminal (or have some person "view it" while one works in it).
Etc.
The command
ssh remote sleep 100 &
only runs ssh in the background. Once ssh is started on the local machine, control returns to the local shell, regardless of what is running (via sshd) on the remote end.
Related
I have a host machine that gets rebooted or reconnected quite a few times.
I want to have a script running on my dev machine that continuously tries to log into that machine and if successful runs a specific command (tailing the log data).
Edit: To clarify, the connection needs to stay open. The log command keeps tailing until I stop it manually.
What I have so far
#!/bin/bash
IP=192.168.178.1
if (("$#" >= 1))
then
IP=$1
fi
LOOP=1
trap 'echo "stopping"; LOOP=0' INT
while (( $LOOP==1 ))
do
if ping -c1 $IP
then
echo "Host $IP reached"
sshpass -p 'password' ssh -o ConnectTimeout=10 -q user#$IP '<command would go here>'
else
echo "Host $IP unreachable"
fi
sleep 1
done
The LOOP flag is not really used. The script is ended via CTRL-C.
Now this works if I do NOT add a command to be executed after the ssh and instead start the log output manually. On a disconnect the script keeps probing the connection and logs back in once the host is available again.
Also when I disconnect from the host (CTRL-D) the script will log right back into the host if CTRL-C is not pressed fast enough.
When I add a command to be executed after ssh the loop is broken. So pressing (CTRL-C) does not only stop the log but also disconnects and ends the script on the dev machine.
I guess I have to spawn another shell somewhere or something like that?
1) I want the script to keep probing, log in and run a command completely automatically and fall back to probing when the connection breaks.
2) I want to be able to stop the log on the host (CTRL-C) and thereby fall back to a logged in ssh connection to use it manually.
How do I fix this?
Maybe best approach on "fixing" would be fixing requirements.
The problematic part is number "2)".
The problem is from how SIGINT works.
When triggered, it is sent to the current control group related to your terminal. Mostly this is the shell and any process started from there. With more modern shells (you seem to use bash), the shell manages control groups such that programs started in the background are disconnected (by having been assigned a different control group).
In your case the ssh is started in the foreground (from a script executed in the foreground), so it will receive the interrupt, forward it to the remote and terminate as soon as the remote end terminated. As by that time the script shell has processed its signal handler (specified by trap) it is going to exit the loop and terminate itself.
So, as you can see, you have overloaded CTRL-C to mean two things:
terminate the monitoring script
terminate the remote command and continue with whatever is specified for the remote side.
You might get closer to what you want if you drop the first effect (or at least make it more explicit). Then, calling a script on the remote side that does not terminate itself but just the tail command, will be step. In that case you will likely need to use -t switch on ssh to get a terminal allocated for allowing normal shell operation later.
This, will not allow for terminating the remote side with just CTRL-C. You always will need to exit the remote shell that is going to be run.
The essence of such a remote script might look like:
tail command
shell
of course you would need to add whatever parts will be necessary for your shell or coding style.
An alternate approach would be to keep the current remote command being terminated and add another ssh call for the case of being interrupted that is spanning the shell for interactive use. But in that case, also `CTRL-C will not be available for terminating the minoring altogether.
To achieve this you might try changing active interrupt handler with your monitoring script to trigger termination as soon as the remote side returns. However, this will cause a race condition between the user being able to recognize remote command terminated (and control has been returned to local script) and the proper interrupt handler being in place. You might be able to sufficiently lower that risk be first activating the new trap handler and then echoing the fact and maybe add a sleep to allow the user to react.
Not really sure what you are saying.
Also, you should disable PasswordAuthentication in /etc/ssh/sshd_config and log by adding the public key of your home computer to `~/.ssh/authorized_keys
! /bin/sh
while [ true ];
do
RESPONSE=`ssh -i /home/user/.ssh/id_host user#$IP 'tail /home/user/log.txt'`
echo $RESPONSE
sleep 10
done
I am trying to execute two scripts which are available as sh files on remote host having 755 permissions.
I try callling them from client host as below:
REMOTE_HOST="host1"
BOUNCE_SCRIPT="
/code/sys/${ENV}/comp/1/${ENV}/scripts/unix/stopScript.sh ${ENV};
/code/sys/${ENV}/comp/1/${ENV}/scripts/unix/startScript.sh ${ENV};
"
ssh ${REMOTE_HOST} "${BOUNCE_SCRIPT}"
Above lines are in a script on local host.
While running the script on local host, the first command on remote host i.e. stopScript.sh gets executed correctly. It kills the running process which it was inteded to kill w/o any error.
However output of second script i.e. startScript.sh gets printed to local host window but the process it intended to start does not start on remote host.
Can anyone please let me know?
Is the way executing script on remote host correct?
Should I see output of running script on remote host locally as well? i.e. on the window of local host?
Thanks
You could try the -n flag for ssh:
ssh -n $REMOTE_HOST "$BOUNCE_SCRIPT" >> $LOG
The man page has further information (http://unixhelp.ed.ac.uk/CGI/man-cgi?ssh+1). The following is a snippet:
-n Redirects stdin from /dev/null (actually, prevents reading from
stdin).
Prefacing your startScript.sh line with 'nohup' may help. Often times if you remotely execute commands they will die when your ssh session ends, nohup allows your process to live after the session has ended. It would be helpful to know if your process is starting at all or if it starts and then dies.
I think cyber-monk is right, you should launch the processes with nohup to create à new independent process. Look if your stop script is killing the right process (the new one included).
I am using SSH to start a background process on a remote server. This is what I have at the moment:
ssh remote_user#server.com "nohup process &"
This works, in that the process does start. But the SSH session itself does not end until I hit Ctr-C.
When I hit Ctr-C, the remote process continues to run in the background.
I would like to place the ssh command in a script that I can run locally, so I would like the ssh session to exit automatically once the remote process has started.
Is there a way to make this happen?
The "-f" option to ssh tells ssh to run the remote command in the background and to return immediately. E.g.,
ssh -f user#host "echo foo; sleep 5; echo bar"
If you type the above, you will get your shell prompt back immediately, you will then see "foo" output. Five seconds later you will then see "bar" output. In the meantime, you could have been using the shell.
When using nohup, make sure you also redirect stdin, stdout and stderr:
ssh user#server 'DISPLAY=:0 nohup xeyes < /dev/null > std.out 2> std.err &'
In this way you will be completely detached from the remote process. Be carefull with using ssh -f user#host... since that will only put the ssh process in the background on the calling side. You can verify this by running a ps -aux | grep ssh on the calling machine and this will show you that the ssh call is still active, but just put in the background.
In my example above I use DISPLAY=:0 since xeyes is an X11 program and I want it started on the remote machine.
You could use screen to run your process on this screen, detach from screen Ctrl-a :detach and exit your current session without problem. Then you can reconnect to SSH and attach to this screen again to continue with your task or check if is finished.
Or you can send the command to an already running screen. Your local script should look like this:
ssh remote_user#server.com
screen -dmS new_screen sh
screen -S new_screen -p 0 -X stuff $'nohup process \n'
exit
For more info see this tutorial
Well this question is almost 10 years old, but I recently had to launch a very long script (taking several hours to complete) on a remote server and I found a way using the crontab.
If can edit your user's crontab on the remote server, connect with ssh to the server, edit the crontab and add an entry that will start your script the next minute. Let's say it's 15h03. Add this line :
4 15 * * * /path/to/your/script.sh
save your crontab, wait a minute for the script to be launched. Then edit again your crontab to remove this entry.
You can then safely exit ssh, even shut down your computer while the script is running.
I'm connecting to my ubuntu server using ssh. I start an encoding program using a command. However, it seems that when my ssh session closes (because I started it on a laptop which went to sleep). Is there a way to avoid this (of course preventing my laptop from sleeping is not a permanent solution).
Run your command with nohup or use screen
nohup is better when your program generate some loging output because it's forward to file and then you can check it, but with screen you can detach ssh session and when you log again you can restore your work-space. For encoding I'll use nohup. It is easier and you probably run it in background, so you really don't need detaching
Screen is the best for you.
screen -S some_name
than run it. Detach it with: ctrl+a d
Next time, attach it with:
screen -rd some_name
You can have more runnning screens. To show the list of them:
screen -ls
Install "screen" on your ubuntu server, that way you can start any program in your session, disconnect the output of the program from your current session and exit.
Later when you connect again to your server, you can restore the program which will continue running and see its progress.
I've searched, googled, sat in IRC for a week and even talked to a friend who is devoutly aligned with linux but I haven't yet received a solid answer.
I have written a shell script that runs as soon as I log into my non-root user and runs basically just does "./myprogram &" without quotation. When I exit shh my program times out and I am unable to connect to it until I log back in. How can I keep my program running after I exit SSH of my non-root user?
I am curious if this has to be done on the program level or what? My apologizes if this does not belong here, I am not sure where it goes to be perfectly honest.
Beside using nohup, you can run your program in terminal multiplexer like screen or tmux. With them, you can reattach to sessions, which is for example quite helpful if you need to run terminal-based interactive programs or long time running scripts over a unstable ssh connections.
boybu is a nice enhancement of screen.
Try nohup: http://linux.die.net/man/1/nohup
Likely your program receives a SIGHUP signal when you exit your ssh session.
There's two signals that can cause your program to die after your ssh session ends: SIGHUP and SIGPIPE.
SIGHUP will be sent to your program because the parent process (ssh) has died. You can get around this either by using the program nohup (i.e. nohup ./myprogram &) or by using the shell builtin disown (./myprogram& disown)
SIGPIPE will be sent to your program if it tries to write to stdout or stderr after the ssh session has been disconnected. To get around this, redirect them to a file or /dev/null, i.e. nohup ./myprogram >/dev/null 2>/dev/null &
You might also want to use the batch (or at) command, in addition to the other answers (nohup, screen, ...). And ssh has a -f option which might interest you.