Bash: Using SSH to start a long-running remote command and collect its PID - linux

When I do the following, then I have to press CTRL-c afterwards or the shell acts weird. Left/right arrows keys e.g. doesn't move correctly and the text is messed up.
# read -r pid < <(ssh 10.10.10.46 'sleep 50 & echo $!') ; echo $pid
2135
# Killed by signal 2.
^C
#
I need this for a script, so I'd like to know why CTRL-c is needed and is it possible to work around it?
Update
It looks like it opens an extra Bash shell, and that is the one that needs to be exited.
The command I am actually interesting in is
read -r pid < <(ssh 10.10.10.46 "mbuffer -4 -v 0 -q -I 8023 > /tmp/mtest & echo $!"); echo $pid

Try this instead:
read -r pid \
< <(ssh 10.10.10.46 'nohup mbuffer >/tmp/mtest </dev/null 2>/tmp/mtest.err & echo $!')
Three important changes:
Use of nohup (you could also get a similar effect with the bash built-in disown)
Redirection of stdin and stderr to files (preventing them from holding handles that connect, eventually, to your terminal).
Use of single quotes for the remote command (with double-quotes, expansions happen before ssh is started, so the $! you get is the PID of the most recently started local background process).

Related

nohup append the executed command at the top of the output file

Let's say that we invoke the nohup in the following way:
nohup foo.py -n 20 2>&1 &
This will write the output to the nohup.out.
How could we achieve to have the whole command nohup foo.py -n 20 2>&1 & sitting at the top of the nohup.out (or any other specified output file) after which the regular output of the executed command will be written to that file?
The reason for this is for purely debugging purpose as there will be thousands of commands like this executed and very often some of them will crash due to various reasons. It's like a basic report kept in a file with the executed command written at the top followed by the output of the executed command.
A straightforward alternative would be something like:
myNohup() {
(
set +m # disable job control
[[ -t 0 ]] && exec </dev/null # redirect stdin away from tty
[[ -t 1 ]] && exec >nohup.out # redirect stdout away from tty
[[ -t 2 ]] && exec 2>&1 # redirect stderr away from tty
set -x # enable trace logging of all commands run
"$#" # run our arguments as a command
) & disown -h "$!" # do not forward any HUP signal to the child process
}
To define a command we can test this with:
waitAndWrite() { sleep 5; echo "finished"; }
...and run:
myNohup waitAndWrite
...will return immediately and, after five seconds, leave the following in nohup.out:
+ waitAndWrite
+ sleep 5
+ echo finished
finished
If you only want to write the exact command run without the side effects of xtrace, replace the set -x with (assuming bash 5.0 or newer) printf '%s\n' "${*#Q}".
For older versions of bash, you might instead consider printf '%q ' "$#"; printf '\n'.
This does differ a little from what the question proposes:
Redirections and other shell directives are not logged by set -x. When you run nohup foo 2>&1 &, the 2>&1 is not passed as an argument to nohup; instead, it's something the shell does before nohup is started. Similarly, the & is not an argument but an instruction to the shell not to wait() for the subprocess to finish before going on to future commands.

How to assign to a bash variable an ssh remote command the pid while capturing its

Introduction
My question is very similar to this one, except that I'd like the output from the command to be redirected to a local file instead of a remote one.
The questioner was asking for a way to retrieve the process ID with a command similar to this one, where the mbuffer command wouldn't cause hanging:
read -r pid < <(ssh 10.10.10.46 "mbuffer -4 -v 0 -q -I 8023 > /tmp/mtest & echo $!"); echo $pid
The answerer responded with the following command to resolve the problem
read -r pid \
< <(ssh 10.10.10.46 'nohup mbuffer >/tmp/mtest </dev/null 2>/tmp/mtest.err & echo $!')
Which is really helpful but still places files on the remote machine, not the local one.
My Attempts
The following is my attempt to capture a log of the output of $command:
read -r PID < <(ssh $remote 'nohup $command >&2 & echo $!' 2> $log)
Which sets PID to the process ID properly but doesn't produce a log.
Question
How can I capture a log on my local machine of the stdout of my $command while still assigning PID to the process ID of $command?
Another approach:
{ read -r pid;
# Do whatever you want with $pid of the process on remote machine
cat > my_local_system_log_file
} <(ssh 10.10.10.46 "mkfifo /tmp/mtest; mbuffer -4 -v 0 -q -I 8023 &> /tmp/mtest & echo $!; cat /tmp/mtest");
Basically, the first line is PID & further lines are logs from the process.

How can I quit the bash when I acident run while with pipe

I run the this command
$(while true;do echo Something && sleep 0.01; done;) | cat
Now I can not exit by Ctrl+C or background it by Ctrl+Z, and ps aux can't tell me which bash it is.How can I quit that bash ?
EDIT
I narrow donw the pid by find the cwd pgrep bash| (while read -r line; do lsof -p $line|grep cwd|grep EXPECTED_CWD && echo "GOT $line"; done;),finally kill that process.There is easier way to find that, but no /proc on mac.
Closing the terminal can help.
If you are using CLI-only os then you can switch to different terminal using Ctrl + Alt + F1 keys.
There you can use who or ps commands to get process-id and kill it.
ps -ef will list all processes.
you can kill second last bash process, as that will be the last command that you executed.
Note : process id assignment in linux is always greater than old one.
you're able to execute your commands with & symbol in the end of the command to run this command in background
In your case command looks like:
$(while true;do echo Something && sleep 0.01; done;) | cat &
It will show the process ID, and you'll be able to kill it, when you want

terminate infinite loop initiated in remote server when exiting bash script

Script which executes commands in infinite loop in background
<SOMETHING ELSE AT START OF SCRIPT>
cmd='while true;
do
ps aux | head;
sleep 1;
done > $FILE'
ssh root#$SERVER $cmd &
...
...
<SOME OTHER TASKS>
...
...
( at the end of this script, how to kill the above snippet executing in remote server)
[ kindly note i dont want to wait as the while loop is infinite ]
Read and tried some posts from stackoverflow, but could not find exact solution for this problem.
Rather than an infinite loop, use a sentinel file:
cmd='while [ -r /tmp/somefile];
do
# stuff
done > $FILE'
ssh root#$SERVER touch /tmp/somefile
ssh root#$SERVER $cmd &
# do other stuff
ssh root#$SERVER rm -f /tmp/somefile
This follows your current practice of putting the remote command in a variable, but the arguments against that cited elsewhere should be considered.
If you want to kill the ssh process running in background at the end of your script, just do:
kill $!
I assume this is the only (or the last) process you started in background.
Try following sequence
CTRL+Z
fg
CTRL+C
or
jobs
kill %jobspec
To kill everything belonging to user logged in you could try:
whois=`w|grep $user|awk '{print $2}'`;user=root; ssh $user#server -C "ps auwx|grep $whois|awk '{print \$2}'"
This will list all the processes owned by the user you just logged in as - just add |xargs kill -9
whois=`w|grep $user|awk '{print $2}'`;user=root; ssh $user#server -C "ps auwx|grep $whois|awk '{print \$2}'|xargs kill -9 "
whois=`w|grep $user|awk '{print $2}'`;user=root; ssh $user#server -C "ps auwx|grep $whois|awk '{print \$2}'|awk '{print "kill -9 " $1}'|/bin/sh "

Obtaining PID of Last Command Run on Other Machine

I am using the following code to launch a command on another machine:
#!/bin/bash
/usr/bin/rsh -n $Host_Name "cat asdf.txt &"
And I am trying to obtain the PID of the cat command by using the following:
/usr/bin/rsh -n $Host_Name pid="$!"
But when I echo $pid, it is just blank. What am I doing incorrectly? Is there an easier way of obtaining the PID of the last command that was executed on a different machine?
Thanks
You can only get the $! of the backgrounded command in the shell in which you started the command. If your command doesn't output anything to stderr, this could work:
/usr/bin/rsh -n $Host_Name "cat asdf.txt & echo $! >&2" 2> pidfile
The pid of the started command will then be stored locally in 'pidfile'.
Just a side-note: I would never use rsh. It is inherently insecure. I'd use ssh instead ;)

Resources