I am running a python 3 script via ssh, and I want to see the stdout and stderr in a file on the remote server.
Also, I would like to see the file updated live while the script is running and that the script will run in the background so the ssh connection will not wait for the script to finish.
While looking at other questions, I managed to answer most of my requests.
Here is what I come up with:
ssh user#machine_ip "(python3 my_script.py 2>&1 | tee output.log) &"
The problem is that the ssh is waiting for the script to finish.
So combining the answer from this question with some hackidy hack nonsense...
Really all you need to do is this:
(( python3 my_script.py 0<&- 2>&1 | tee -a ${OUTFILE} | nc -kl ${PORT} &) &)
Explanation:
Run your python script: python3 my_script.py
Detach stdin (so it can run independently): ... 0<&-
Redirect stderr to stdout so we can pipe them along together: ... 2>&1
Append output to some file but also keep the output going to stdout so we can pipe it someplace else: ... | tee -a ${OUTFILE} |
Pipe stdout to a netcat listening port so this machine can essentially "serve" the stdout from your python script: ... | nc -kl ${PORT}
Create a "double nested background subshell" this is explained in the link above but this will allow you to orphan "blah" so that it'll run even if your ssh connection ends (( ... blah ... &) &)
To view the stdout/stderr of my_script.py you now have several options. If you are still logged into that remote machine you can:
tail -f ${OUTFILE} # Same "OUTFILE" used in explanation component 4
nc localhost $PORT # Same "PORT" used in explanation component 5
If you are no longer logged in and you are now on a different machine (but on the same network) you can:
nc ${remote_machine} $PORT # Same "PORT" used in explanation component 5
Where ${remote_machine} is the hostname or IP address of the machine you ssh'ed into to run your command
Related
I logged in a virtual machine via ssh and I tried to run a script in background, the script is shown below:
#!/bin/bash
APP_NAME=`basename $0`
CFG_FILE=$1
. $CFG_FILE #just some variables
CMD=$2
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
CUR_LOG_DIR=$LOGS_RUNNING
echo $$ > $PID_FILE
#Main script code
#This script shall be called using the following syntax
# $ nohup script_name output_dir &
TIMESTAMP=`date +"%Y%m%d%H%M%S"`
CAP_INTERFACE="eth0"
/usr/sbin/tcpdump -nei $CAP_INTERFACE -s 65535 -w file_result
rm $PID_FILE
The result should be tcpdump running in background, redirecting the command result to file_result.
The script is called with:
nohup $SCRIPT_NAME $CFG_FILE start &
And It is stopped calling the STOP_SCRIPT:
##STOP_SCRIPT
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
if [ -f $PID_FILE ]
then
PID=`cat $PID_FILE`
# send SIGTERM to kill all children of $PID
pkill -TERM -P $PID
fi
When I check the file_result, after running the stop script, It is empty.
What is happening? How can I solve it?
I found this link: https://it.toolbox.com/question/launching-tcpdump-processes-in-background-using-ssh-060614
The author seems to have faced a similar issue. They debate about race conditions, but I didn't understand completely.
I'm not sure what you're trying to accomplish by having the startup script itself continue to run, but here's an approach that I think accomplishes what you're trying to do, namely start tcpdump and have it continue to run immune to hangups via nohup. I've simplified things a bit for illustrative purposes - feel free to add any variables back as you see fit, such as the nohup.out output directory, TIMESTAMP, etc.
Script #1: tcpdump_start.sh
#!/bin/sh
rm -f nohup.out
nohup /usr/sbin/tcpdump -ni eth0 -s 65535 -w file_result.pcap &
# Write tcpdump's PID to a file
echo $! > /var/run/tcpdump.pid
Script #2: tcpdump_stop.sh
#!/bin/sh
if [ -f /var/run/tcpdump.pid ]
then
kill `cat /var/run/tcpdump.pid`
echo tcpdump `cat /var/run/tcpdump.pid` killed.
rm -f /var/run/tcpdump.pid
else
echo tcpdump not running.
fi
To start tcpdump, just run tcpdump_start.sh.
To stop the tcpdump instance started with tcpdump_start.sh, just run tcpdump_stop.sh.
The captured packets will be written to the file_result.pcap file, and yes, it's a pcap file, not a text file, so it helps to name it with the proper file extension. The tcpdump statistics will be written to the nohup.out file when tcpdump is terminated.
I too had faced problems when running tcpdump over an SSH session.
In my case, I was running
sudo nohup tcpdump -w {pcap_dump_file} {filter} > /dev/null 2>&1 &
Where, running this command over Paramiko SSH session as a background process was the problem.
To get around this, I used screen utility of Linux.
screen is an easy to use tool for long-running of processes as a service.
Might be an old post, but this is also relevant. I couldn;t understand why no file was being created only to realise that the file might not be created until a certain amount of data had been captured.
https://github.com/the-tcpdump-group/tcpdump/issues/485
What is a good way to be able to directly send to STDIN and receive from STDOUT of a process? I'm specifically interested in SSH, as I want to do the following:
[ssh into a remote server]
[run remote commands]
[run local commands]
[run remote commands]
etc...
For example, let's say I have a local script "localScript" that will output the next command I want to run remotely, depending on the output of "remoteScript". I could do something like:
output=$(ssh myServer "./remoteScript")
nextCommand=$(./localScript $output)
ssh myServer "$nextCommand"
But it would be nice to do this without closing/reopening the SSH connection at every step.
You can redirect SSH input and output to FIFO-s and then use these for two-way communication.
For example local.sh:
#!/bin/sh
SSH_SERVER="myServer"
# Redirect SSH input and output to temporary named pipes (FIFOs)
SSH_IN=$(mktemp -u)
SSH_OUT=$(mktemp -u)
mkfifo "$SSH_IN" "$SSH_OUT"
ssh "$SSH_SERVER" "./remote.sh" < "$SSH_IN" > "$SSH_OUT" &
# Open the FIFO-s and clean up the files
exec 3>"$SSH_IN"
exec 4<"$SSH_OUT"
rm -f "$SSH_IN" "$SSH_OUT"
# Read and write
counter=0
echo "PING${counter}" >&3
cat <&4 | while read line; do
echo "Remote responded: $line"
sleep 1
counter=$((counter+1))
echo "PING${counter}" >&3
done
And simple remote.sh:
#!/bin/sh
while read line; do
echo "$line PONG"
done
The method you are using works, but I don't think you can reuse the same connection everytime. You can, however, do this using screen, tmux or nohup, but that would greatly increase the complexity of your script because you will now have to emulate keypresses/shortcuts. I'm not even sure if you can if you do directly in bash. If you want to emulate keypresses, you will have to run the script in a new x-terminal and use xdotool to emulate the keypresses.
Another method is to delegate the whole script to the SSH server by just running the script on the remote server itself:
ssh root#MachineB 'bash -s' < local_script.sh
Why my named pipe input command line just hangs when it is called?
Based on the answers:
Writing to stdin of background process
Accessing bash command line args $# vs $*
Send command to a background process
Can I redirect output to a log file and background a process at the same time?
I wrote two shell scripts to communicate with my game server. And worked the first time I did it. Since it them they do not work anymore. Every time I do ./send.sh commands the command line hangs until I hit Ctrl+C.
It also hangs and does nothing when I do directly echo commamd > /tmp/srv-input
The scripts
It does start the server and configure it to read/receive my commands while it run in background:
start_czero_server.sh
#!/bin/sh
# Go to the game server application folder where the game application `hlds_run` is
cd /home/user/Half-Life
pkill -f hlds
# Set up a pipe named `/tmp/srv-input`
rm /tmp/srv-input
mkfifo /tmp/srv-input
cat > /tmp/srv-input &
echo $! > /tmp/srv-input-cat-pid
# Start the server reading from the pipe named `/tmp/srv-input`
# And also output all its console to the file `/home/user/Half-Life/my_logs.txt`
cat /tmp/srv-input | ./hlds_run -console -game czero +port 27015 > my_logs.txt 2>&1 &
# Successful execution
exit 0
This second script it just a wrapper which allow me easily to send commands to the my server:
send.sh
#!/bin/sh
echo "$#" > /tmp/srv-input
# Successful execution
exit 0
Now every time I want to send a command to my server I just do on the terminal:
./send.sh mp_timelimit 30
I always keep another open terminal open just to listen to my server server console. To do it just use the tail command with the -f flag to follow my server console output:
./tail -f /home/user/Half-Life/my_logs.txt
You would be better off just having hlds_run read directly from the pipe instead of having cat pipe it in.
Try
./hlds_run … > my_logs.txt 2>&1 < /tmp/srv-input &
Instead of
cat /tmp/srv-input | ./hlds_run …
I send a tee command from host 1 to host 2:
ssh user#host2 '/path/run |& tee myFile.txt'
I use tee so that I get the output of the binary to be added to myFile.txt
The problem I have then is after a bit of time, I want to regain control of my local host without having a lot of printout. So I do CTRL+C. This lets the process on host2 continue to run, which is what I want, but it stops the tee process itself, so the file is not populated.
I tried to replace |& tee myFile.txt' by 2>&1 myFile.txt' & but it did not help.
How can I ensure that the file continues to be populated on host2, while regaining control to my session on host1?
If you want to record the results in some file (work with IO redirection inside of the nohup), you need to enclose all the pipeline in the nohup. It does not use shell expansions, since argument is only COMMAND ARGS, so using a sh is a good way:
ssh user#host2 'nohup sh -c "/path/run |& tee myFile.txt" &'
but note that nohup will disconnect the terminal from the command ant it might fail. Useful would be to redirect it directly to the file:
ssh user#host2 'nohup sh -c "/path/run &> myFile.txt" &'
Inspiration from the SO answer.
use nohup, screen or tmux for backgrounding a process.
In a very simplified scenario, I have a script that looks like this:
mv test _test
sleep 10
echo $1
mv _test test
and if I execute it with:
ssh localhost "test.sh foo"
the test file will have an underscore in the name as long as the script is running, and when the script is finished, it will send foo back. The script SHOULD keep running, even if you terminate the ssh command by pressing ctrl+c or if you lose connection the the server, but it doesn't (the file is not renamed back to "test"). So, I tried the following:
nohup ssh localhost "test.sh foo"
and it makes ssh immune to ctrl+c but flaky connection to the server still causes trouble. After some debugging, it turns out that the script WILL actually reach the end IF THERE IS NO ECHO IN IT. And when you think about it, it makes sense - when the connection is dropped, there is no more stdout (ssh socket) to echo to, so it will fail, silently.
I can, of course, echo to a file and then get the file, but I would prefer something smarter, along the lines of test tty && echo $1 (but tty invoked like this always returns false). Any suggestions are greatly appreciated.
The following command does what you want:
ssh -t user#host 'nohup ~/test.sh foo > nohup.out 2>&1 & p1=$!; tail -f ~/nohup.out & wait $p1'
... test.sh is located in the users home directory
Explanation:
1.) "ssh -t user#host " ... pretty clear ... starts remote session
2.) "nohup ~/test.sh foo > nohup.out 2>&1" ... starts the test.sh script with nohup in background
3.) "p1=$!;" ... stores the child pid of the previous command in p1
4.) "tail -f ~/nohup.out &" ... tail nohup.out in background to see the output of test.sh
5.) "wait $p1" ... waits for proccess test.sh (which pid is stored in p1) to finish
The above command works even if you interrupt it with ctrl+c.
you can use ...
ssh -t localhost "test.sh foo"
... to force a tty allocation
As st0ne suggested, tail fails, but does not cause the script to terminate, as opposed to cat and echo. So, there is no need for nohup, redirecting stdout to a temporary file, etc. just plain and simple:
mv test _test
sleep 10
echo $1 | tail
mv _test test
and execute it with:
ssh localhost "test.sh foo"