I have been trying to make a looping work while accessing via SSH Remote command but it doesn't work. The command works when ran directly in host.
scp file root#${SERVER}:/tmp
ssh ${SERVER} "
while IFS= read -r line; do
echo "$line"
done </tmp/file
"
I have tried using the single quotes in main script but it causing errors.
bash: line n: warning: here-document at line n delimited by end-of-file
Any advise will be much appreciated.
UPDATE
testfile
1
2
3
4
5
6
Script test
SERVER='client'
ssh ${SERVER} '
echo "inside remote ClientServer"
echo "cat testfile"
cat /tmp/testfile
while read line; do
echo "${line}"
done <</tmp/testfile
'
echo "Back to MasterServer..."
Terminal Result
root#server]# ./test
S
Kernel 4.14.35-1902.10.7.el7uek.x86_64 on an x86_64
inside remote ClientServer
cat testfile
1
2
3
4
5
6
bash: line 8: warning: here-document at line 8 delimited by end-of-file (wanted `/tmp/testfile')
Back to MasterServer...
Thank you.
You will probably want to use single quotes to pass the remote commands verbatim:
scp file root#${SERVER}:/tmp
ssh ${SERVER} '
while IFS= read -r line; do
echo "$line"
done </tmp/file
'
Ensure you're using </tmp/file, not <</tmp/file. The sequence << is used to start a here-document which is not what you want in this case.
Related
I have the following shell script. The purpose is to loop thru each line of the target file (whose path is the input parameter to the script) and do work against each line. Now, it seems only work with the very first line in the target file and stops after that line got processed. Is there anything wrong with my script?
#!/bin/bash
# SCRIPT: do.sh
# PURPOSE: loop thru the targets
FILENAME=$1
count=0
echo "proceed with $FILENAME"
while read LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done < $FILENAME
echo "\ntotal $count targets"
In do_work.sh, I run a couple of ssh commands.
The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.
This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.
To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.
A very simple and robust workaround is to change the file descriptor from which the read command receives input.
This is accomplished by two modifications: the -u argument to read, and the redirection operator for < $FILENAME.
In BASH, the default file descriptor values (i.e. values for -u in read) are:
0 = stdin
1 = stdout
2 = stderr
So just choose some other unused file descriptor, like 9 just for fun.
Thus, the following would be the workaround:
while read -u 9 LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done 9< $FILENAME
Notice the two modifications:
read becomes read -u 9
< $FILENAME becomes 9< $FILENAME
As a best practice, I do this for all while loops I write in BASH.
If you have nested loops using read, use a different file descriptor for each one (9,8,7,...).
More generally, a workaround which isn't specific to ssh is to redirect standard input for any command which might otherwise consume the while loop's input.
while read -r line; do
((count++))
echo "$count $line"
sh ./do_work.sh "$line" </dev/null
done < "$filename"
The addition of </dev/null is the crucial point here, though the corrected quoting is also somewhat important for robustness; see also When to wrap quotes around a shell variable?. You will want to use read -r unless you specifically require the slightly odd legacy behavior you get for backslashes in the input without -r. Finally, avoid upper case for your private variables.
Another workaround of sorts which is somewhat specific to ssh is to make sure any ssh command has its standard input tied up, e.g. by changing
ssh otherhost some commands here
to instead read the commands from a here document, which conveniently (for this particular scenario) ties up the standard input of ssh for the commands:
ssh otherhost <<'____HERE'
some commands here
____HERE
ssh -n option prevents checking the exit status of ssh when using HEREdoc while piping output to another program.
So use of /dev/null as stdin is preferred.
#!/bin/bash
while read ONELINE ; do
ssh ubuntu#host_xyz </dev/null <<EOF 2>&1 | filter_pgm
echo "Hi, $ONELINE. You come here often?"
process_response_pgm
EOF
if [ ${PIPESTATUS[0]} -ne 0 ] ; then
echo "aborting loop"
exit ${PIPESTATUS[0]}
fi
done << input_list.txt
This was happening to me because I had set -e and a grep in a loop was returning with no output (which gives a non-zero error code).
I have the following shell script. The purpose is to loop thru each line of the target file (whose path is the input parameter to the script) and do work against each line. Now, it seems only work with the very first line in the target file and stops after that line got processed. Is there anything wrong with my script?
#!/bin/bash
# SCRIPT: do.sh
# PURPOSE: loop thru the targets
FILENAME=$1
count=0
echo "proceed with $FILENAME"
while read LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done < $FILENAME
echo "\ntotal $count targets"
In do_work.sh, I run a couple of ssh commands.
The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.
This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.
To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.
A very simple and robust workaround is to change the file descriptor from which the read command receives input.
This is accomplished by two modifications: the -u argument to read, and the redirection operator for < $FILENAME.
In BASH, the default file descriptor values (i.e. values for -u in read) are:
0 = stdin
1 = stdout
2 = stderr
So just choose some other unused file descriptor, like 9 just for fun.
Thus, the following would be the workaround:
while read -u 9 LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done 9< $FILENAME
Notice the two modifications:
read becomes read -u 9
< $FILENAME becomes 9< $FILENAME
As a best practice, I do this for all while loops I write in BASH.
If you have nested loops using read, use a different file descriptor for each one (9,8,7,...).
More generally, a workaround which isn't specific to ssh is to redirect standard input for any command which might otherwise consume the while loop's input.
while read -r line; do
((count++))
echo "$count $line"
sh ./do_work.sh "$line" </dev/null
done < "$filename"
The addition of </dev/null is the crucial point here, though the corrected quoting is also somewhat important for robustness; see also When to wrap quotes around a shell variable?. You will want to use read -r unless you specifically require the slightly odd legacy behavior you get for backslashes in the input without -r. Finally, avoid upper case for your private variables.
Another workaround of sorts which is somewhat specific to ssh is to make sure any ssh command has its standard input tied up, e.g. by changing
ssh otherhost some commands here
to instead read the commands from a here document, which conveniently (for this particular scenario) ties up the standard input of ssh for the commands:
ssh otherhost <<'____HERE'
some commands here
____HERE
ssh -n option prevents checking the exit status of ssh when using HEREdoc while piping output to another program.
So use of /dev/null as stdin is preferred.
#!/bin/bash
while read ONELINE ; do
ssh ubuntu#host_xyz </dev/null <<EOF 2>&1 | filter_pgm
echo "Hi, $ONELINE. You come here often?"
process_response_pgm
EOF
if [ ${PIPESTATUS[0]} -ne 0 ] ; then
echo "aborting loop"
exit ${PIPESTATUS[0]}
fi
done << input_list.txt
This was happening to me because I had set -e and a grep in a loop was returning with no output (which gives a non-zero error code).
I have the following shell script. The purpose is to loop thru each line of the target file (whose path is the input parameter to the script) and do work against each line. Now, it seems only work with the very first line in the target file and stops after that line got processed. Is there anything wrong with my script?
#!/bin/bash
# SCRIPT: do.sh
# PURPOSE: loop thru the targets
FILENAME=$1
count=0
echo "proceed with $FILENAME"
while read LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done < $FILENAME
echo "\ntotal $count targets"
In do_work.sh, I run a couple of ssh commands.
The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.
This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.
To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.
A very simple and robust workaround is to change the file descriptor from which the read command receives input.
This is accomplished by two modifications: the -u argument to read, and the redirection operator for < $FILENAME.
In BASH, the default file descriptor values (i.e. values for -u in read) are:
0 = stdin
1 = stdout
2 = stderr
So just choose some other unused file descriptor, like 9 just for fun.
Thus, the following would be the workaround:
while read -u 9 LINE; do
let count++
echo "$count $LINE"
sh ./do_work.sh $LINE
done 9< $FILENAME
Notice the two modifications:
read becomes read -u 9
< $FILENAME becomes 9< $FILENAME
As a best practice, I do this for all while loops I write in BASH.
If you have nested loops using read, use a different file descriptor for each one (9,8,7,...).
More generally, a workaround which isn't specific to ssh is to redirect standard input for any command which might otherwise consume the while loop's input.
while read -r line; do
((count++))
echo "$count $line"
sh ./do_work.sh "$line" </dev/null
done < "$filename"
The addition of </dev/null is the crucial point here, though the corrected quoting is also somewhat important for robustness; see also When to wrap quotes around a shell variable?. You will want to use read -r unless you specifically require the slightly odd legacy behavior you get for backslashes in the input without -r. Finally, avoid upper case for your private variables.
Another workaround of sorts which is somewhat specific to ssh is to make sure any ssh command has its standard input tied up, e.g. by changing
ssh otherhost some commands here
to instead read the commands from a here document, which conveniently (for this particular scenario) ties up the standard input of ssh for the commands:
ssh otherhost <<'____HERE'
some commands here
____HERE
ssh -n option prevents checking the exit status of ssh when using HEREdoc while piping output to another program.
So use of /dev/null as stdin is preferred.
#!/bin/bash
while read ONELINE ; do
ssh ubuntu#host_xyz </dev/null <<EOF 2>&1 | filter_pgm
echo "Hi, $ONELINE. You come here often?"
process_response_pgm
EOF
if [ ${PIPESTATUS[0]} -ne 0 ] ; then
echo "aborting loop"
exit ${PIPESTATUS[0]}
fi
done << input_list.txt
This was happening to me because I had set -e and a grep in a loop was returning with no output (which gives a non-zero error code).
I am trying to set my Linux shell script to read from a file (which I have working) but if there isn't any file then I need to read from stdin.
The command for reading a file looks like this:
./stats -row test_file
How would I be able to read what the user enters with something like this:
./stats -row 4 2 3 5 3 4 5 3 6 5 6 3 4
When I enter a command like this I get 'no such file or directory'
I broke my script down to the problem I need help with.
#!/bin/sh
INPUT_FILE=$2 #Argument 2 from command line is the input file
exec 5< $INPUT_FILE #assign input file to file descriptor #5
while read -u 5 line #read from file descriptor 5 (input file)
do
echo "$line"
done
exec 5<&- #close file descriptor #5
This also won't work for the input I need.
while read line
do
echo "$line"
done <$2
InArtful Solution
A very in-artful if statement will do the trick:
INPUT_FILE=$2 #Argument 2 from command line is the input file
if [ -f "$INPUT_FILE" ]; then
while read -r line
do
echo "$line"
done <"$INPUT_FILE"
else
while read -r line
do
echo "$line"
done
fi
Note: this presumes you are still looking for the filename as the 2nd argument.
Artful Solution
I cannot take credit, but the artful solution was already answered here: How to read from file or stdin in bash?
INPUT_FILE=${2:-/dev/stdin} #Argument 2 from command line is the input file
while read -r line
do
echo "$line"
done <"$INPUT_FILE"
exit 0
I was picking around with a solution like this but missed the stdin device /dev/stdin as the default for INPUT_FILES. note this solution is limited to OS's with a proc-filesystem.
In bash scripts, I usually put code that reads from a file (or a pipe) in a function, where the redirection can be separated from the logic.
Also, when reading from a file or from STDIN, it's a good idea for the logic to not care which is which. So, it's best to capture STDIN into a temp file and then the rest of the file reading code is the same.
Here's an example script that reads from ARG 1 or from STDIN, and just counts the lines in the file. It also invokes wc -l on the same input and shows the results from both methods.
#!/bin/bash
# default input is this script
input=$0
# If arg given, read from it
if (( $# > 0 )); then
input=$1
echo 1>&2 "Reading from $input"
else
# otherwise, read from STDIN
# since we're reading twice, need to capture it into
# a temp file
input=/tmp/$$.tmp
cat >$input
trap "rm -f $input" EXIT ERR HUP INT QUIT
echo 1>&2 "Reading from STDIN (saved to $input)"
fi
count_lines() {
local count=0
while read line ; do
let count+=1
done
echo $count
}
lines1=`count_lines <$input`
lines2=`wc -l <$input`
fmt="%15s: %d\n"
printf "$fmt" 'count_lines' $lines1
printf "$fmt" 'wc -l' $lines2
exit
Here are two invocations: one with a file on arg 1, and one with no argument, reading from STDIN:
$ ./t2.sh t2.sh
Reading from t2.sh
count_lines: 35
wc -l: 35
$ ./t2.sh <t2.sh
Reading from STDIN (saved to /tmp/8757.tmp)
count_lines: 35
wc -l: 35
I have this:
while read -r line; do echo "hello $line"; read -p "Press any key" -n 1; done < file
hello This is line 1
hello his is line 2
hello his is line 3
hello his is line 4
hello his is line 5
hello his is line 6
hello his is line 7
Why do I not see the prompt "Press any key" ?
Quote from man bash:
-p prompt
Display prompt on standard error, without a trailing new
line, before attempting to read any input. The prompt is
displayed only if input is coming from a terminal.
So, because you read lines from file but not from terminal prompt not displayed.
As others mentioned, you don't see the prompt because bash only prints the prompt when stdin is a terminal. In your case, stdin is a file.
But there's a bigger bug here: It seems to me that you want to read from two places: a file and the user. You'll have to do some redirection magic to accomplish this:
# back up stdin
exec 3<&0
# read each line of a file. the IFS="" prevents read from
# stripping leading and trailing whitespace in the line
while IFS="" read -r line; do
# use printf instead of echo because ${line} might have
# backslashes in it which some versions of echo treat
# specially
printf '%s\n' "hello ${line}"
# prompt the user by reading from the original stdin
read -p "Press any key" -n 1 <&3
done <file
# done with the stdin backup, so close the file descriptor
exec 3<&-
Note that the above code won't work with /bin/sh because it's not POSIX compliant. You'll have to use bash. I'd recommend making it POSIX compliant by changing the line that prompts the user:
printf 'Press enter to continue' >&2
read <&3
You may explicitly read from the controlling terminal /dev/tty:
while IFS="" read -r line; do
echo "hello $line"
read -p "Press any key" -n 1 </dev/tty
done < file