Looping through lines in a file in bash, without using stdin - linux

I am foxed by the following situation.
I have a file list.txt that I want to run through line by line, in a loop, in bash. A typical line in list.txt has spaces in. The problem is that the loop contains a "read" command. I want to write this loop in bash rather than something like perl. I can't do it :-(
Here's how I would usually write a loop to read from a file line by line:
while read p; do
echo $p
echo "Hit enter for the next one."
read x
done < list.txt
This doesn't work though, because of course "read x" will be reading from list.txt rather than the keyboard.
And this doesn't work either:
for i in `cat list.txt`; do
echo $i
echo "Hit enter for the next one."
read x
done
because the lines in list.txt have spaces in.
I have two proposed solutions, both of which stink:
1) I could edit list.txt, and globally replace all spaces with "THERE_SHOULD_BE_A_SPACE_HERE" . I could then use something like sed, within my loop, to replace THERE_SHOULD_BE_A_SPACE_HERE with a space and I'd be all set. I don't like this for the stupid reason that it will fail if any of the lines in list.txt contain the phrase THERE_SHOULD_BE_A_SPACE_HERE (so malicious users can mess me up).
2) I could use the while loop with stdin and then in each loop I could actually launch e.g. a new terminal, which would be unaffected by the goings-on involving stdin in the original shell. I tried this and I did get it to work, but it was ugly: I want to wrap all this up in a shell script and I don't want that shell script to be randomly opening new windows. What would be nice, and what might somehow be the answer to this question, would be if I could figure out how to somehow invoke a new shell in the command and feed commands to it without feeding stdin to it, but I can't get it to work. For example this doesn't work and I don't really know why:
while read p; do
bash -c "echo $p; echo ""Press enter for the next one.""; read x;";
done < list.txt
This attempt seems to fail because "read x", despite being in a different shell somehow, is still seemingly reading from list.txt. But I feel like I might be close with this one -- who knows.
Help!

You must open as a different file descriptor
while read p <&3; do
echo "$p"
echo 'Hit enter for the next one'
read x
done 3< list.txt
Update: Just ignore the lengthy discussion in the comments below. It has nothing to do with the question or this answer.

I would probably count lines in a file and iterate each of those using eg. sed. It is also possible to read infinitely from stdin by changing while condition to: while true; and exit reading with ctrl+c.
line=0 lines=$(sed -n '$=' in.file)
while [ $line -lt $lines ]
do
let line++
sed -n "${line}p" in.file
echo "Hit enter for the next ${line} of ${lines}."
read -s x
done
AWK is also great tool for this. Simple way to iterate through input would be like:
awk '{ print $0; printf "%s", "Hit enter for the next"; getline < "-" }' file

As an alternative, you can read from stderr, which by default is connected to the tty as well. The following then also includes a test for that assumption:
(
tty -s <& 2|| exit 1
while read -r line; do
echo "$line"
echo 'Hit enter'
read x <& 2
done < file
)

Related

As with the command: "echo '#!/bin/bash' |tee file", but with "echo '#!/bin/bash' | myscript file"

What "... | tee file" does is take stdin (standard input) and divert it to two places: stdout (standard output) and to a path/file named "file". In effect it does this, as far as I can judge:
#!/bin/bash
var=(cat) # same as var=(cat /dev/stdin)
echo -e "$var"
for file in "$#"
do
echo -e "$var" > "${file}"
done
exit 0
So I use the above code to create tee1 to see if I could emulate what tee does. But my real intent is to write a modified version that appends to existing file(s) rather than redo them from scratch. I call this one tee2:
#!/bin/bash
var=(cat) # same as var=(cat /dev/stdin)
echo -e "$var"
for file in "$#"
do
echo -e "$var" >> "${file}"
done
exit 0
It makes sense to me, but not to bash. Now an alternative approach is to do something like this:
echo -e "$var"
for file in "$#"
do
echo -e "$var"| tee tmpfile
cat tmpfile >> "${file}"
done
rm tmpfile
exit 0
It also makes sense to me to do this:
#!/bin/bash
cp -rfp /dev/stdin tmpfile
cat tmpfile
for file in "$#"
do
cat tmpfile >> "${file}"
done
exit 0
Or this:
#!/bin/bash
cat /dev/stdin
for file in "$#"
do
cat /dev/stdin >> "${file}"
done
exit 0
Some online searches suggest that printf be used in place of echo -e for more consistency across platforms. Other suggest that cat be used in place of read, though since stdin is a device, it should be able to be used in place of catm as in:
> tmpfile
IFS=\n
while read line
do
echo $line >> tmpfile
echo $line
done < /dev/stdin
unset IFS
Then the for loop follows. But I can't get that to work. How can I do it with bash?
But my real intent is to write a modified version that appends to existing file(s) rather than redo them from scratch.
The tee utility is specified to support an -a option, meaning "Append the output to the files." [spec]
(And I'm not aware of any implementations of tee that deviate from the spec in this regard.)
Edited to add: If your question is really "what's wrong with all the different things I tried", then, that's probably too broad for a single Stack Overflow question. But here's a short list:
var=(cat) means "Set the array variable var to contain a single element, namely, the string cat."
Note that this does not, in any way, involve the program cat.
You probably meant var=$(cat), which means "Run the command cat, capturing its standard output. Discard any null bytes, and discard any trailing sequence of newlines. Save the result in the regular variable var."
Note that even this version is not useful for faithfully implementing tee, since tee does not discard null bytes and trailing newlines. Also, tee forwards input as it becomes available, whereas var=$(cat) has to wait until input has completed. (This is a problem if standard input is coming from the terminal — in which case the user would expect to see their input echoed back — or from a program that might be trying to communicate with the user — in which case you'd get a deadlock.)
echo -e "$var" makes a point of processing escape sequences like \t. (That's what the -e means.) This is not what you want. In addition, it appends an extra newline, which isn't what you want if you've managed to set $var correctly. (If you haven't managed to set $var correctly, then this might help compensate for that, but it won't really fix the problem.)
To faithfully print the contents of var, you should write printf %s "$var".
I don't understand why you switched to the | tee tmpfile approach. It doesn't improve anything so far as I can tell, and it introduces the bug that now if you're copying to n files, then you will also write n copies to standard output. (You fixed that bug in later versions, though.)
The versions where you write directly to a file, instead of saving to a variable first, are a massive improvement in terms of faithfully copying the contents of standard input. But they still have the problem of waiting until input is complete.
The version where you cat /dev/stdin multiple times (once for each destination) won't work, because there's no "rewinding" of standard input. Once something is consumed, it's gone. (This makes sense when you consider that standard input is frequently passed around from program to program — your cat-s, for example, are inheriting it from your Bash script, and your Bash script may be inheriting it from the terminal. If some sort of automatic rewinding were to happen, how would it decide how far back to go?) (Note: if standard input is coming from a regular file, then it's possible to explicitly seek backward along it, and thereby "unconsume" already-consumed input. But that doesn't happen automatically, and anyway that's not possible when standard input is coming from a terminal, from a pipe, etc.)

Delete words from given files with sed

I have this assignment to solve:
"Write a shell script that continuously reads words from the keyboard and
deletes them from all the files given in the command line."
I've tried to solve it, here's my attempt:
#!/bin/bash
echo "Enter words"
while (true)
do
read wrd
if [ "$wrd" != "exit" ]
then
for i in $#
do
sed -i -e 's/$wrd//g' $i
done
else
break
fi
done
This is the error that I receive after introducing the command: ./h84a.sh fisier1.txt
Enter words
suc
sed: can't read 1: No such file or directory
Sorry if I'm not very specific, it's my first time posting in here. I'm working in a terminal on Linux Mint which is installed on another partition of my PC. Please help me with my problem. Thanks!
I think you can simplify your script quite a lot:
#!/bin/bash
echo "Enter words"
while read -r wrd
do
[ "$wrd" = exit ] && break
sed -i "s/$wrd//g" "$#"
done
Some key changes:
The double quotes around the sed command are essential, as shell variables are not expanded within single quotes
Instead of using a loop, it is possible to pass all of the file names to sed at once, using "$#"
read -r is almost always what you want to use
I would suggest that you take care with in-place editing using the -i switch. In some versions of sed, you can specify the suffix of a backup file like -i.bak, so the original file is not lost.
In case you're not familiar with the syntax [ "$wrd" = exit ] && break, it is functionally equivalent to:
if [ "$wrd" = exit ]
then break
fi
$# expands to the number of arguments (so 1 in this case)
You probably meant to use $* or "$#"

Directing awk output to variable

New guy here with a problem that will hopefully have an easy solution, but I just can't seem to manage.
So, I have a large list of files that I need to process using the same command line program, and I'm trying to write a small shell script to automate this. I wrote something that will read the input file name from a text file, and repeat the command for each of those files. So far so good. My problem though is with naming the output. Each file is named in the general format "lane_number_bla_bla_bla", and they are processed in pairs. So, there will be a "lane_1_bla_bla_bla_001" and "lane_1_bla_bla_bla_002" that need to combine into a single output file. For this, I'm trying to use awk to read the sample number from the .txt list of input files and parse it into the output file number. Here's the code I came up with (note that the echo statement before the command is there just for testing; it's removed when it comes to run the actual program; also this is not the actual command which is rather more complicated, but the principle still applies):
echo "Which input1 should I use?"
read text
input1=$text
echo "Which input2 should I use?"
read text
input2=$text
echo "How many lines?"
read text
n=$text
for i in $(seq 1 $n)
do
awkinput1=$(awk NR==$i $input1)
awkinput2=$(awk NR==$i $input2)
num=$(awk 'NR==$i{print $2 }' FS="_" $input1)
lane=$(awk 'NR==$i{print $1 }' FS="_" $input1)
echo "command $awkinput1.in > $awkinput1.out && command $awkinput2.in > $awkinput2.out && command cat $awkinput1.out $awkinput2.in > $num-$lane-CAT.out &"
if (( $i % 10 == 0 )); then wait; fi # Limit to 10 concurrent subshells.
done
When I run this, both $awkinput fields get replaced properly in the comand line by the appropriate filename, but not the $num and $lane fields, which print nothing.
So, what am I doing wrong? I'm sure it's pretty simple, but I tried quite a lot of different ways to format the relevant awk command, and nothing seems to work. I'm doing this on a remote linux server using SSH protocol, if it makes a difference.
Thanks a lot!
Shell does not parse $i quoted by single quote ('). So quoted string should be terminated before $i.
FS should be set before parsing lines.
Following code will work.
num=$(awk 'BEGIN{FS="_"} NR=='$i'{print $2 }' $input1)
lane=$(awk 'BEGIN{FS="_"} NR=='$i'{print $1 }' $input1)
Code below will be more efficient:
while read in1 ; do
read in2 <&3
num=$(awk 'BEGIN{FS="_"} {print $2 }' <<<"$in1")
lane=$(awk 'BEGIN{FS="_"} {print $1 }' <<<"$in1")
...
done <$input1 3<$input2

How to read from user within while-loop read line?

I had a bash file which prompted the user for some parameters and used defaults if nothing was given. The script then went on to perform some other commands with the parameters.
This worked great - no problems until most recent addition.
In an attempt to read the NAMES parameter from a txt file, I've added a while-loop to take in the names in the file, but I would still like the remaining parameters prompted for.
But once I added the while loop, the output shows the printed prompt in get_ans() and never pauses for a read, thus all the defaults are selected.
I would like to read the first parameter from a file, then all subsequent files from prompting the user.
What did I break by adding the while-loop?
cat list.txt |
while read line
do
get_ans "Name" "$line"
read NAME < $tmp_file
get_ans "Name" "$line"
read NAME < $tmp_file
done
function get_ans
{
if [ -f $tmp_file ]; then
rm $tmp_file
PROMPT=$1
DEFAULT=$2
echo -n "$PROMPT [$DEFAULT]: "
read ans
if [ -z "$ans" ]; then
ans="$DEFAULT"
fi
echo "$ans" > $tmp_file
}
(NOTE: Code is not copy&paste so please excuse typos. Actual code has function defined before the main())
You pipe data into your the while loops STDIN. So the read in get_ans is also taking data from that STDIN stream.
You can pipe data into while on a different file descriptor to avoid the issue and stop bothering with temp files:
while read -u 9 line; do
NAME=$(get_ans Name "$line")
done 9< list.txt
get_ans() {
local PROMPT=$1 DEFAULT=$2 ans
read -p "$PROMPT [$DEFAULT]: " ans
echo "${ans:-$DEFAULT}"
}
To read directly from the terminal, not from stdin (assuming you're on a *NIX machine, not a Windows machine):
while read foo</some/file; do
read bar</dev/tty
echo "got <$bar>"
done
When you pipe one command into another on the command line, like:
$ foo | bar
The shell is going to set it up so that bar's standard input comes from foo's standard output. Anything that foo sends to stdout will go directly to bar's stdin.
In your case, this means that the only thing that your script can read from is the standard output of the cat command, which will contain the contents of your file.
Instead of using a pipe on the command line, make the filename be the first parameter of your script. Then open and read from the file inside your code and read from the user as normal.

Print output of cat statement in bash script loop

I'm trying to execute a command for each line coming from a cat command. I'm basing this on sample code I got from a vendor.
Here's the script:
for tbl in 'cat /tmp/tables'
do
echo $tbl
done
So I was expecting the output to be each line in the file. Instead I'm getting this:
cat
/tmp/tables
That's obviously not what I wanted.
I'm going to replace the echo with an actual command that interfaces with a database.
Any help in straightening this out would be greatly appreciated.
You are using the wrong type of quotes.
You need to use the back-quotes rather than the single quote to make the argument being a program running and piping out the content to the forloop.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done
Also for better readability (if you are using bash), you can write it as
for tbl in $(cat /tmp/tables)
do
echo "$tbl"
done
If your expectations are to get each line (The for-loops above will give you each word), then you may be better off using xargs, like this
cat /tmp/tables | xargs -L1 echo
or as a loop
cat /tmp/tables | while read line; do echo "$line"; done
The single quotes should be backticks:
for tbl in `cat /etc/tables`
Although, this will not get you output/input by line, but by word. To process line by line, you should try something like:
cat /etc/tables | while read line
echo $line
done
With while loop:
while read line
do
echo "$line"
done < "file"
while IFS= read -r tbl; do echo "$tbl" ; done < /etc/tables
read this.
You can do a lot of parsing in bash by redefining the IFS (Input Field Seperator), for example
IFS="\t\n" # You must use double quotes for escape sequences.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done

Resources