I'm trying to execute a command for each line coming from a cat command. I'm basing this on sample code I got from a vendor.
Here's the script:
for tbl in 'cat /tmp/tables'
do
echo $tbl
done
So I was expecting the output to be each line in the file. Instead I'm getting this:
cat
/tmp/tables
That's obviously not what I wanted.
I'm going to replace the echo with an actual command that interfaces with a database.
Any help in straightening this out would be greatly appreciated.
You are using the wrong type of quotes.
You need to use the back-quotes rather than the single quote to make the argument being a program running and piping out the content to the forloop.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done
Also for better readability (if you are using bash), you can write it as
for tbl in $(cat /tmp/tables)
do
echo "$tbl"
done
If your expectations are to get each line (The for-loops above will give you each word), then you may be better off using xargs, like this
cat /tmp/tables | xargs -L1 echo
or as a loop
cat /tmp/tables | while read line; do echo "$line"; done
The single quotes should be backticks:
for tbl in `cat /etc/tables`
Although, this will not get you output/input by line, but by word. To process line by line, you should try something like:
cat /etc/tables | while read line
echo $line
done
With while loop:
while read line
do
echo "$line"
done < "file"
while IFS= read -r tbl; do echo "$tbl" ; done < /etc/tables
read this.
You can do a lot of parsing in bash by redefining the IFS (Input Field Seperator), for example
IFS="\t\n" # You must use double quotes for escape sequences.
for tbl in `cat /tmp/tables`
do
echo "$tbl"
done
Related
I'm sorry for my poor English, first.
I want to read a file (tel.txt) that contains many tel numbers (a number per line) and use that line to grep command to search about the specific number in the source file (another file)!
I wrote this code :
dir="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A"
file="$dir/tel.txt"
datafile="$dir/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.txt"
while IFS= read -r line
do
current="$line"
echo `grep -F $current "$datafile" >> output.txt`
done < $file
the tel file sample :
44001547
44001478
55421487
but that code returns nothing!
when I declare 'current' variable with literals it works correctly!
what happened?!
Your grep command is redirected to write its output to a file, so you don't see it on the terminal.
Anyway, you should probably be using the much simpler and faster
grep -Ff "$file" "$datafile"
Add | tee -a output.txt if you want to save the output to a file and see it at the same time.
echo `command` is a buggy and inefficient way to write command. (echo "`command`" would merely be inefficient.) There is no reason to capture standard output into a string just so that you can echo that string to standard output.
Why don't you search for the line var directly? I've done some tests, this script works on my linux (CentOS 7.x) with bash shell:
#!/bin/bash
file="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A/tel.txt"
while IFS= read -r line
do
echo `grep "$line" /home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.tx >> output.txt`
done < $file
Give it a try... It shows nothing on the screen since you're redirecting the output to the file output.txt so the matching results are saved there.
You should use file descriptors when reading with while loop.instead use for loop to avoid false re-directions
dir="/home/mujan/Desktop/data/ADSL_CDR_Text_Parts_A"
file="$dir/tel.txt"
datafile="$dir/ADSL_CDR_Like_Tct4_From_960501_to_97501_Part0.txt"
for line in `cat $file`
do
current="$line"
echo `grep -F $current "$datafile" >> output.txt`
done
I am trying to parse each line of a file and look for a particular string. The script seems to be doing its intended job, however, in parallel it tries to execute the if command on line 6:
#!/bin/bash
for line in $(cat $1)
do
echo $line | grep -e "Oct/2015"
if($?==0); then
echo "current line is: $line"
fi
done
and I get the following (my script is readlines.sh)
./readlines.sh: line 6: 0==0: command not found
First: As Mr. Llama says, you need more spaces. Right now your script tries to look for a file named something like /usr/bin/0==0 to run. Instead:
[ "$?" -eq 0 ] # POSIX-compliant numeric comparison
[ "$?" = 0 ] # POSIX-compliant string comparison
(( $? == 0 )) # bash-extended numeric comparison
Second: Don't test $? at all in this case. In fact, you don't even have good cause to use grep; the following is both more efficient (because it uses only functionality built into bash and requires no invocation of external commands) and more readable:
if [[ $line = *"Oct/2015"* ]]; then
echo "Current line is: $line"
fi
If you really do need to use grep, write it like so:
if echo "$line" | grep -q "Oct/2015"; then
echo "Current line is: $line"
fi
That way if operates directly on the pipeline's exit status, rather than running a second command testing $? and operating on that command's exit status.
#Charles Duffy has a good answer which I have up-voted as correct (and it is), but here's a detailed, line by line breakdown of your script and the correct thing to do for each part of it.
for line in $(cat $1)
As I noted in my comment elsewhere this should be done as a while read construct instead of a for cat construct.
This construct will wordsplit each line making spaces in the file separate "lines" in the output.
All empty lines will be skipped.
In addition when you cat $1 the variable should be quoted. If it is not quoted spaces and other less-usual characters appearing in the file name will cause the cat to fail and the loop will not process the file.
The complete line would read:
while IFS= read -r line
An illustrative example of the tradeoffs can be found here. The linked test script follows. I tried to include an indication of why IFS= and -r are important.
#!/bin/bash
mkdir -p /tmp/testcase
pushd /tmp/testcase >/dev/null
printf '%s\n' '' two 'three three' '' ' five with leading spaces' 'c:\some\dos\path' '' > testfile
printf '\nwc -l testfile:\n'
wc -l testfile
printf '\n\nfor line in $(cat) ... \n\n'
let n=1
for line in $(cat testfile) ; do
echo line $n: "$line"
let n++
done
printf '\n\nfor line in "$(cat)" ... \n\n'
let n=1
for line in "$(cat testfile)" ; do
echo line $n: "$line"
let n++
done
let n=1
printf '\n\nwhile read ... \n\n'
while read line ; do
echo line $n: "$line"
let n++
done < testfile
printf '\n\nwhile IFS= read ... \n\n'
let n=1
while IFS= read line ; do
echo line $n: "$line"
let n++
done < testfile
printf '\n\nwhile IFS= read -r ... \n\n'
let n=1
while IFS= read -r line ; do
echo line $n: "$line"
let n++
done < testfile
rm -- testfile
popd >/dev/null
rmdir /tmp/testcase
Note that this is a bash-heavy example. Other shells do not tend to support -r for read, for example, nor is let portable. On to the next line of your script.
do
As a matter of style I prefer do on the same line as the for or while declaration, but there's no convention on this.
echo $line | grep -e "Oct/2015"
The variable $line should be quoted here. In general, meaning always unless you specifically know better, you should double-quote all expansion--and that means subshells as well as variables. This insulates you from most unexpected shell weirdness.
You decclared your shell as bash which means you will have there "Here string" operator <<< available to you. When available it can be used to avoid the pipe; each element of a pipeline executes in a subshell, which incurs extra overhead and can lead to unexpected behavior if you try to modify variables. This would be written as
grep -e "Oct/2015" <<<"$line"
Note that I have quoted the line expansion.
You have called grep with -e, which is not incorrect but is needless since your pattern does not begin with -. In addition you have full-quoted a string in shell but you don't attempt to expand a variable or use other shell interpolation inside of it. When you don't expect and don't want the contents of a quoted string to be treated as special by the shell you should single quote them. Furthermore, your use of grep is inefficient: because your pattern is a fixed string and not a regular expression you could have used fgrep or grep -F, which does string contains rather than regular expression matching (and is far faster because of this). So this could be
grep -F 'Oct/2015' <<<"$line"
Without altering the behavior.
if($?==0); then
This is the source of your original problem. In shell scripts commands are separated by whitespace; when you say if($?==0) the $? expands, probably to 0, and bash will try to execute a command called if(0==0) which is a legal command name. What you wanted to do was invoke the if command and give it some parameters, which requires more whitespace. I believe others have covered this sufficiently.
You should never need to test the value of $? in a shell script. The if command exists for branching behavior based on the return code of whatever command you pass to it, so you can inline your grep call and have if check its return code directly, thus:
if grep -F 'Oct/2015` <<<"$line" ; then
Note the generous whitespace around the ; delimiter. I do this because in shell whitespace is usually required and can only sometiems be omitted. Rather than try to remember when you can do which I recommend an extra one space padding around everything. It's never wrong and can make other mistakes easier to notice.
As others have noted this grep will print matched lines to stdout, which is probably not something you want. If you are using GNU grep, which is standard on Linux, you will have the -q switch available to you. This will suppress the output from grep
if grep -q -F 'Oct/2015' <<<"$line" ; then
If you are trying to be strictly standards compliant or are in any environment with a grep that doesn't know -q the standard way to achieve this effect is to redirect stdout to /dev/null/
if printf "$line" | grep -F 'Oct/2015' >/dev/null ; then
In this example I also removed the here string bashism just to show a portable version of this line.
echo "current line is: $line"
There is nothing wrong with this line of your script, except that although echo is standard implementations vary to such an extent that it's not possible to absolutely rely on its behavior. You can use printf anywhere you would use echo and you can be fairly confident of what it will print. Even printf has some caveats: Some uncommon escape sequences are not evenly supported. See mascheck for details.
printf 'current line is: %s\n' "$line"
Note the explicit newline at the end; printf doesn't add one automatically.
fi
No comment on this line.
done
In the case where you did as I recommended and replaced the for line with a while read construct this line would change to:
done < "$1"
This directs the contents of the file in the $1 variable to the stdin of the while loop, which in turn passes the data to read.
In the interests of clarity I recommend copying the value from $1 into another variable first. That way when you read this line the purpose is more clear.
I hope no one takes great offense at the stylistic choices made above, which I have attempted to note; there are many ways to do this (but not a great many correct) ways.
Be sure to always run interesting snippets through the excellent shellcheck and explain shell when you run into difficulties like this in the future.
And finally, here's everything put together:
#!/bin/bash
input_file="$1"
while IFS= read -r line ; do
if grep -q -F 'Oct/2015' <<<"$line" ; then
printf 'current line is %s\n' "$line"
fi
done < "$input_file"
If you like one-liners, you may use AND operator (&&), for example:
echo "$line" | grep -e "Oct/2015" && echo "current line is: $line"
or:
grep -qe "Oct/2015" <<<"$line" && echo "current line is: $line"
Spacing is important in shell scripting.
Also, double-parens is for numerical comparison, not single-parens.
if (( $? == 0 )); then
My code is given below. Echo works fine. But, the moment I redirect output of echo to touch, I get an error "no such file or directory". Why ? How do i fix it ?
If I copy paste the output of only echo, then the file is created, but not with touch.
while read line
do
#touch < echo -e "$correctFilePathAndName"
echo -e "$correctFilePathAndName"
done < $file.txt
If you have file names in each line of your input file file.txt then you don't need to do any loop. You can just do:
touch $(<file.txt)
to create all the files in one single touch command.
You need to provide the file name as argument and not via standard input. You can use command substitution via $(…) or `…`:
while read line
do
touch "$(echo -e "$correctFilePathAndName")"
done < $file.txt
Ehm, lose the echo part... and use the correct variable name.
while read line; do
touch "$line"
done < $file.txt
try :
echo -e "$correctFilePathAndName" | touch
EDIT : Sorry correct piping is :
echo -e "$correctFilePathAndName" | xargs touch
The '<' redirects via stdin whereas touch needs the filename as an argument. xargs transforms stdin in an argument for touch.
I want to read the first line of a command output with. but I get an empty message.
I used
ls | read line
echo $line #nothing displayed
Why the line variable is empty and how to fix that?
The following code works. but I want to read only the first line
ls | while read line; do
echo $line
done
If it's not possible with read, is it possible to do it with other functions like grep, awk, sed ?
The read does read the first line, but it is being executed in a subshell. But you can do:
ls | { read line;
echo $line;
# other commands using $line;
}
# After the braces, $line is whatever it was before the braces.
You can use
read line <<< "$(ls)"
printf "%s\n" "$line"
But as already mentioned, please, PLEASE, PLEASE do not parse ls output!
I have a text file that contains references to variables and lets a user set up the formatting they want around variables, say something like
The date is $DATE
The time is $TIME
I then want to read this text file in, replace the variables, and print the result to stdout using a bash script. The closest thing I've gotten is using "echo" to output it,
DATE="1/1/2010"
TIME="12:00"
TMP=`cat file.txt`
echo $TMP
However, the output ends up all on one line, and I don't want to have \n at the end of every line in the text file. I tried using "cat << $TMP", but then there are no newlines and the variables inside the text aren't getting replaced with values.
You can use eval to ensure that variables are expanded in your data file:
DATE="1/1/2010"
TIME="12:00"
while read line
do
eval echo ${line}
done < file.txt
Note that this allows the user with control over the file content to execute arbitrary commands.
If the input file is outside your control, a much safer solution would be:
DATE="1/1/2010"
TIME="12:00"
sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" < file.txt
It assumes that neither $DATE nor $TIME contain the # character and that no other variables should be expanded.
Slightly more compact than Adam's response:
DATE="1/1/2010"
TIME="12:00"
TMP=`cat file.txt`
eval echo \""$TMP"\"
The downside to all of this is that you end up squashing quotes. Better is to use a real template. I'm not sure how to do that in shell, so I'd use another language if at all possible. The plus side to templating is that you can eliminate that "arbitrary command" hole that Adam mentions, without writing code quite as ugly as sed.
Just quote your variable to preserve newline characters.
DATE="1/1/2010"
TIME="12:00"
TMP=$(cat file.txt)
echo "$TMP"
For your modification of values in file with variables you can do -
while read -r line
do
sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" <<< "$line"
done < file.txt
Test:
[jaypal:~/Temp] cat file.txt
The date is $DATE
The time is $TIME
[jaypal:~/Temp] DATE="1/1/2010"
[jaypal:~/Temp] TIME="12:00"
[jaypal:~/Temp] while read -r line; do sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" <<< "$line"; done < file.txt
The date is 1/1/2010
The time is 12:00