I have a Bash script where I want to count how many things were done when looping through a file. The count seems to work within the loop but after it the variable seems reset.
nKeys=0
cat afile | while read -r line
do
#...do stuff
let nKeys=nKeys+1
# this will print 1,2,..., etc as expected
echo Done entry $nKeys
done
# PROBLEM: this always prints "... 0 keys"
echo Finished writing $destFile, $nKeys keys
The output of the above is something alone the lines of:
Done entry 1
Done entry 2
Finished writing /blah, 0 keys
The output I want is:
Done entry 1
Done entry 2
Finished writing /blah, 2 keys
I am not quite sure why nKeys is 0 after the loop :( I assume it's something basic but damned if I can spot it despite looking at http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-7.html and other resources.
Fingers crossed someone else can look at it and go "well duh! You have to ..."!
In the just-released Bash 4.2, you can do this to prevent creating a subshell:
shopt -s lastpipe
Also, as you'll probably see at the link Ignacio provided, you have a Useless Use of cat.
while read -r line
do
...
done < afile
As mentioned in the accepted answer, this happens because pipes spawn separate subprocesses. To avoid this, command grouping has been the best option for me. That is, doing everything after the pipe in a subshell.
nKeys=0
cat afile |
{
while read -r line
do
#...do stuff
let nKeys=nKeys+1
# this will print 1,2,..., etc as expected
echo Done entry $nKeys
done
# PROBLEM: this always prints "... 0 keys"
echo Finished writing $destFile, $nKeys keys
}
Now it will report the value of $nKeys "correctly" (i.e. what you wish).
I arrived at the desired result in the following way without using pipes or here documents
#!/bin/sh
counter=0
string="apple orange mango egg indian"
str_len=${#string}
while [ $str_len -ne 0 ]
do
c=${string:0:1}
if [[ "$c" = [aeiou] ]]
then
echo -n "vowel : "
echo "- $c"
counter=$(( $counter + 1 ))
fi
string=${string:1}
str_len=${#string}
done
printf "The number of vowels in the given string are : %s "$counter
echo
Related
#!/bin/bash
#if there are no args supplied exit with 1
if [ "$#" -eq 0 ]; then
echo "Unfortunately you have not passed any parameter"
exit 1
fi
#loop over each argument
for arg in "$#"
do
if [ -f arg ]; then
echo "$arg is a file."
#iterates over the files stated in arguments and reads them $
cat $arg | while read line;
do
#should access only first line of the file
if [ head -n 1 "$arg" ]; then
process line
echo "Script has ran successfully!"
exit 0
#should access only last line of the file
elif [ tail -n 1 "$arg" ]; then
process line
echo "Script has ran successfully!"
exit 0
#if it accesses any other line of the file
else
echo "We only process the first and the last line of the file."
fi
done
else
exit 2
fi
done
#function to process the passed string and decode it in base64
process() {
string_to_decode = "$1"
echo "$string_to_decode = " | base64 --decode
}
Basically what I want this script to do is to loop over the arguments passed to the script and then if it's a file then call the function that decodes in base64 but just on the first and the last line of the chosen file. Unfortunately when I run it even with calling a right file it does nothing. I think it might be encountering problems with the if [ head -n 1 "$arg" ]; then part of the code. Any ideas?
EDIT: So I understood that I am actually just extracting first line over and over again without really comparing it to anything. So I tried changing the if conditional of the code to this:
first_line = $(head -n 1 "$arg")
last_line = $(tail -n 1 "$arg")
if [ first_line == line ]; then
process line
echo "Script has ran successfully!"
exit 0
#should access only last line of the file
elif [ last_line == line ]; then
process line
echo "Script has ran successfully!"
exit 0
My goal is to iterate through files for example one is looking like this:
MTAxLmdvdi51awo=
MTBkb3duaW5nc3RyZWV0Lmdvdi51awo=
MXZhbGUuZ292LnVrCg==
And to decode the first and the last line of each file.
To decode the first and last line of each file given to your script, use this:
#! /bin/bash
for file in "$#"; do
[ -f "$file" ] || exit 2
head -n1 "$file" | base64 --decode
tail -n2 "$file" | base64 --decode
done
Yea, as the others already said the true goal of the script isn't really clear. That said, i imagine every variation of what you may have wanted to do would be covered by something like:
#!/bin/bash
process() {
encoded="$1";
decoded="$( echo "${encoded}" | base64 --decode )";
echo " Value ${encoded} was decoded into ${decoded}";
}
(( $# )) || {
echo "Unfortunately you have not passed any parameter";
exit 1;
};
while (( $# )) ; do
arg="$1"; shift;
if [[ -f "${arg}" ]] ; then
echo "${arg} is a file.";
else
exit 2;
fi;
content_of_first_line="$( head -n 1 "${arg}" )";
echo "Content of first line: ${content_of_first_line}";
process "${content_of_first_line}";
content_of_last_line="$( tail -n 1 "${arg}" )";
echo "Content of last line: ${content_of_last_line}";
process "${content_of_last_line}";
line=""; linenumber=0;
while IFS="" read -r line; do
(( linenumber++ ));
echo "Iterating over all lines. Line ${linenumber}: ${line}";
process "${line}";
done < "${arg}";
done;
some additions you may find useful:
If the script is invoked with multiple filenames, lets say 4 different filenames, and the second file does not exist (but the others do),
do you really want the script to: process the first file, then notice that the second file doesnt exist, and exit at that point ? without processing the (potentially valid) third and fourth file ?
replacing the line:
exit 2;
with
continue;
would make it skip any invalid filenames, and still process valid ones that come after.
Also, within your process function, directly after the line:
decoded="$( echo "${encoded}" | base64 --decode )";
you could check if the decoding was successful before echoing whatever the resulting garbage may be if the line wasnt valid base64.
if [[ "$?" -eq 0 ]] ; then
echo " Value ${encoded} was decoded into ${decoded}";
else
echo " Garbage.";
fi;
--
To answer your followup question about the IFS/read-construct, it is a mixture of a few components:
read -r line
reads a single line from the input (-r tells it not to do any funky backslash escaping magic).
while ... ; do ... done ;
This while loop surrounds the read statement, so that we keep repeating the process of reading one line, until we run out.
< "${arg}";
This feeds the content of filename $arg into the entire block of code as input (so this becomes the source that the read statement reads from)
IFS=""
This tells the read statement to use an empty value instead of the real build-in IFS value (the internal field separator). Its generally a good idea to do this for every read statement, unless you have a usecase that requires splitting the line into multiple fields.
If instead of
IFS="" read -r line
you were to use
IFS=":" read -r username _ uid gid _ homedir shell
and read from /etc/passwd which has lines such as:
root:x:0:0:root:/root:/bin/bash
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
then that IFS value would allow it to load those values into the right variables (in other words, it would split on ":")
The default value for IFS is inherited from your shell, and it usually contains the space and the TAB character and maybe some other stuff. When you only read into one single variable ($line, in your case). IFS isn't applied but when you ever change a read statement and add another variable, word splitting starts taking effect and the lack of a local IFS= value will make the exact same script behave very different in different situations. As such it tends to be a good habbit to control it at all times.
The same goes for quoting your variables like "$arg" or "${arg}" , instead of $arg . It doesn't matter when ARG="hello"; but once the value starts containing spaces suddenly all sorts of things can act different; suprises are never a good thing.
In a previous question I asked how to write to a program's stdin. This question builds off of that one. Now I am asking "how do I parse my called program's output".
The fact that I'm calling a program I can edit (Sum.sh) in this example is irrelevant to the question. I want this behavior in the event that I do not have control over the implementation of the program I am calling. I want to "replace" the physical human that a program expects and drive it via a script.
Given two scripts
Sum.sh:
#!/bin/bash
sum=0
inp=0
while true;
do
printf "Sum: %d\n" $sum
printf "Give me an integer: "
read inp
if [ "$inp" == 0 ]; then
exit
fi
sum=$((sum+inp))
done
And driver.sh:
#!/bin/bash
rm -f /tmp/sumpipe
mkfifo /tmp/sumpipe
./Sum.sh < /tmp/sumpipe &
echo 2 > /tmp/sumpipe
echo 5 > /tmp/sumpipe
echo 0 > /tmp/sumpipe
sleep 1 # this is required for some reason or the script hangs
And I run driver.sh I get the following:
$ ./driver.sh
Sum: 0
Give me an integer: Sum: 2
Give me an integer: Sum: 7
Give me an integer: $
The formatting of the terminal output is not important to this question.
What I want to do is parse said output in order to make decisions in driver.sh. In the following psuedo code of driver.sh I try to show what I want to do:
#!/bin/bash
rm -f /tmp/sumpipe
mkfifo /tmp/sumpipe
output=$(./Sum.sh < /tmp/sumpipe &)
while true; do
if [ "$output" != empty ]; then
(make some decision based on what output is and echo back to my process if necessary)
fi
done
Unfortunately, $output never contains anything, and I have tried many versions of abusing read and piping and none of them have worked for me in this application. How can I redirect my backgrounded program's stdout so I can parse it?
I attempted OrangesV's proposed solution (and with deference to William Pursell's suggestion about the out buffer being too empty) with the following:
#!/bin/bash
rm -f /tmp/sumpipe
mkfifo /tmp/sumpipe
while read output; do
echo "2347823" > /tmp/sumpipe
if [ -z $output ]; then
continue
else
echo "got it!"
echo $output
sleep 1
fi
done < <(./Sum.sh < /tmp/sumpipe &)
Unfortunately, "got it!" is never echoed, so it seems it didn't work.
In case it's relevant, my bash version:
$ bash --version
GNU bash, version 4.4.0(7)-rc2 (x86_64-unknown-linux-gnu)
There is now a partial solution. Diego Torres Milano's answer does work. However, my stdio is being buffered so that I won't get anything till stdio has at least 4096 bytes in it. I can force output with the following version of driver.sh:
#!/bin/bash
inpipe=/tmp/sumpipei
outpipe=/tmp/sumpipeo
rm -f $inpipe
rm -f $outpipe
mkfifo $inpipe
mkfifo $outpipe
./Sum.sh < $inpipe > $outpipe &
count=0
while true;
do
while read line
do
echo "line=$line"
done < $outpipe &
echo 2 > $inpipe
done
I tried all sorts of invocations of stdbuf (like: stdbuf -o0 ./Sum.sh <$inpipe > $outpipe &). stdbuf does not apparently work in this situation to force off buffering because, according to the GNU coreutils manual on stdbuf for command in stdbuf option… command:
command must start with the name of a program that does ... not adjust
the buffering of its standard streams (note the program tee is not in
this category).
Which is unfortunately exactly what I'm doing here.
I'm marking Diego's answer as correct, even though it doesn't work until the stdio buffer issue is fixed. I'll make my next question to address the stdio buffering problem. I can almost taste the prize here.
Use another pipe for the output
mkfifo /tmp/output
./Sum.sh < /tmp/sumpipe > /tmp/output &
and you can process it with
while read line
do
echo "line=$line"
done < /tmp/output &
You can do a while read within your driver.sh.
while read output; do
if [ "${output}" == "something" ]; then
#do something
fi
done < <(./Sum.sh < /tmp/sumpipe &)
EDIT
This loop will not start iterating until the script's stdin receives input. So, in another shell/process an echo # > /tmp/sumpipe.
I am trying to create following string
Beta-3.8.0
but shell script always omits the . period char no matter what I do.
echo "$readVersion"
if [ -z $readVersion ]
then
echo "readVersion is empty"
exit 1
fi;
IFS=.
set $readVersion
newVersion=$(echo "$2 + 1" | bc)
newBranch="Beta-$1.$newVersion.$3"
echo $newBranch
prints:
3.8.0
Beta-3 9 0
I have also tried
newBranch='Beta-'$1'.'$newVersion'.'$3
or
newBranch="Beta-{$1}.{$newVersion}.{$3}"
although this seems printing the right value echo "$1.$newVersion.$3" why not variable doesnt work ?
I need the variable to use later on in the script...
You can save and restore the IFS once you are done.
oldIFS=$IFS
IFS=.
set $readVersion
newVersion=$(echo "$2 + 1" | bc)
IFS=$oldIFS
newBranch="Beta-$1.$newVersion.$3"
echo "$newBranch"
Or you can quote when printing:
echo "$newBranch"
The former is a better idea IMO since it conveys your intention and would make the rest of the code use the "correct" IFS. The latter just circumvents the problem.
I'm writing my first Bash script, I have some experience with C and C# so I think the logic of the program is correct, it's just the syntax is so complicated because apparently there are many different ways to write the same thing!
Here is the script, it simply checks if the argument (string) is contained in a certain file. If so it stores each line of the file in an array and writes an item of the array in a file. I'm sure there must be easier ways to achieve that but I want to do some practice with bash loops
#!/bin/bash
NOME=$1
c=0
#IF NAME IS FOUND IN THE PHONEBOOK THEN STORE EACH LINE OF THE FILE INTO ARRAY
#ONCE THE ARRAY IS DONE GET THE INDEX OF MATCHING NAME AND RETURN ARRAY[INDEX+1]
if grep "$NOME" /root/phonebook.txt ; then
echo "CREATING ARRAY"
while read line
do
myArray[$c]=$line # store line
c=$(expr $c + 1) # increase counter by 1
done < /root/phonebook.txt
else
echo "Name not found"
fi
c=0
for i in myArray;
do
if myArray[$i]="$NOME" ; then
echo ${myArray[i+1]} >> /root/numbertocall.txt
fi
done
This code returns the only the second item of myArray (myArray[2]) or the second line of the file, why?
The first part (where you build the array) looks ok, but the second part has a couple of serious errors:
for i in myArray; -- this executes the loop once, with $i set to "myArray". In this case, you want $i to iterate over the indexes of myArray, so you need to use
for i in "${!myArray[#]}"
or
for ((i=0; i<${#a[#]}; i++))
(although I generally prefer the first, since it'll work with noncontiguous and associative arrays).
Also, you don't need the ; unless do is on the same line (in shell, ; is mostly equivalent to a line break so having a semicolon at the end of a line is redundant).
if myArray[$i]="$NOME" ; then -- the if statement takes a command, and will therefore treat myArray[$i]="$NOME" as an assignment command, which is not at all what you wanted. In order to compare strings, you could use the test command or its synonym [
if [ "${myArray[i]}" = "$NOME" ]; then
or a bash conditional expression
if [[ "${myArray[i]}" = "$NOME" ]]; then
The two are very similar, but the conditional expression has much cleaner syntax (e.g. in a test command, > redirects output, while \> is a string comparison; in [[ ]] a plain > is a comparison).
In either case, you need to use an appropriate $ expression for myArray, or it'll be interpreted as a literal. On the other hand, you don't need a $ before the i in "${myArray[i]}" because it's in a numeric expression context and therefore will be expanded automatically.
Finally, note that the spaces between elements are absolutely required -- in shell, spaces are very important delimiters, not just there for readability like they usually are in c.
1.-This is what you wrote with small adjustments
#!/bin/bash
NOME=$1
#IF NAME IS FOUND IN THE PHONE-BOOK **THEN** READ THE PHONE BOOK LINES INTO AN ARRAY VARIABLE
#ONCE THE ARRAY IS COMPLETED, GET THE INDEX OF MATCHING LINE AND RETURN ARRAY[INDEX+1]
c=0
if grep "$NOME" /root/phonebook.txt ; then
echo "CREATING ARRAY...."
IFS= while read -r line #IFS= in case you want to preserve leading and trailing spaces
do
myArray[c]=$line # put line in the array
c=$((c+1)) # increase counter by 1
done < /root/phonebook.txt
for i in ${!myArray[#]}; do
if myArray[i]="$NOME" ; then
echo ${myArray[i+1]} >> /root/numbertocall.txt
fi
done
else
echo "Name not found"
fi
2.-But you can also read the array and stop looping like this:
#!/bin/bash
NOME=$1
c=0
if grep "$NOME" /root/phonebook.txt ; then
echo "CREATING ARRAY...."
readarray myArray < /root/phonebook.txt
for i in ${!myArray[#]}; do
if myArray[i]="$NOME" ; then
echo ${myArray[i+1]} >> /root/numbertocall.txt
break # stop looping
fi
done
else
echo "Name not found"
fi
exit 0
3.- The following improves things. Supposing a)$NAME matches the whole line that contains it and b)there's always one line after a $NOME found, this will work; if not (if $NOME can be the last line in the phone-book), then you need to do small adjustments.
!/bin/bash
PHONEBOOK="/root/phonebook.txt"
NUMBERTOCALL="/root/numbertocall.txt"
NOME="$1"
myline=""
myline=$(grep -A1 "$NOME" "$PHONEBOOK" | sed '1d')
if [ -z "$myline" ]; then
echo "Name not found :-("
else
echo -n "$NOME FOUND.... "
echo "$myline" >> "$NUMBERTOCALL"
echo " .... AND SAVED! :-)"
fi
exit 0
I want to run a command that gives the following output and parse it:
[VDB VIEW]
[VDB] vhctest
[BACKEND] domain.computername: ENABLED:RW:CONSISTENT
[BACKEND] domain.computername: ENABLED:RW:CONSISTENT
...
I'm only interested in some key works, such as 'ENABLED' etc. I can't search just for ENABLED as I need to parse each line at a time.
This is my first script, and I want to know if anyone can help me?
EDIT:
I now have:
cmdout=`mycommand`
while read -r line
do
#check for key words in $line
done < $cmdout
I thought this did what I wanted but it always seems to output the following right before the command output.
./myscript.sh: 29: cannot open ... : No such file
I don't want to write to a file to have to achieve this.
Here is the psudo code:
cmdout=`mycommand`
loop each line in $cmdout
if line contains $1
if line contains $2
output 1
else
output 0
The reason for the error is that
done < $cmdout
thinks that the contents of $cmdout is a filename.
You can either do:
done <<< $cmdout
or
done <<EOF
$cmdout
EOF
or
done < <(mycommand) # without using the variable at all
or
done <<< $(mycommand)
or
done <<EOF
$(mycommand)
EOF
or
mycommand | while
...
done
However, the last one creates a subshell and any variables set in the loop will be lost when the loop exits.
"How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?"
"I set variables in a loop. Why do they suddenly disappear after the loop terminates? Or, why can't I pipe data to read?"
$ cat test.sh
#!/bin/bash
while read line ; do
if [ `echo $line|grep "$1" | wc -l` != 0 ]; then
if [ `echo $line|grep "$2" | wc -l` != 0 ]; then
echo "output 1"
else
echo "output 0"
fi
fi
done
USAGE
$ cat in.txt | ./test.sh ENABLED RW
output 1
output 1
This isn't the best solution, but its a word by word translation of what you want and should give you something to start with and add your own logic