Expanding a string with a variable reference later, after the variable is assigned - linux

I'm trying to combine two lists containing names (if available) and emails with a standard email text in bash (shell)
(I had to delete the irrelevant code as it contains some private info, so some of the code might look unusal.)
The first half of the code checks if there is a name list along with the email list.
The second half combines only the email address and text if no name is available, if the name list is available it also 'tries' to combine the name, email and text.
f1 = email list and f2 = name list.
As you can see in the first half of the code below, $f2 should show the names if the list is available but it does not show anything in the log file.
I been trying to sort this problem out for two days but nothing has worked. When names are available it always outputs as "Hello ..." when it should be "Hello John D..."
#FIRST HALF
if [ "$names" = "no" ]
then
text="Hello..."
elif [ "$names" = "yes" ]
then
text="Hello $f2..."
fi
#SECOND HALF
if [ "$names" = "no" ]
then
for i in $(cat $emaillist); do
echo "$text" >> /root/log
echo "$i" >> /root/log
done
elif [ "$names" = "yes" ]
then
paste $emaillist $namelist | while IFS="$(printf '\t')" read -r f1 f2
do
echo "$text" >> /root/log
echo "$f1" >> /root/log
done
fi

When you run text="Hello $f2", $f2 is looked up at the time of the assignment; an exact string is assigned to text, and only that exact string is used later, on echo "$text".
This is very desirable behavior: If shell variables' values could run arbitrary code, it would be impossible to write shell scripts that handled untrusted data safely... but it does mean that implementing your program requires some changes.
If you want to defer evaluation (looking up the value of $f2 at expansion time rather than assignment), don't use a shell variable at all: Use a function instead.
case $names in
yes) write_greeting() { echo "Hello $name..."; };;
*) write_greeting() { echo "Hello..."; };;
esac
while read -r name <&3 && read -r email <&4; do
write_greeting
echo "$email"
done 3<"$namelist" 4<"$emaillist" >>/root/log
Some enhancements in the code above:
You don't need paste to read from two streams in lockstep; you can simply open them on different file descriptors (above, FDs 3 and 4 are chosen; only 0, 1 and 2 are reserved, so larger numbers could have been selected as well) with a separate read command for each.
Opening your output sink only once for the entire loop (by putting the redirection after the done) is far more efficient than re-opening it every time you want to write a single line.
Expansions, such as "$namelist" and "$emaillist", are always quoted; this makes code more reliable if dealing with filenames with unusual characters (including spaces and glob expressions), or if IFS is at a non-default value.

Related

How do you compare the value of an array to a variable in bash script?

I'm practicing bash and honestly, it is pretty fun. However, I'm trying to write a program that compares an array's value to a variable and if they are the same then it should print the array's value with an asterisk to the left of it.
#!/bin/bash
color[0]=red
color[1]=blue
color[2]=black
color[3]=brown
color[4]=yellow
favorite="black"
for i in {0..4};do echo ${color[$i]};
if {"$favorite"=$color[i]}; then
echo"* $color[i]"
done
output should be *black
There's few incorrect statements in your code that prevent it from doing what you ask it to. The comparison in bash is done withing square brackets, leaving space around them. You correctly use the = for string comparison, but should enclose in " the string variable. Also, while you correctly address the element array in the echo statement, you don't do so inside the comparison, where it should read ${color[$i]} as well. Same error in the asterisk print. So, here a reworked code with the fixes, but read more below.
#!/bin/bash
color[0]=red
color[1]=blue
color[2]=black
color[3]=brown
color[4]=yellow
favorite=black
for i in {0..4};do
echo ${color[$i]};
if [ "$favorite" = "${color[$i]}" ]; then
echo "* ${color[$i]}"
fi
done
While that code works now, few things that probably I like and would suggest (open to more expert input of course by the SO community): always enclose strings in ", as it makes evident it is a string variable; when looping an array, no need to use index variables; enclose variables always within ${}.
So my version of the same code would be:
#!/bin/bash
color=("red" "blue" "black" "brown" "yellow")
favorite="black"
for item in ${color[#]}; do
echo ${item}
if [ "${item}" = "${favorite}" ]; then
echo "* $item"
fi
done
And a pointer to the great Advanced Bash-Scripting Guide here: http://tldp.org/LDP/abs/html/

How to extract key value pairs from a file when values span multiple lines?

I'm a few weeks into bash scripting and I haven't advanced enough yet to get my head wrapped around this problem. Any help would be appreciated!
I have a "script.conf" file that contains the following:
key1=value1
key2=${HOME}/Folder
key3=( "k3v1" "k3 v2" "k3v3")
key4=( "k4v1"
"k4 v2"
"k4v3"
)
key5=value5
#key6="Do Not Include Me"
In a bash script, I want to read the contents of this script.conf file into an array. I've learned how to handle the scenarios for keys 1, 2, 3, and 5, but the key4 scenario throws a wrench into it with it spanning across multiple lines.
I've been exploring the use of sed -n '/=\s*[(]/,/[)]/{/' which does capture key4 and its value, but I can't figure out how to mix this so that the other keys are also captured in the matches. The range syntax is also new to me, so I haven't figured out how to separate the key/value. I feel like there is an easy regex that would accomplish what I want... in plain-text: "find and group the pattern ^(.*)= (for the key), then group everything after the '=' char until another ^(.*)= match is found, rinse and repeat". I guess if I do this, I need to change the while read line to not handle the key/value separation for me (I'll be looking into this while I'm waiting for a response). BTW, I think a solution where the value of key4 is flattened (new lines removed) would be acceptable; I know for key3 I have to store the value as a string and then convert it to an array later when I want to iterate over it since an array element apparently can't contain a list.
Am I on the right path with sed or is this a job for awk or some other tool? (I haven't ventured into awk yet). Is there an easier approach that I'm missing because I'm too deep into the forest (like changing the while read line in the LoadConfigFile function)?
Here is the code that I have so far in script.sh for processing and capturing the other pairs into the $config array:
__AppDir=$(dirname $0)
__AppName=${__ScriptName%.*}
typeset -A config #init config array
config=( #Setting Default Config values
[key1]="defaultValue1"
[key2]="${HOME}/defaultFolder"
[QuietMode]=0
[Verbose]=0 #Ex. Usage: [[ "${config[Verbose]}" -gt 0 ]] && echo ">>>Debug print"
)
function LoadConfigFile() {
local cfgFile="${1}"
shopt -s extglob #Needed to remove trailing spaces
if [ -f ${cfgFile} ]; then
while IFS='=' read -r key value; do
if [[ "${key:0:1}" == "#" ]]; then
#echo "Skipping Comment line: ${key}"
elif [ "${key:-EMPTY}" != "EMPTY" ]; then
value="${value%%\#*}" # Delete in-line, right comments
value="${value%%*( )}" # Delete trailing spaces
value="${value%%( )*}" # Delete leading spaces
#value="${value%\"*}" # Delete opening string quotes
#value="${value#\"*}" # Delete closing string quotes
#Manipulate any variables included in the value so that they can be expanded correctly
# - value must be stored in the format: "${var1}". `backticks`, "$var2", and "doubleQuotes" are left as is
value="${value//\"/\\\"}" # Escape double quotes for eval
value="${value//\`/\\\`}" # Escape backticks for eval
value="${value//\$/\\\$}" # Escape ALL '$' for eval
value="${value//\\\${/\${}" # Undo the protection of '$' if it was followed by a '{'
value=$(eval "printf '%s\n' \"${value}\"")
config[${key}]=${value} #Store the value into the config array at the specified key
echo " >>>DBG: Key = ${key}, Value = ${value}"
#else
# echo "Skipped Empty Key"
fi
done < "${cfgFile}"
fi
}
CONFIG_FILE=${__AppDir}/${__AppName}.conf
echo "Config File # ${CONFIG_FILE}"
LoadConfigFile ${CONFIG_FILE}
#Print elements of $config
echo "Script Config Values:"
echo "----------------------------"
for key in "${!config[#]}"; do #The '!' char gets an array of the keys, without it, we would get an array of the values
printf " %-20s = %s\n" "${key}" "${config[${key}]}"
done
echo "------ End Script Config ------"
#To convert to an array...
declare -a valAsArray=${config[RequiredAppPackages]} #Convert the value from a string to an array
echo "Count = ${#valAsArray[#]}"
for itemCfg in "${valAsArray[#]}"; do
echo " item = ${itemCfg}"
done
As I mentioned before, I'm just starting to learn bash and Linux scripting in general, so if you see that I'm doing some taboo things in other areas of my code too, please feel free to provide feedback in the comments... I don't want to start bad habits early on :-).
*If it matters, the OS is Ubuntu 14.04.
EDIT:
As requested, after reading the script.conf file, I would like for the elements in $config[#] to be equivalent to the following:
typeset -A config #init config array
config=(
[key1]="value1"
[key2]="${HOME}/Folder"
[key3]="( \"k3v1\" \"k3 v2\" \"k3v3\" )"
[key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )"
[key5]="value5"
)
I want to be able to convert the values of elements 'key4' and 'key3' into an array and iterated over them the same way in the following code:
declare -a keyValAsArray=${config[keyN]} #Convert the value from a string to an array
echo "Count = ${#keyValAsArray[#]}"
for item in "${keyValAsArray[#]}"; do
echo " item = ${item}"
done
I don't think it matters if \n is preserved for key4's value or not... that depends on if declare has a problem with it.
A shell is an environment from which to call tools with a language to sequence those calls. It is NOT a tool to manipulate text. The standard UNIX tool to manipulate text is awk. Trying to manipulate text in shell IS a bad habit, see why-is-using-a-shell-loop-to-process-text-considered-bad-pr‌​actice for SOME of the reasons why
You still didn't post the expected result of populating the config array so I'm not sure but I think this is what you wanted:
$ cat tst.sh
declare -A config="( $(awk '
{ gsub(/^[[:space:]]+|([[:space:]]+|#.*)$/,"") }
!NF { next }
/^[^="]+=/ {
name = gensub(/=.*/,"",1)
value = gensub(/^[^=]+=/,"",1)
n2v[name] = value
next
}
{ n2v[name] = n2v[name] OFS $0 }
END {
for (name in n2v) {
value = gensub(/"/,"\\\\&","g",n2v[name])
printf "[%s]=\"%s\"\n", name, value
}
}
' script.conf
) )"
declare -p config
$ ./tst.sh
declare -A config='([key5]="value5" [key4]="( \"k4v1\" \"k4 v2\" \"k4v3\" )" [key3]="( \"k3v1\" \"k3 v2\" \"k3v3\")" [key2]="/home/Ed/Folder" [key1]="value1" )'
The above uses GNU awk for gensub(), with other awks you'd use [g]sub() instead.

bash script function scope

function generateFileList {
for entry in "$ORIGINATION_PATH"/*
do
entry=${entry%.*} # retain the part before the dot
entry=${entry##*/} # retain the part after the last slash
if [ $(contains "${FILENAME[#]}" $entry) == "n" ]; then
FILENAME[$fn_counter]=$entry
fn_counter=(expr $fn_counter + 1)
echo $entry "added to filelist"
echo ${FILENAME[$fn_counter]}
fi
done
NUMBER_OF_FILES=$(expr ${#FILENAME[#]} + 1)}
I have this function .My $ORIGINATION_PATH has many files in it. However, when I call this function my $FILENAME array gets populated only with one entry.Why? Inside the function everything seems fine, and it seems that $FILENAME array gets all the values it needs to get, but when I check outside the function I only get one value in the $FILENAME aray
Problems with your code and suggestions for improvement:
You should initialize ${FILENAME[#]} to an empty array (either in the function itself if you always want the function to generate a new list of files from scratch, or before calling the function if you want to be able to build up a composite list of files by calling the function repeatedly on different base directories).
You should initialize $fn_counter to zero before starting the loop. Or, for the composite build-up idea, to the number of elements currently in ${FILENAME[#]}. Actually, another, perhaps preferable solution, would be to remove the $fn_counter variable entirely and replace it with ${#FILENAME[#]}, since it should always be equal to that value.
In the line fn_counter=(expr $fn_counter + 1), you're assigning $fn_counter to an array, rather than incrementing it. This is because you forgot the dollar before the open parenthesis. If you ran fn_counter=$(expr $fn_counter + 1) then it would work. But there's a better way to increment a numeric variable: let ++fn_counter.
You don't have to dollar-prefix variables in arithmetic expressions. So, for example, we can say ${FILENAME[fn_counter]} instead of ${FILENAME[$fn_counter]}.
You're trying to echo the element of ${FILENAME[#]} that was just added in the current iteration, but indexing it with $fn_counter after it was incremented, which is incorrect. You can solve this by subtracting 1 from it, i.e. echo "${FILENAME[fn_counter-1]}". Or, if removing $fn_counter, echo "${FILENAME[${#FILENAME[#]}-1]}".
When assigning $NUMBER_OF_FILES, I don't know why you're adding 1 to ${#FILENAME[#]}. The number of elements in the ${FILENAME[#]} array should be equal to the number of files, without requiring an increment, no? I recommend removing this variable entirely, since the value can be accessed directly as ${#FILENAME[#]}.
I recommend you pass inputs as arguments (e.g. pass $ORIGINATION_PATH as an argument) and use the local keyword to reduce the likelihood of variable clashes between functions. Globals are the default in bash, which creates dangerous possibilities for different functions to step on each others' toes. For example, imagine if the contains function (assuming it's a shell function) assigned a value to the global $entry variable.
I recommend always using the [[ command rather than [, as it's more powerful, and it's good to be consistent.
As written, your script won't work correctly on an empty directory. You could test in advance if the directory is empty (e.g. [[ -n "$(find "$ORIGINATION_PATH" -maxdepth 0 -empty)" ]]). Another solution is to set nullglob. Another solution is to skip glob words that don't actually exist (e.g. if [[ ! -e "$entry" ]]; then continue; fi;).
Always double-quote variable expansions to protect against word splitting, which takes place after variable expansion. For example, the contains call should be contains "${FILENAME[#]}" "$entry" (notice the double-quoting around $entry). The only exceptions are (1) when assigning a string variable to a string variable, i.e. new=$old, in which case you don't have to quote it, and (2) when expanding a numeric variable, which is guaranteed not to be corrupted by word splitting.
Here's a working solution, filling in the missing pieces:
function contains {
local target="${#:$#:1}";
local -a array=("${#:1:$#-1}");
local elem='';
for elem in "${array[#]}"; do
if [[ "$elem" == "$target" ]]; then
echo 'y';
return;
fi;
done;
echo 'n';
} ## end contains()
function generateFileList {
local path="$1";
local entry='';
for entry in "$path"/*; do
if [[ ! -e "$entry" ]]; then continue; fi;
entry=${entry%.*}; ## retain the part before the dot
entry=${entry##*/}; ## retain the part after the last slash
if [[ "$(contains "${FILENAME[#]}" "$entry")" == 'n' ]]; then
FILENAME[${#FILENAME[#]}]=$entry;
echo "$entry added to filelist";
echo "${FILENAME[${#FILENAME[#]}-1]}";
fi;
done;
} ## end generateFileList()
ORIGINATION_PATH='...';
FILENAME=(); ## build up result on global ${FILENAME[#]} var
generateFileList "$ORIGINATION_PATH";
echo "\${#FILENAME[#]} == ${#FILENAME[#]}";
echo "\${FILENAME[#]} == (${FILENAME[#]})";

CSV Bash loop Issue with Variables

I have a csv file which im trying to loop through with the purpose to find out if an User Input is found inside the csv data. I wrote the following code which sometimes works and others doesn't. It always stops working when I try to compare to a 2+ digit number. It works OK for numbers 1 through 9, but once u enter lets say 56 , or 99 or 100, it stops working.
the csv data is comma delimited, i have about 300 lines they are just like this.
1,John Doe,Calculus I,5.0
1,John Doe,Calculus II,4.3
1,John Doe,Physics II,3.5
2,Mary Poppins,Calculus I,3.7
2,Mary Poppins,Calculus II,4.7
2,Mary Poppins,Physics I,3.7
Data is just like that, all the way down until ID #100 for a total of 300 lines. Both the sh file and csv file are in the same folder, I'm using a fresh installation of Ubuntu 12.04.3, using gedit as the text editor.
I tried Echoing the variables ID and inside the IF conditionals but it doesn't behave the way it should when testing for the same value. Could someone point me out in the right direction. Thanks
Here's the code:
#s!/bin/bash
echo "enter your user ID";
read user;
INPUT_FILE=notas.csv
while IFS="," read r- ID name asignature final;
do
if [$ID = $user]; then
userType=1;
else
userType=2;
fi
done < notas.csv
Well, your code as written has a few issues.
You have r- instead of -r on the read line - I assume that's a typo not present in your actual code or you wouldn't get very far.
Similarly, you need space around the [...] brackets: [$ID is a syntax error.
You need to quote the parameter expansions in your if clause, and/or switch bracket types. You probably make it a numeric comparison as #imp25 suggested, which I would do by using ((...)).
You probably don't want to set userType to 2 in an else clause, because that will set it to 2 for everyone except whoever is listed last in the file (ID 100, presumably). You want to set it to 2 first, outside the loop. Then, inside the loop when you find a match, set it to 1 and break out of the loop:
userType=2
while IFS=, read -r ID name asignature final; do
if (( $ID == $user )); then
userType=1;
break
fi
done < notas.csv
You could also just use shell tools like awk:
userType=$(awk -F, -vtype=2 '($1=="'"$user"'") {type=1}; END {print type}' notas.csv)
or grep:
grep -q "^$user," notas.csv
userType=$(( $? + 1 ))
etc.
You should quote your variables in the if test statement. You should also perform a numeric test -eq rather than a string comparison =. So your if statement should look like:
if [[ "$ID" -eq "$user" ]]

Shell Script that performs different functions based on input from file

I am trying to merge two very different scripts together for consolidation and ease of use purposes. I have an idea of how I want these scripts to look and operate, but I could use some help getting started. Here is the flow and look of the script:
The input file would be a standard text file with this syntax:
#Vegetables
Broccoli|Green|14
Carrot|Orange|9
Tomato|Red|7
#Fruits
Apple|Red|15
Banana|Yellow|5
Grape|Purple|10
The script would take the input of this file. It would ignore the commented portions, but use them to dictate the output. So based on the fact that it is a Vegetable, it would perform a specific function with the values listed between the delimiter (|). Then it would go to the Fruits and do something different with the values, based on that delimiter. Perhaps, I would add Vegetable/Fruit to one of the values and dependent on that value it would perform the function while in this loop to read the file. Thank you for your help in getting this started.
UPDATE:
So I am trying to implement the IFS setup and thought of a more logical arrangement. The input file will have the "categories" displayed within the parameters. So the setup will be like this:
Vegetable|Carrot|Yellow
Fruit|Apple|Red
Vegetable|Tomato|Red
From there, the script will read in the lines and perform the function. So basically this type of setup in shell:
while read -r category item color
do
if [[ $category == "Vegetable" ]] ; then
echo "The $item is $color"
elif [[ $category == "Fruit" ]] ; then
echo "The $item is $color"
else
echo "Bad input"
done < "$input_file"
Something along those lines...I am just having trouble putting it all together.
Use read to input the lines. Do a case statement on their prefix:
{
while read DATA; do
case "$DATA" in
\#*) ... switch function ...;;
*) eval "$FUNCTION";;
esac
done
} <inputfile
Dependent on your problem you might want to experiment with setting $IFS before reading and read multiple variables in 1 go.
You can redefine the processing function each time you meet a # directive:
#! /bin/bash
while read line ; do
if [[ $line == '#Vegetables' ]] ; then
process () {
echo Vegetables: "$#"
}
elif [[ $line == '#Fruits' ]] ; then
process () {
echo Fruits: "$#"
}
else
process $line
fi
done < "$1"
Note that the script does not skip empty lines.

Resources