I have a file named mail_status.txt The content of the file is as follows.
1~auth_flag~
2~download_flag~
3~copy_flag~
4~auth_flag~
5~auth_flag~
6~copy_flag~
I want to perform some operation on this file so that at the end I should be getting three variables and their respective values should be as follows:
auth_flag_ids="1,4,5"
download_flag_ids="2"
copy_flag_ids="3,6"
I am quite new to this language. Please let me know if some more details are required on this.
Thanks
If you want to generate bash variables based on the file content,
please try the following:
# read the file and extract information line by line
declare -A hash # delcare hash as an associative array
while IFS= read -r line; do
key="${line#*~}" # convert "1~auth_flag~" to "auth_flag~"
key="${key%~*}_ids" # convert "auth_flag~" to "auth_flag_ids"
hash[$key]+="${line%%~*}," # append the value to the hash
done < "mail_status.txt"
# iterate over the hash to create variables
for r in "${!hash[#]}"; do # r is assigned to "auth_flag_ids", "download_flag_ids" and "copy_flag_ids" in tern
printf -v "$r" "%s" "${hash[$r]%,}" # create a variable named "$r" and assign it to the hash value by trimming the trailing comma off
done
# check the result
printf "%s=\"%s\"\n" "auth_flag_ids" "$auth_flag_ids"
printf "%s=\"%s\"\n" "download_flag_ids" "$download_flag_ids"
printf "%s=\"%s\"\n" "copy_flag_ids" "$copy_flag_ids"
First it reads the lines of the file and extracts the variable name
and the value line by line. They are stored in an associative array hash.
Next it iterates over the keys of hash to create variables whose names are
"auth_flag_ids", "download_flag_ids" and "copy_flag_ids".
printf -v var creates a variable var. This mechanism is useful to cause an
indirect reference to a variable.
I'm not going to explain in detail about the bash specific notations
such as ${parameter#word}, ${parameter%%word} or ${!name[#]}.
You can easily find the references and well-explained documents including
the bash man page.
Hope this helps.
Related
I have a small little script that deals with pipelines on Jenkins. It needs to be able to grab a file from a folder named after the pipeline name.
Most pipeline names follow this pattern: {Name}Pipeline/{Name}Pipeline.properties
However, a few pipelines have a three-digit version number appended, like so: {Name}Pipeline122/{Name}Pipeline122.properties
In my script, I have a line that stores the path to this properties file in a variable: APP_PROPERTY_FILE=/path/to/file/${NAME}Pipeline/${NAME}Pipeline.properties
Herein lies the problem! How can I allow bash to match pipeline names without the version number AND pipeline names with the version number?
Thanks!
I believe that the user want to select a file that has optional element (3 digit number), and store the file name into shell variable.
Two challenges: (1) regular assignment var=/path/to/something* do not perform pathname expansion and (2) regular pattern matching do not support optional elements.
Possible solutions are 'if-then-else' or using extended globs. Both solutions assumed that one of the files exists.
APP_PROPERTY_FILE=/path/to/file/${NAME}Pipeline/${NAME}Pipeline.properties
if [ ! -f "$APP_PROPERY_FILE" ] ; then
APP_PROPERTY_FILE=$(echo /path/to/file/${NAME}Pipeline/${NAME}Pipeline[0-9][-0-9][0-9].properties)
Using extglob can also work.
APP_PROPERTY_FILE=$(shopt -s extglob ; echo /path/to/file/${NAME}Pipeline/${NAME}Pipeline?([0-9][-0-9][0-9]).properties ; echo $1)
I have a quick question, and I am sure most of you have an answer to this.
I have a delimited file with the following data:
server1;user1;role
server1;user2;role,role 2
server2;user1;role,role 2,role 3
Please note that the role 'column' is comma-delimited and possibly with multi-valued information and names using spaces, different from the rest of the file that is semicolon-delimited and single-valued.
I need to show each 'role' into a different line, but related to the server and user information. For example:
server1;user1;role
server1;user2;role
server1;user2;role 2
server2;user1;role
server2;user1;role 2
server2;user1;role 3
Instead of having all roles on one server/user line, I require to have one role per line.
Do you have any suggestion to create this on a Bash script? I tried nested while read combos, and also for loops reading arrays, but so far I was unable to accomplished that (I know that probably I will have to use those functions, but in different manner).
This is the Bash script I have been working on:
#!/bin/bash
input="/file/input.csv"
output="/file/output.csv"
declare -a ARRAYROLES
while IFS=';' read -r f1 f2 f3
do
ARRAYROLES=($f3)
field1=$f1
field2=$f2
for element in "${ARRAYROLES[#]}"
do
echo "$field1;$field2;$element" >> "$output"
done
field1=''
field2=''
done < "$input"
And this is the output that I have so far (pretty close but not good enough):
server1;user1;role
server1;user2;role,role
server1;user2;2
server2;user1;role,role
server2;user1;2,role
server2;user1;3
Note that the role 'column' is divided per spaces (I am sure that is because of the for element statement reading the array)
Any idea will be greatly appreciated.
Regards,
Andres.
Change
ARRAYROLES=($f3)
to
IFS=, read -ra ARRAYROLES <<< "$f3"
while IFS=';' read -r server user roles; do
IFS=',' read -r -a arr <<< "$roles"
printf '%s\n' "${arr[#]/#/$server;$user;}"
done < "$input" > "$output"
From help read:
-a array
assign the words read to sequential indices of the array variable ARRAY,
starting at zero
${arr[#]/#/...} is a parameter expansion that in this case eliminated the need for an extra loop.
I'm new to bash and unfamiliar. I'm attempting to move through a file with data separated by a space and ","and store information into a list. The only container bash seems to have is an array. Any help with this is appreciated.
Say I have a file titled sample.txt with usernames passwords and birthrates and wanted to iterate through and store the users, passwords and birthdate in separate lists, what would be the easiest way to accomplish this
sample.txt
user1, password1, 081192
user2, password2, 092578
user3, password3, 020564
Bash version 4 has associative arrays, which is what I think you're looking for.
Be warned, you require a lot of noisy syntax (braces, brackets and quotes) to work with arrays in bash.
IFS+="," # add comma to the list of characters for word splitting
# you now cannot use a comma in your passwords.
declare -A passwords ids # 2 associative arrays
while read -r user password id; do
passwords["$user"]=$password
ids["$user"]=$id
done < sample.txt
# now that they are stored, let's print them out:
# iterate over the keys of the ids array
for user in "${!ids[#]}"; do
printf "%s:%s:%s\n" "$user" "${passwords["$user"]}" "${ids["$user"]}"
done
I'll provide some links to documentation in the bash manual: it is very dense reading, but is the source of wisdom for bash.
the read command
how the shell splits text into words, using the IFS variable: Word Splitting
and the order of shell expansions is here
bash arrays
You can use pure bashisms for this like so:
# read a csv line by line and fill an array
# called "myArray" with values from somefile.csv
while IFS=$',' read -r -a myArray; do
echo "${myArray[0]}"
echo "${myArray[1]}"
echo "${myArray[2]}"
done < somefile.csv
Example output:
foo
bar
baz
tum
di
dum
Example somefile.csv:
foo,bar,baz
tum,di,dum
I am writing a curl bash script to test webservices. I will have file_1 which would contain the URL paths
/path/to/url/1/{dynamic_path}.xml
/path/to/url/2/list.xml?{query_param}
Since the values in between {} is dynamic, I am creating a separate file, which will have values for these params. the input would be in key-value pair i.e.,
dynamic_path=123
query_param=shipment
By combining two files, the input should become
/path/to/url/1/123.xml
/path/to/url/2/list.xml?shipment
This is the background of my problem. Now my questions
I am doing it in bash script, and the approach I am using is first reading the file with parameters and parse it based on '=' and store it in key/value pair. so it will be easy to replace i.e., for each url I will find the substring between {} and whatever the text it comes with, I will use it as the key to fetch the value from the array
My approach sounds okay (at least to me) BUT, I just realized that
declare -A input_map is only supported in bashscript higher than 4.0. Now, I am not 100% sure what would be the target environment for my script, since it could run in multiple department.
Is there anything better you could suggest ? Any other approach ? Any other design ?
P.S:
This is the first time i am working on bash script.
Here's a risky way to do it: Assuming the values are in a file named "values"
. values
eval "$( sed 's/^/echo "/; s/{/${/; s/$/"/' file_1 )"
Basically, stick a dollar sign in front of the braces and transform each line into an echo statement.
More effort, with awk:
awk '
NR==FNR {split($0, a, /=/); v[a[1]]=a[2]; next}
(i=index($0, "{")) && (j=index($0,"}")) {
key=substr($0,i+1, j-i-1)
print substr($0, 1, i-1) v[key] substr($0, j+1)
}
' values file_1
There are many ways to do this. You seem to think of putting all inputs in a hashmap, and then iterate over that hashmap. In shell scripting it's more common and practical to process things as a stream using pipelines.
For example, your inputs could be in a csv file:
123,shipment
345,order
Then you could process this file like this:
while IFS=, read path param; do
sed -e "s/{dynamic_path}/$path/" -e "s/{query_param}/$param/" file_1
done < input.csv
The output will be:
/path/to/url/1/123.xml
/path/to/url/2/list.xml?shipment
/path/to/url/1/345.xml
/path/to/url/2/list.xml?order
But this is just an example, there can be so many other ways.
You should definitely start by writing a proof of concept and test it on your deployment server. This example should work in old versions of bash too.
So yes, normally you would increment an array using a for loop or a similar method, but in this scenario, that's not possible as I am already in a loop and need to only set one value in the array and move on. Here's the code:
n=1
cat databases/$IPS | grep -v \# | while read LINE; do
ENDPOINT[$n]="`echo $LINE | cut -d":" -f1`";
echo "TESTING: ${ENDPOINT[$n]}"
let n++
done
So the while loop will read several lines from a flat file database and pull the first colon separated value out which needs to be stored in the array. Things are just getting muddy from working on this project too long. I might have to just rethink this part.
So the official question is, without using another loop, how can increment elements using a variable so I can setup the array in the current while loop.
----- UPDATE -----
I know what im trying to do seems a little weird and unorthodox. So the while loop reads a database line by line. During the first iteration of the while loop, the database gets read and the first "cell" on the first row of the database gets stored to in the array as array element 1 (or could be 0, doesen't matter). The while loops then goes back for its second pass, and pulls the first cell of the second row in database and it needs to store it to the array under element 2. The loop continues to do this until the end of the database.
So I know how to read the contents of an array using a loop, but I have never set up an array using a loop. I cannot get the element number in the array to increment so that when the while loop does its next pass, it doesn't just keep overwriting array[1]
Does that help?
----- UPDATE -----
So it looks like this kinda works. The whole point of storing this database value in this array is because not only is it being used inside this while loop (Omitted irrelevant code). but I need to use these values elsewhere in the script.
cat databases/$IPS | grep -v # | while read LINE; do
VAR="echo $LINE | cut -d"|" -f1";
ENDPOINT[$n]="$VAR"
echo "Heres the array variable: ${ENDPOINT[$n]}"
done
Problem still is though that it doesn't appear to be truly storing to the array. It's just overwriting the first element in the array. If I try to echo any of the lower array values somewhere in the script, the only one that comes up is the the value in array element 0 and its set to the last entry in the database.
I seems to be that you are executing a command, may be this is a separate process and the array gets lost when it completes
Do this instead:
n=1
endpoints=`grep -v \# databases/$IPS | cut -d":" -f1`
for i in $endpoints; do
ENDPOINT[$n]=$i;
echo "TESTING: ${ENDPOINT[$n]}"
let n++
done
Didn't understand your question at 100%. I assume you can't get access to array elements outside of while loop.
That is because while is executed within cat command context, that is started as separate process. To get access to your array elements, you must set their values within the same bash context.
For example, you can use sed to get first of colon separated values and taking care of comments:
file_name=data
# extract values and put it into bash array
ENDPOINT=(`sed 's/[ \t]*#.*//g;s/^\([^:]*\):.*/\1/g' $file_name`)
echo "size: ${#ENDPOINT[#]}"
echo "readed data: ${ENDPOINT[#]}"
Also this variant is much very faster than while read loop, that have calls to another commands.