How read line by line in bash and assign the value to a variable [duplicate] - linux

This question already has answers here:
Read a file line by line assigning the value to a variable [duplicate]
(10 answers)
Closed 2 years ago.
I have a file called ips.txt and it has values like below:
cat ips.txt
abc.com. 10.120.20.4 10.120.20.5 ... # there can be many ips separated by a space
xyz.com. 10.120.20.6
I want to read this file line by line with a loop and do some other works.
And als I need to remove that . as well at the end of each domain name abc.com. and assign each ip to a variable (if it can be stored to an array, it is great)
So here's what I tried:
input="ips.txt"
while IFS= read -r line
do
echo "$line"
domain="" # need to assign abc.com
ip_one= "" # need to assign the first ip occurence
ip_two= "" # need to assign the second ip occurence
...
ip_n= "" # need to assign the nth ip occurence
# some other commands I need to execute with domain name and all the ips collected
done < "$input"
how can I assign the ip values to different variables? and use them? It is better if I can Store the IP's in a array like data type so it is more easier as I don't know how many IP's are there for some line in the ips.txt file.
Can someone help me do this?

Using set, and shift:
while IFS= read -r line; do
set $line
domain=$(sed 's/\.$//' <<< "$1")
shift
for ip in $#; do
echo "domain: $domain ip: $ip"
done
done < "$input"

Start with a standard array read. Assign to $#, use parameter expansion to strip the dot, then shift it off the stack, and you can assign back to your original array.
input="ips.txt"
declare -A lookup=()
while read -a ips # ips is an array
do set -- "${ips[#]}" # assign the array to $#
domain="${1%.}" # assign abc.com without the dot
shift # dump the first column; now $# is just the IP's
for ip in "$#"; do lookup[$ip]="$domain"; done # assign the domain to each IP
done < "$input"
now
$: echo ${lookup[10.120.20.5]}
abc.com

Related

How do I fetch host and pass from file in linux

#!/bin/bash
hosts=(sarv savana simra punit)
pass=(sarva 1save xvyw23 asdwe87)
for i in "${!hosts[#]}"; do
sshpass -p "${pass[i]}" ssh-copy-id -f root#"${hostnames[i]}" -p 22
done
Is it possible to fetch the password and hostname from a different file which consists of all hosts and their corresponding passwords in the following format:
host pass
sarv sarva
savana 1save
simra xvyw23
punit asdwe87
I apologize for not describing it properly.
The first word of each line in the file is the host-name and the second word is it's password.
Instead of writing hosts=(sarv savana simra punit) pass=(sarva 1save xvyw23 asdwe87) in the script.
Reading them from a file which combines both host and password could be done like this:
while IFS=' ' read -r host pass
do
# your command using them goes here:
echo "$host=$pass"
done < hostpass-list.txt
IFS=' ' sets the input field separator to a space. The first space delimited word will be put in host and the rest of the line will be put in pass.
You may want to read all hosts/passwords first and use them later. In that case you could use an associative array.
# declare an associative array
declare -A hp
# fill the array
while IFS=' ' read -r host pass
do
hp["$host"]="$pass"
done < hostpass-list.txt
# use the array
for host in "${!hp[#]}"
do
pass="${hp["$host"]}"
# your command using them goes here:
echo "$host=$pass"
done
For reading from separate files (if that becomes necessary) I suggest using readarray:
readarray -t hosts < hosts-list.txt
readarray -t pass < passwords-list.txt
The -t means:
Remove a trailing delim (default newline) from each line read.
Of course:
#!/bin/bash
hosts=$(cat hosts-list.txt)
pass=$(cat passwords-list.txt)
for i in "${!hosts[#]}"; do
sshpass -p "${pass[i]}" ssh-copy-id -f root#"${hostnames[i]}" -p 22
done

Characters in string getting replaced when echoed

I am writing a simple script to collect 2 IP addresses. I am using the open stack client to gather the allocation pool of a provider network. I used awk to gather the 2 IP addresses (start and end) and put them into 2 variables. When I echo the 2 variables alone they print out how I expect. However, if I try to echo something after the variable, it seems to replace the first few characters of the IP address.
It hard to explain, but if you refer to the output it should make more sense. If you look at my script below, I just put the string "hello" after the variable in each echo statement for demonstrative purposes.
#!/bin/bash
NETWORK=$1
#just gets the allocation pool IP addresses from openstack
ALLOCATION_POOLS=$(openstack subnet show $NETWORK --insecure|grep -w "allocation_pools"|awk -F " " '{print $4}')
POOL_START=$(awk -F "-" '{print $1}' <<< "$ALLOCATION_POOLS")
echo $POOL_START"hello"
POOL_END=$(awk -F "-" '{print $2}' <<< "$ALLOCATION_POOLS")
echo $POOL_END"hello"
Here is the output:
hello.146.87
hello.146.126
If I did not put "hello" in the echo statement, the output looks more like this:
10.28.146.87
10.28.146.126
Another thing I did was tested the length of the strings, and the length was larger then the number of characters in the ip address. I believe that there is some strange character after the IP addresses that is causing this. If that is the case, how can I remove it?

Bash Issue: AWK

I came back to work from a break to see that my Bash script wasn't working like it used to. The below tid-bit of code would grab and filter what's in a file. Here's the contents of said file:
# A colon, ':', is used as the field terminator. A new line terminates
# the entry. Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
# $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively. The third filed indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
OEM:/software/oracle/agent/agent12c/core/12.1.0.3.0:N
*:/software/oracle/agent/agent11g:N
dev068:/software/oracle/ora-10.02.00.04.11:Y
dev299:/software/oracle/ora-10.02.00.04.11:Y
xtst036:/software/oracle/ora-10.02.00.04.11:Y
xtst161:/software/oracle/ora-10.02.00.04.11:Y
dev360:/software/oracle/ora-11.02.00.04.02:Y
dev361:/software/oracle/ora-11.02.00.04.02:Y
xtst215:/software/oracle/ora-11.02.00.04.02:Y
xtst216:/software/oracle/ora-11.02.00.04.02:Y
dev298:/software/oracle/ora-11.02.00.04.03:Y
xtst160:/software/oracle/ora-11.02.00.04.03:Y
What the code used to produce and throw into an array:
dev068
dev299
xtst036
xtst161
dev360
dev361
xtst215
xtst216
dev298
xtst160
It would look at the file (oratab), find the database names (e.g. xtst160), and put them into an array. I then used this array for other tasks later in the script. Here's the relevant Bash script code:
# Collect the databases using a mixture of AWK and regex, and throw it into an array.
printf "\n2) Collecting databases on %s:\n" $HOSTNAME
declare -a arr_dbs=(`awk -F: -v key='/software/oracle/ora' '$2 ~ key{print $ddma_input}' /etc/oratab`)
# Loop through and print the array of databases.
for i in ${arr_dbs[#]}
do
printf "%s " $i
done
It doesn't seem anyone has modified the code or that the oratab file format has changed. So I'm not 100% sure what's going on now. Instead of grabbing the few characters, it's grabbing the entire line:
dev068:/software/oracle/ora-10.02.00.04.11:Y
I'm trying to understand Bash and regex more but I'm stumped. Definitely not my forte. A broken down explanation of the awk line would be greatly appreciated.
I found the error. We changed the amount of arguments being passed in and the order they are received.
printing $1 instead $ddma_input and resolve the issue as well.
declare -a arr_dbs=(`awk -F ":" -v key='/software/oracle/ora' '$2 ~ key{print $1}' /etc/oratab`)
# Loop through and print the array of databases.
for i in ${arr_dbs[#]}
do
printf "%s " $i
done
You could easily implement this whole thing in native bash with no external tools at all:
arr_dbs=( )
while IFS= read -r line; do
case $line in
"#"*) continue ;;
*:/software/oracle/ora*:*) arr_dbs+=( "${line%%:*}" ) ;;
esac
done </etc/oratab
printf ' %s\n' "${arr_dbs[#]}"
This actually avoids some bugs you had in your original implementation. Let's say you had a line like the following:
*:/software/oracle/ora-default:Y
If you aren't careful with how you handle that *, it'll be replaced with a list of filenames in the current directory by the shell whenever expansion occurs.
What does "whenever expansion occurs" mean in this context? Well:
# this will expand a * into a list of filenames during the assignment to the array
arr=( $(echo "*") ) # vs the correct read -a arr < <(echo "*")
# this will expand a * into a list of filenames while generating items to iterate over
for i in ${arr[#]} # vs the correct for i in "${arr[#]}"
# this will expand a * into a list of filenames while building the argument list for echo
i="*"
echo $i # vs the correct printf '%s\n' "$i"
Note the use of printf over echo -- see the APPLICATION USAGE section of the POSIX specification of echo.

Bash for loop - change variable name

I have a for loop that takes a CIDR that contains an illegal character for file names ("/").
part of the command is to save the output with the name of the CIDR.
for i in $(more subnets.lst);do shodan download $i-shodan net:$i;done
the download argument is followed by the result of the more command, which are the CIDR (192.168.21.0/24).
is there a way in bash to rename a variable while the loop is running ?
I remember doing this years back in batch files by subtracting from the str length, but that won't help me as I just need to replace the "/" with a "-"(or any other compliant char.
You can do this with using bash Parameter Expansion:
$ echo $var
192.168.21.0/24
$ echo "${var//\//-}"
192.168.21.0-24
So in your command, just use "${i//\//-}" whenever needed without changing the original value of $i. If you want to set variable i to the new value:
i="${i//\//-}"
On a side note, use while loop to read lines from a file, not cat, more or brothers, like:
while IFS= read -r line; do ....; done
I worked it out for myself:
for i in $(more subnets-test);do shodan download $(echo $i | tr "/" "-") net:$i;done

Extract substrings from a file and store them in shell variables

I am working on a script. I have a file called test.txt whose contents are as follows:
a. parent = 192.168.1.2
b. child1 = 192.168.1.21
c. child2 = 192.154.1.2
I need to store the values in three different variables called parent, child1and child2 as follows and then my script will use these values:
parent = 192.168.1.2
child1= 192.168.1.21
child2= 192.154.1.2
How can I do that using sed or awk? I know there is a way to extract substrings using awk function substr but my particular requirement is tostore them in variables as mentioned above. Thanks
Try this if you're using bash:
$ declare $(awk '{print $2"="$4}' file)
$ echo "$parent"
192.168.1.2
If the file contained white space in the values you want to init the variables with then you'd just have to set IFS to a newline before invoking declare, e.g. (simplified the input file to highlight the important part of white space on the right of the = signs):
$ cat file
parent=192.168.1.2 is first
child1=192.168.1.21 comes after it
child2=and then theres 192.154.1.2
$ IFS=$'\n'; declare $(awk -F'=' '{print $1"="$2}' file)
$ echo "$parent"
192.168.1.2 is first
$ echo "$child1"
192.168.1.21 comes after it
Ed Morton's answer is the way to go for the specific problem at hand - elegant and concise.
Update: Ed has since updated his answer to also provide a solution that correctly deals with variable value values with embedded spaces - the original lack of which prompted this answer.
His solution is superior to this one - more concise and more efficient (the only caveat is that you may have to restore the previous $IFS value afterward).
This solution may still be of interest if you need to process variable definitions one by one, e.g., in order to transform variable values based on other shell functions or variables before assigning them.
The following uses bash with process substitution on a simplified problem to process variable definitions one by one:
#!/usr/bin/env bash
while read -r name val; do # read a name-value pair
# Assign the value after applying a transformation to it; e.g.:
# 'value of' -> 'value:'
declare $name="${val/ of /: }" # `declare "$name=${val/ of /: }"` would work too.
done < <(awk -F= '{print $1, $2}' <<<$'v1=value of v1\nv2= value of v2')
echo "v1=[$v1], v2=[$v2]" # -> 'v1=[value: v1], v2=[value: v2]'
awk's output lines are read line by line, split into name and value, and declared as shell variables individually.
Since read, which trims by whitespace, is only given 2 variable names to read into, the 2nd one receives everything from the 2nd token _through the end of the line, thus preserving interior whitespace (and, as written, will trim leading and trailing whitespace in the process).
Note that declare normally does not require a variable reference on the RHS of the assignment (the value) to be double-quoted (e.g. a=$b; though it never hurts). In this particular case, however - seemingly because the LHS (the name) is also a variable reference - the double quotes are needed.
I also got it done finally . Thanks everyone for helping.
counter=0
while read line
do
declare $(echo $line | awk '{print $2"="$4}')
#echo "$parent"
if [ $counter = 0 ]
then
parent=$(echo $parent)
fi
if [ $counter = 1 ]
then
child1=$(echo $child)
else
child2=$(echo $child)
fi
counter=$((counter+1))
done < "/etc/cluster_info.txt"
eval "$( sed 's/..//;s/ *//g' YourFile )"
just a sed equivalent to Ed solution and with an eval instead of declare.

Resources