Command substitution with strings in Bash [duplicate] - linux

This question already has answers here:
Why does adding spaces around bash comparison operator change the result?
(2 answers)
How to pass the value of a variable to the standard input of a command?
(9 answers)
Closed 1 year ago.
I want to check which lines of the file /etc/passwd end with the "/bin/bash" string (field number 7, ":" as delimiter).
So far, I've written the following code:
while read line
do
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
then
echo $line | cut -d : -f 1
echo "\n"
fi
done < /etc/passwd
Currently, executing the script throws errors that show a bad interpretation (most likely due to the syntax).
I'd appreciate if you could help me.

You MUST surround the == operator with spaces. [ and [[ do different things based on how many arguments are given:
if [ "$( echo "$line" | cut -d: -f7 )" == "/bin/bash" ]; ...
I would actually do this: parse the line into fields while you're reading it.
while IFS=: read -ra fields; do
[[ ${fields[-1]} == "/bin/bash" ]] && printf "%s\n\n" "${fields[0]}"
done < /etc/passwd

This line is wrong:
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
Also, this is not going to do what you want:
echo "\n"
Bash echo doesn't understand backslash-escaped characters without
-e. If you want to print a new line use just echo but notice that
the previous echo:
echo $line | cut -d : -f 1
will add a newline already.
You should always check your scripts with
shellcheck. The correct script would be:
#!/usr/bin/env bash
while read -r line
do
if [ "$(echo "$line" | cut -d : -f 7)" == "/bin/bash" ]
then
echo "$line" | cut -d : -f 1
fi
done < /etc/passwd
But notice that you don't really need a loop which is very slow and
could use the following awk one-liner:
awk -v FS=: '$7 == "/bin/bash" {print $1}' /etc/passwd

Instead of looping through the rows, and then checking for the /bin/bash part, why not use something like grep to get all the desired rows, like so:
grep ':/bin/bash$' /etc/passwd
Optionality, you can loop over the rows by using a simple while;
grep ':/bin/bash$' /etc/passwd | while read -r line ; do
echo "Processing $line"
done

Don't do while read | cut. Use IFS as:
#!/bin/sh
while IFS=: read name passwd uid gid gecos home shell; do
if test "$shell" = /bin/bash; then
echo "$name"
fi
done < /etc/passwd
But for this particular use case, it's probably better to do:
awk '$7 == "/bin/bash"{print $1}' FS=: /etc/passwd
The issue your code has is a common error. Consider the line:
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
Assume you have a value in $line in which the final field is /bin/dash. The process substitution will insert the string /bin/dash, and bash will attempt to execute:
if [ /bin/dash==/bin/bash ]
since /bin/bash==/bin/bash is a non-empty string, the command [ /bin/bash==/bin/bash ] returns succesfully. It does not perform any sort of string comparison. In order for [ to do a string comparison, you need to pass it 4 arguments. For example, [ /bin/dash = /bin/bash ] would fail. Note the 4 arguments to that call are /bin/dash, =, /bin/bash, and ]. [ is an incredibly bizarre command that requires its final argument to be ]. I strongly recommend never using it, and replacing it instead with its cousin test (very closely related, indeed both test and [ used to be linked to the same executable) which behaves exactly the same but does not require its final argument to be ].

Related

Unix Script loop through individual variables in a list and execute code

I have been busting my head all day long without coming up with a sucessfull solution.
Setup:
We have Linux RHEL 8.3 and a file, script.sh
There is an enviroment variable set by an application with a dynamic string in it.
export PROGARM_VAR="abc10,def20,ghi30"
The delimiter is always "," and the values inside vary from 1 to 20.
Inside the script I have defined 20 variables which take the values
using "cut" command I take each value and assign it to a variable
var1=$(echo $PROGARM_VAR | cut -f1 -d,)
var2=$(echo $PROGARM_VAR | cut -f2 -d,)
var3=$(echo $PROGARM_VAR | cut -f3 -d,)
var4=$(echo $PROGARM_VAR | cut -f4 -d,)
etc
In our case we will have:
var1="abc10" var2="def20" var3="ghi30" and var4="" which is empty
The loop must take each variable, test if its not empty and execute 10 pages of code using the tested variable. When it reaches an empty variable it should break.
Could you give me a hand please?
Thank you
Just split it with a comma. There are endless possibilities. You could:
10_pages_of_code() { echo "$1"; }
IFS=, read -a -r vars <<<"abc10,def20,ghi30"
for i in "${vars[#]}"; do 10_pages_of_code "$i"; done
or:
printf "%s" "abc10,def20,ghi30" | xargs -n1 -d, bash -c 'echo 10_pages_of_code "$1"' _
A safer code could use readarray instead of read to properly handle newlines in values, but I doubt that matters for you:
IFS= readarray -d , -t vars < <(printf "%s" "abc10,def20,ghi30")
You could also read in a stream up:
while IFS= read -r -d, var || [[ -n "$var" ]]; do
10_pages_of_code "$var"
done < <(printf "%s" "abc10,def20,ghi30")
But still you could do it with cut... just actually write a loop and use an iterator.
i=0
while var=$(printf "%s\n" "$PROGARM_VAR" | cut -f"$i" -d,) && [[ -n "$var" ]]; do
10_pages_of_code "$var"
((i++))
done
or
echo "$PROGRAM_VAR" | tr , \\n | while read var; do
: something with $var
done

issues with grep variable and white space

I am trying to get this bit of code to work. and I am getting hung up on the second piped grep with the variable $pkgname. I am unable to find a way to get it to read the variable correctly either I get no output or as the code is currently written I get grep: illegal byte sequence. if I put either text with no space in the variable it works or I enter the text as part of the grep statement it works.
#!/bin/bash
counter=0
results2=Executing\ SSH\ MOTD\ banner
pkgname=SSH\ MOTD\ banner
until [ $counter = 1 ]
do
echo $counter
echo $pkgname
echo $results2
result=$(grep "$(date +"%b %d")" /var/log/test.log | grep “$pkgname” | cut -d':' -f 4 | sed 's/^ *//g')
echo $result
if [ “$result” == “$results2” ]; then
counter=1
fi
done
echo finished
so the log file line I am looking for looks like this.
Tue Jun 28 10:58:57 machinename process: Executing SSH MOTD banner
change to
pkgname="SSH MOTD banner" # Here use quotes to avoid using \
As [ #jack ] rightly pointed out in this [ comment ], you need the neutral quotation mark " for a variable to be expanded. That said you can simplify the regex to below
pkgname="SSH MOTD banner"
d=$(date +"%b %d")
result="$(awk -v FS=":" -v d="$d" -v pkg="$pkg_name" '{if($0 ~ date && $0 ~ pkg){sub(/^ */,"",$4);print $4}}' /var/log/test.log)"

Reading and comparing column in .txt file in bash

I got .txt file which its content is
5742060626,Ms.Pimpan Tantivaravong,Female
5742065826,Ms.Kaotip Tanti,Female
-
I create an interface script to add list in this file
First, I have to compare the input id with the exitsting id in a list.
I use cut command to read only 1st column of .txt file.
But,I got a problem when I am trying to compare it.
Here is my code.
-
!/bin/bash
#
datafile='student-2603385.txt'
while read p;
do
if [ "$id" == (echo $p | cut -d, -f1) ]
then
echo 'duplicate id'
fi
done <$datafile
-
could anyone suggest me, how should I do?
Thank you
Your script has numerous quoting bugs, always quote variable expansion when the variable contains a file name, it is also expected when you want to avoid word splitting and pathname expansion by shell.
Letting that aside, in if [ "$id" == (echo $p | cut -d, -f1) ]:
You need command substitution, $() around echo ... | cut ..., not a subshell ()
you also need quotes around $() to prevent word splitting (and pathname expansion)
== is bash-ism, not defined by POSIX, just a reminder
try to use [[ as much as possible, being a shell keyword [[ handles word splitting
So with test ([):
if [ "$id" == "$(echo "$p" | cut -d, -f1)" ]
better:
if [[ $id == $(echo "$p" | cut -d, -f1) ]]

Dynamic indirect Bash array

I have logs in this format:
log1,john,time,etc
log2,peter,time,etc
log3,jack,time,etc
log4,peter,time,etc
I want to create a list for every person in the format
"name"=("no.lines" "line" "line" ...)
For example:
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
I already have this structure and know how to create variables like
declare "${FIELD[1]}"=1
but I don't know how to increase number of records and I am getting an error if I want to create a list like this and append into it.
#!/bin/bash
F=("log1,john,time,etc" "log2,peter,time,etc" "log3,jack,time,etc" "log4,peter,time,etc")
echo "${F[#]}"
declare -a CLIENTS
for LINE in "${F[#]}"
do
echo "$LINE"
IFS=',' read -ra FIELD < <(echo "$LINE")
if [ -z "${!FIELD[1]}" ] && [ -n "${FIELD[1]}" ] # check if there is already record for given line, if not create
then
CLIENTS=("${CLIENTS[#]}" "${FIELD[1]}") # add person to list of variables records for later access
declare -a "${FIELD[1]}"=("1" "LINE") # ERROR
elif [ -n "${!FIELD[1]}" ] && [ -n "${FIELD[1]}" ] # if already record for client
then
echo "Increase records number" # ???
echo "Append record"
"${FIELD[#]}"=("${FIELD[#]}" "$LINE") # ERROR
else
echo "ELSE"
fi
done
echo -e "CLIENTS: \n ${CLIENTS[#]}"
echo "Client ${CLIENTS[0]} has ${!CLIENTS[0]} records"
echo "Client ${CLIENTS[1]} has ${!CLIENTS[1]} records"
echo "Client ${CLIENTS[2]} has ${!CLIENTS[2]} records"
echo "Client ${CLIENTS[3]} has ${!CLIENTS[3]} records"
Be warned: The below uses namevars, a new bash 4.3 feature.
First: I would strongly suggest namespacing your arrays with a prefix to avoid collisions with unrelated variables. Thus, using content_ as that prefix:
read_arrays() {
while IFS= read -r line && IFS=, read -r -a fields <<<"$line"; do
name=${fields[1]}
declare -g -a "content_${fields[1]}"
declare -n cur_array="content_${fields[1]}"
cur_array+=( "$line" )
unset -n cur_array
done
}
Then:
lines_for() {
declare -n cur_array="content_$1"
printf '%s\n' "${#cur_array[#]}" ## emit length of array for given person
}
...or...
for_each_line() {
declare -n cur_array="content_$1"; shift
for line in "${cur_array[#]}"; do
"$#" "$line"
done
}
Tying all this together:
$ read_arrays <<'EOF'
log1,john,time,etc
log2,peter,time,etc
log3,jack,time,etc
log4,peter,time,etc
EOF
$ lines_for peter
2
$ for_each_line peter echo
log2,peter,time,etc
log4,peter,time,etc
...and, if you really want the format you asked for, with the number of columns as explicit data, and variable names that aren't safely namespaced, it's easy to convert from one to the other:
# this should probably be run in a subshell to avoid namespace pollution
# thus, (generate_stupid_format) >output
generate_stupid_format() {
for scoped_varname in "${!content_#}"; do
unscoped_varname="${scoped_varname#content_}"
declare -n unscoped_var=$unscoped_varname
declare -n scoped_var=$scoped_varname
unscoped_var=( "${#scoped_var[#]}" "${scoped_var[#]}" )
declare -p "$unscoped_varname"
done
}
Bash with Coreutils, grep and sed
If I understand your code right, you try to have multidimensional arrays, which Bash doesn't support. If I were to solve this problem from scratch, I'd use this mix of command line tools (see security concerns at the end of the answer!):
#!/bin/bash
while read name; do
printf "%s=(\"%d\" \"%s\")\n" \
"$name" \
"$(grep -c "$name" "$1")" \
"$(grep "$name" "$1" | tr $'\n' ' ' | sed 's/ /" "/g;s/" "$//')"
done < <(cut -d ',' -f 2 "$1" | sort -u)
Sample output:
$ ./SO.sh infile
jack=("1" "log3,jack,time,etc")
john=("1" "log1,john,time,etc")
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
This uses process substitution to prepare the log file so we can loop over unique names; the output of the substitution looks like
$ cut -d ',' -f 2 "$1" | sort -u
jack
john
peter
i.e., a list of unique names.
For each name, we then print the summarized log line with
printf "%s=(\"%d\" \"%s\")\n"
Where
The %s string is just the name ("$name").
The log line count is the output of a grep command,
grep -c "$name" "$1"
which counts the number of occurrences of "$name". If the name can occur elsewhere in the log line, we can limit the search to just the second field of the log lines with
grep -c "$name" <(cut -d ',' -f 2 "$1")
Finally, to get all log lines on one line with proper quoting and all, we use
grep "$name" "$1" | tr $'\n' ' ' | sed 's/ /" "/g;s/" "$//'
This gets all lines containing "$name", replaces newlines with spaces, then surrounds the spaces with quotes and removes the extra quotes from the end of the line.
Pure Bash
After initially thinking that pure Bash would be too cumbersome, it turned out to be not all that complicated:
#!/bin/bash
declare -A count
declare -A lines
old_ifs=IFS
IFS=,
while read -r -a line; do
name="${line[1]}"
(( ++count[$name] ))
lines[$name]+="\"${line[*]}\" "
done < "$1"
for name in "${!count[#]}"; do
printf "%s=(\"%d\" %s)\n" "$name" "${count[$name]}" "${lines[$name]% }"
done
IFS="$old_ifs"
This updates two associative arrays while looping over the input file: count keeps track of the number of times a certain name occurs, and lines appends the log lines to an entry per name.
To separate fields by commas, we set the input field separator IFS to a comma (but save it beforehand so it can be reset at the end).
read -r -a reads the lines into an array line with comma separated fields, so the name is now in ${line[1]}. We increase the count for that name in the arithmetic expression (( ... )), and append (+=) the log line in the next line.
${line[*]} prints all fields of the array separated by IFS, which is exactly what we want. We also add a space here; the unwanted space at the end of the line (after the last element) will be removed later.
The second loop iterates over all the keys of the count array (the names), then prints the properly formatted line for each. ${lines[$name]% } removes the space from the end of the line.
Security concerns
As it seems that the output of these scripts is supposed to be reused by the shell, we might want to prevent malicious code execution if we can't trust the contents of the log file.
A way to do that for the Bash solution (hat tip: Charles Duffy) would be the following: the for loop would have to be replaced by
for name in "${!count[#]}"; do
IFS=' ' read -r -a words <<< "${lines[$name]}"
printf -v words_str '%q ' "${words[#]}"
printf "%q=(\"%d\" %s)\n" "$name" "${count[$name]}" "${words_str% }"
done
That is, we split the combined log lines into an array words, print that with the %q formatting flag into a string words_str and then use that string for our output, resulting in escaped output like this:
peter=("2" \"log2\,peter\,time\,etc\" \"log4\,peter\,time\,etc\")
jack=("1" \"log3\,jack\,time\,etc\")
john=("1" \"log1\,john\,time\,etc\")
The analogous could be done for the first solution.
You can use awk. As a demo:
awk -F, '{a1[$2]=a1[$2]" \""$0"\""; sum[$2]++} END{for (e in sum){print e"=(" "\""sum[e]"\""a1[e]")"}}' file
john=("1" "log1,john,time,etc")
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
jack=("1" "log3,jack,time,etc")

Command to count the characters present in the variable

I am trying to count the number of characters present in the variable. I used the below shell command. But I am getting error - command not found in line 4
#!/bin/bash
for i in one; do
n = $i | wc -c
echo $n
done
Can someone help me in this?
In bash you can just write ${#string}, which will return the length of the variable string, i.e. the number of characters in it.
Something like this:
#!/bin/bash
for i in one; do
n=$(echo $i | wc -c)
echo $n
done
Assignments in bash cannot have a space before the equals sign. In addition, you want to capture the output of the command you run and assign that to $n, rather than that statement which would probably just assign $i to $n.
Use the following instead:
#!/bin/bash
for i in one; do
n=`$i | wc -c`
echo $n
done
It can be as simple as that:
str="abcdef"; wc -c <<< "$str"
7
But mind you that end of line counts as a character:
str="abcdef"; cat -A <<< "$str"
abcdef$
If you need to remove it:
str="abcdef"; tr -d '\n' <<< "$str" | wc -c
6

Resources