This question already has answers here:
Why does adding spaces around bash comparison operator change the result?
(2 answers)
How to pass the value of a variable to the standard input of a command?
(9 answers)
Closed 1 year ago.
I want to check which lines of the file /etc/passwd end with the "/bin/bash" string (field number 7, ":" as delimiter).
So far, I've written the following code:
while read line
do
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
then
echo $line | cut -d : -f 1
echo "\n"
fi
done < /etc/passwd
Currently, executing the script throws errors that show a bad interpretation (most likely due to the syntax).
I'd appreciate if you could help me.
You MUST surround the == operator with spaces. [ and [[ do different things based on how many arguments are given:
if [ "$( echo "$line" | cut -d: -f7 )" == "/bin/bash" ]; ...
I would actually do this: parse the line into fields while you're reading it.
while IFS=: read -ra fields; do
[[ ${fields[-1]} == "/bin/bash" ]] && printf "%s\n\n" "${fields[0]}"
done < /etc/passwd
This line is wrong:
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
Also, this is not going to do what you want:
echo "\n"
Bash echo doesn't understand backslash-escaped characters without
-e. If you want to print a new line use just echo but notice that
the previous echo:
echo $line | cut -d : -f 1
will add a newline already.
You should always check your scripts with
shellcheck. The correct script would be:
#!/usr/bin/env bash
while read -r line
do
if [ "$(echo "$line" | cut -d : -f 7)" == "/bin/bash" ]
then
echo "$line" | cut -d : -f 1
fi
done < /etc/passwd
But notice that you don't really need a loop which is very slow and
could use the following awk one-liner:
awk -v FS=: '$7 == "/bin/bash" {print $1}' /etc/passwd
Instead of looping through the rows, and then checking for the /bin/bash part, why not use something like grep to get all the desired rows, like so:
grep ':/bin/bash$' /etc/passwd
Optionality, you can loop over the rows by using a simple while;
grep ':/bin/bash$' /etc/passwd | while read -r line ; do
echo "Processing $line"
done
Don't do while read | cut. Use IFS as:
#!/bin/sh
while IFS=: read name passwd uid gid gecos home shell; do
if test "$shell" = /bin/bash; then
echo "$name"
fi
done < /etc/passwd
But for this particular use case, it's probably better to do:
awk '$7 == "/bin/bash"{print $1}' FS=: /etc/passwd
The issue your code has is a common error. Consider the line:
if [ $("$line" | cut -d : -f 7)=="/bin/bash" ]
Assume you have a value in $line in which the final field is /bin/dash. The process substitution will insert the string /bin/dash, and bash will attempt to execute:
if [ /bin/dash==/bin/bash ]
since /bin/bash==/bin/bash is a non-empty string, the command [ /bin/bash==/bin/bash ] returns succesfully. It does not perform any sort of string comparison. In order for [ to do a string comparison, you need to pass it 4 arguments. For example, [ /bin/dash = /bin/bash ] would fail. Note the 4 arguments to that call are /bin/dash, =, /bin/bash, and ]. [ is an incredibly bizarre command that requires its final argument to be ]. I strongly recommend never using it, and replacing it instead with its cousin test (very closely related, indeed both test and [ used to be linked to the same executable) which behaves exactly the same but does not require its final argument to be ].
I have logs in this format:
log1,john,time,etc
log2,peter,time,etc
log3,jack,time,etc
log4,peter,time,etc
I want to create a list for every person in the format
"name"=("no.lines" "line" "line" ...)
For example:
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
I already have this structure and know how to create variables like
declare "${FIELD[1]}"=1
but I don't know how to increase number of records and I am getting an error if I want to create a list like this and append into it.
#!/bin/bash
F=("log1,john,time,etc" "log2,peter,time,etc" "log3,jack,time,etc" "log4,peter,time,etc")
echo "${F[#]}"
declare -a CLIENTS
for LINE in "${F[#]}"
do
echo "$LINE"
IFS=',' read -ra FIELD < <(echo "$LINE")
if [ -z "${!FIELD[1]}" ] && [ -n "${FIELD[1]}" ] # check if there is already record for given line, if not create
then
CLIENTS=("${CLIENTS[#]}" "${FIELD[1]}") # add person to list of variables records for later access
declare -a "${FIELD[1]}"=("1" "LINE") # ERROR
elif [ -n "${!FIELD[1]}" ] && [ -n "${FIELD[1]}" ] # if already record for client
then
echo "Increase records number" # ???
echo "Append record"
"${FIELD[#]}"=("${FIELD[#]}" "$LINE") # ERROR
else
echo "ELSE"
fi
done
echo -e "CLIENTS: \n ${CLIENTS[#]}"
echo "Client ${CLIENTS[0]} has ${!CLIENTS[0]} records"
echo "Client ${CLIENTS[1]} has ${!CLIENTS[1]} records"
echo "Client ${CLIENTS[2]} has ${!CLIENTS[2]} records"
echo "Client ${CLIENTS[3]} has ${!CLIENTS[3]} records"
Be warned: The below uses namevars, a new bash 4.3 feature.
First: I would strongly suggest namespacing your arrays with a prefix to avoid collisions with unrelated variables. Thus, using content_ as that prefix:
read_arrays() {
while IFS= read -r line && IFS=, read -r -a fields <<<"$line"; do
name=${fields[1]}
declare -g -a "content_${fields[1]}"
declare -n cur_array="content_${fields[1]}"
cur_array+=( "$line" )
unset -n cur_array
done
}
Then:
lines_for() {
declare -n cur_array="content_$1"
printf '%s\n' "${#cur_array[#]}" ## emit length of array for given person
}
...or...
for_each_line() {
declare -n cur_array="content_$1"; shift
for line in "${cur_array[#]}"; do
"$#" "$line"
done
}
Tying all this together:
$ read_arrays <<'EOF'
log1,john,time,etc
log2,peter,time,etc
log3,jack,time,etc
log4,peter,time,etc
EOF
$ lines_for peter
2
$ for_each_line peter echo
log2,peter,time,etc
log4,peter,time,etc
...and, if you really want the format you asked for, with the number of columns as explicit data, and variable names that aren't safely namespaced, it's easy to convert from one to the other:
# this should probably be run in a subshell to avoid namespace pollution
# thus, (generate_stupid_format) >output
generate_stupid_format() {
for scoped_varname in "${!content_#}"; do
unscoped_varname="${scoped_varname#content_}"
declare -n unscoped_var=$unscoped_varname
declare -n scoped_var=$scoped_varname
unscoped_var=( "${#scoped_var[#]}" "${scoped_var[#]}" )
declare -p "$unscoped_varname"
done
}
Bash with Coreutils, grep and sed
If I understand your code right, you try to have multidimensional arrays, which Bash doesn't support. If I were to solve this problem from scratch, I'd use this mix of command line tools (see security concerns at the end of the answer!):
#!/bin/bash
while read name; do
printf "%s=(\"%d\" \"%s\")\n" \
"$name" \
"$(grep -c "$name" "$1")" \
"$(grep "$name" "$1" | tr $'\n' ' ' | sed 's/ /" "/g;s/" "$//')"
done < <(cut -d ',' -f 2 "$1" | sort -u)
Sample output:
$ ./SO.sh infile
jack=("1" "log3,jack,time,etc")
john=("1" "log1,john,time,etc")
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
This uses process substitution to prepare the log file so we can loop over unique names; the output of the substitution looks like
$ cut -d ',' -f 2 "$1" | sort -u
jack
john
peter
i.e., a list of unique names.
For each name, we then print the summarized log line with
printf "%s=(\"%d\" \"%s\")\n"
Where
The %s string is just the name ("$name").
The log line count is the output of a grep command,
grep -c "$name" "$1"
which counts the number of occurrences of "$name". If the name can occur elsewhere in the log line, we can limit the search to just the second field of the log lines with
grep -c "$name" <(cut -d ',' -f 2 "$1")
Finally, to get all log lines on one line with proper quoting and all, we use
grep "$name" "$1" | tr $'\n' ' ' | sed 's/ /" "/g;s/" "$//'
This gets all lines containing "$name", replaces newlines with spaces, then surrounds the spaces with quotes and removes the extra quotes from the end of the line.
Pure Bash
After initially thinking that pure Bash would be too cumbersome, it turned out to be not all that complicated:
#!/bin/bash
declare -A count
declare -A lines
old_ifs=IFS
IFS=,
while read -r -a line; do
name="${line[1]}"
(( ++count[$name] ))
lines[$name]+="\"${line[*]}\" "
done < "$1"
for name in "${!count[#]}"; do
printf "%s=(\"%d\" %s)\n" "$name" "${count[$name]}" "${lines[$name]% }"
done
IFS="$old_ifs"
This updates two associative arrays while looping over the input file: count keeps track of the number of times a certain name occurs, and lines appends the log lines to an entry per name.
To separate fields by commas, we set the input field separator IFS to a comma (but save it beforehand so it can be reset at the end).
read -r -a reads the lines into an array line with comma separated fields, so the name is now in ${line[1]}. We increase the count for that name in the arithmetic expression (( ... )), and append (+=) the log line in the next line.
${line[*]} prints all fields of the array separated by IFS, which is exactly what we want. We also add a space here; the unwanted space at the end of the line (after the last element) will be removed later.
The second loop iterates over all the keys of the count array (the names), then prints the properly formatted line for each. ${lines[$name]% } removes the space from the end of the line.
Security concerns
As it seems that the output of these scripts is supposed to be reused by the shell, we might want to prevent malicious code execution if we can't trust the contents of the log file.
A way to do that for the Bash solution (hat tip: Charles Duffy) would be the following: the for loop would have to be replaced by
for name in "${!count[#]}"; do
IFS=' ' read -r -a words <<< "${lines[$name]}"
printf -v words_str '%q ' "${words[#]}"
printf "%q=(\"%d\" %s)\n" "$name" "${count[$name]}" "${words_str% }"
done
That is, we split the combined log lines into an array words, print that with the %q formatting flag into a string words_str and then use that string for our output, resulting in escaped output like this:
peter=("2" \"log2\,peter\,time\,etc\" \"log4\,peter\,time\,etc\")
jack=("1" \"log3\,jack\,time\,etc\")
john=("1" \"log1\,john\,time\,etc\")
The analogous could be done for the first solution.
You can use awk. As a demo:
awk -F, '{a1[$2]=a1[$2]" \""$0"\""; sum[$2]++} END{for (e in sum){print e"=(" "\""sum[e]"\""a1[e]")"}}' file
john=("1" "log1,john,time,etc")
peter=("2" "log2,peter,time,etc" "log4,peter,time,etc")
jack=("1" "log3,jack,time,etc")
I want to add some users who are in this file like:
a b
c d
e f
firstname lastname always
#!/bin/bash
Lines=$(cat newusers.txt | wc -l)
first=$(cat newusers.txt | awk '{print $1}')
last=$(cat newusers.txt | awk '{print $2}')
#test
echo $Lines;
echo $first;
echo $last;
until [ -z $1]; then
useradd - m -d /home/$1 -c "$1 + $2" $1
fi
before loop it works fine but I can't add newline.
The echo shows a c e and second for lastname b d f.
I tried to add newline in but it doesn't works.
What can i use for this?
Because I guess I can't add the user because of the newline problem.
I also searched on stackoverflow to find out a way to check if the user already exists by /dev/null but which variable do i have to use for it?
It's easier to process the file line by line:
while read first last ; do
useradd -m -d /home/"$first" -c "$fist + $last" "$first"
done < newusers.txt
I do not understand what you mean to do by your code, but if you want to read the file line by line and get the values of different fields then you can use the following code snippet:
#!/bin/bash
filename="newusers.txt"
while read -r line
do
fn=$( echo "$line" |cut -d" " -f1 )
ln=$( echo "$line" |cut -d" " -f2 )
echo "$fn $ln"
done < "$filename"
Note: You cannot add users the way you want to using bash script; since you will be prompted for password which must be supplied using tty you can use expect to program it; or use system calls.
I loaded a csv file into a variable and then I am trying to cut out some columns which results in this error /usr/bin/cut: Argument list too long. Here is what I did:
if [ $# -ne 1 ]; then
echo "you need to add the file name as argument"
fi
echo "input file name $1"
input_file=$(<$1)
#cut the required columns.
cut_input_file=$(cut -f 1,2,3,5,8,9,10 -d \| $input_file)
echo $(head $cut_input_file)
What am I missing?
Reason of that error is your use of $input_file which has full file data.
You need to run cut on the file not on the file content so use:
cut -f 1,2,3,5,8,9,10 -d '|' "$1"
To run cut against file content use:
cut -f 1,2,3,5,8,9,10 -d '|' <<< "$input_file"
I'm trying to get a specific line of a variable. The line I must get is stored in i. My code looks like this right now.
$(echo "$data" | sed '$iq;d')
It looks like I'm putting i in there wrong, Putting a number in for i works fine but $i gets me the entire string.
I haven't found a solution that works with a variable yet and I'm not too familiar with bash and would appreciate help,
Edit: a bit of context
i=5
data=$(netstat -a | grep ESTAB)
line=$(echo "$data" | sed "${i}p")
echo $line
Use sed -n "${i}p" instead.
Example:
i=4; seq 1 10 | sed -n "${i}p"
Output:
4
Bonus:
i=5
readarray -O 1 -t data < <(exec netstat -a | grep ESTAB) ## Stores data as an array of lines starting at index 1
line=${data[i]}
echo "$line"
# printf '%s\n' "${data[#]}" ## Prints whole data.
Here is way you can do this in BASH itself:
IFS=$'\n' arr=($data)
echo "${arr[$i]}"