how to extract grep and cut into a bash array - linux

I tried:
here is content of file.txt
some other text
#1.something1=kjfk
#2.something2=dfkjdk
#3.something3=3232
some other text
bash script:
ids=( `grep "something" file.txt | cut -d'.' -f1` )
for id in "${ids[#]}"; do
echo $id
done
result:
(nothing newline...)
(nothing newline...)
(nothing newline...)
but all it prints is nothing like newline for every such id found what am i missing?

Your grep and cut should be working but you can use awk and reduce 2 commands into one:
while read -r id;
echo "$id"
done < <(awk -F '\\.' '/something/{print $1}' file.txt)
To populate an array:
ids=()
while read -r id;
ids+=( "$id" )
done < <(awk -F '\\.' '/something/{print $1}' file.txt)

You can use grep's -o option to output only the text matched by a regular expression:
$ ids=($(grep -Eo '^#[0-9]+' file.txt))
$ echo ${ids[#]}
#1 #2 #3
This of course doesn't check for the existence of a period on the line... If that's important, then you could either expand things with another pipe:
$ ids=($(grep -Eo '^#[0-9]+\.something' file.txt | grep -o '^#[0-9]*'))
or you could trim the array values after populating the array:
$ ids=($(grep -Eo '^#[0-9]+\.something' file.txt))
$ echo ${ids[#]}
#1.something #2.something #3.something
$ for key in "${!ids[#]}"; do ids[key]="${ids[key]%.*}"; done
$ echo ${ids[#]}
#1 #2 #3

Related

Extract left and right from filename based on last delimiter using shell script

I would like to extract left and right from filename based on last delimiter using shell script
Ex : filename = 'test_testone_test3_1234'
I'm expecting below values
file = 'test_testone_test3'
seq = '1234'
Is there a better approach to do it ?
You may want to learn a little bit about Parameter Expansion
$ filename="test_testone_test3_1234"
$ echo "${filename#*_}"
testone_test3_1234
$ echo "${filename##*_}"
1234
$ echo "${filename%_*}"
test_testone_test3
$ echo "${filename%%_*}"
test
$ echo "${filename#*_*_}"
test3_1234
$ echo "${filename%_*_*}"
test_testone
You can also use cut.-d is for field separator and -f for the fields.
$ filename="test_testone_test3_1234"
$ echo "$filename" | cut -d_ -f1-3
test_testone_test3
$ echo "$filename" | cut -d_ -f4
1234
$ echo "$filename" | cut -d_ -f1-2
test_testone
$ echo "$filename" | cut -d_ -f3-
test3_1234

Increment variable when matched awk from tail

I'm monitoring from an actively written to file:
My current solution is:
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | grep -q -e "enterpriseID:"
if [ $? = 0 ]
then
((ws_trans++))
fi
echo $LINE | grep -q -e "sc_ID:"
if [ $? = 0 ]
then
((sc_trans++))
fi
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
However when attempting to do this with AWK I don't get the output - the $ws_trans and $sc_trans remains 0
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | awk '/enterpriseID:/ {++ws_trans} END {print | ws_trans}'
echo $LINE | awk '/sc_ID:/ {++sc_trans} END {print | sc_trans}'
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
Attempting to do this to reduce load. I understand that AWK doesn't deal with bash variables, and it can get quite confusing, but the only reference I found is a non tail application of AWK.
How can I assign the AWK Variable to the bash ws_trans and sc_trans? Is there a better solution? (There are other search terms being monitored.)
You need to pass the variables using the option -v, for example:
$ var=0
$ printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} {print awk_var}'
To set the variable "back" you could use declare, for example:
$ declare $(printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} END {print "var=" awk_var}')
$ echo $var
$ 10
Your script could be rewritten like this:
ws_trans=0
sc_trans=0
tail -F /var/log/system.log |
while read LINE
do
declare $(echo $LINE | awk -v ws=${ws_trans} '/enterpriseID:/ {++ws} END {print "ws_trans="ws}')
declare $(echo $LINE | awk -v sc=${sc_trans} '/sc_ID:/ {++sc} END {print "sc_trans="sc}')
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done

Linux usernames /etc/passwd listing

I want to print the longest and shortest username found in /etc/passwd. If I run the code below it works fine for the shortest (head -1), but doesn't run for (sort -n |tail -1 | awk '{print $2}). Can anyone help me figure out what's wrong?
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |head -1 | awk '{print $2}'
sort -n |tail -1 | awk '{print $2}'
Here the issue is:
Piping finishes with the first sort -n |head -1 | awk '{print $2}' command. So, input to first command is provided through piping and output is obtained.
For the second command, no input is given. So, it waits for the input from STDIN which is the keyboard and you can feed the input through keyboard and press ctrl+D to obtain output.
Please run the code like below to get desired output:
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |head -1 | awk '{print $2}'
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n |tail -1 | awk '{print $2}
'
All you need is:
$ awk -F: '
NR==1 { min=max=$1 }
length($1) > length(max) { max=$1 }
length($1) < length(min) { min=$1 }
END { print min ORS max }
' /etc/passwd
No explicit loops or pipelines or multiple commands required.
The problem is that you only have two pipelines, when you really need one. So you have grep | while read do ... done | sort | head | awk and sort | tail | awk: the first sort has an input (i.e., the while loop) - the second sort doesn't. So the script is hanging because your second sort doesn't have an input: or rather it does, but it's STDIN.
There's various ways to resolve:
save the output of the while loop to a temporary file and use that as an input to both sort commands
repeat your while loop
use awk to do both the head and tail
The first two involve iterating over the password file twice, which may be okay - depends what you're ultimately trying to do. But using a small awk script, this can give you both the first and last line by way of the BEGIN and END blocks.
While you already have good answers, you can also use POSIX shell to accomplish your goal without any pipe at all using the parameter expansion and string length provided by the shell itself (see: POSIX shell specifiction). For example you could do the following:
#!/bin/sh
sl=32;ll=0;sn=;ln=; ## short len, long len, short name, long name
while read -r line; do ## read each line
u=${line%%:*} ## get user
len=${#u} ## get length
[ "$len" -lt "$sl" ] && { sl="$len"; sn="$u"; } ## if shorter, save len, name
[ "$len" -gt "$ll" ] && { ll="$len"; ln="$u"; } ## if longer, save len, name
done </etc/passwd
printf "shortest (%2d): %s\nlongest (%2d): %s\n" $sl "$sn" $ll "$ln"
Example Use/Output
$ sh cketcpw.sh
shortest ( 2): at
longest (17): systemd-bus-proxy
Using either pipe/head/tail/awk or the shell itself is fine. It's good to have alternatives.
(note: if you have multiple users of the same length, this just picks the first, you can use a temp file if you want to save all names and use -le and -ge for the comparison.)
If you want both the head and the tail from the same input, you may want something like sed -e 1b -e '$!d' after you sort the data to get the top and bottom lines using sed.
So your script would be:
#!/bin/bash
grep -Eo '^([^:]+)' /etc/passwd |
while read NAME
do
echo ${#NAME} ${NAME}
done |
sort -n | sed -e 1b -e '$!d'
Alternatively, a shorter way:
cut -d":" -f1 /etc/passwd | awk '{ print length, $0 }' | sort -n | cut -d" " -f2- | sed -e 1b -e '$!d'

How to search through a string and extract the required value in unix

I have a string like below
QUERY_RESULT='88371087|COB-A#2014-04-22,COB-C#2014-04-22,2014-04-22,2014-04-23 88354188|COB-W#2014-04-22,2014-04-22,2014-04-22 88319898|COB-X#2014-04-22,COB-K#2014-04-22,2014-04-22,2014-04-22'
This is a result taken by querying the database. Now I want to take all the values before the pipe and separate it with coma. So the output needed is :
A='88371087,88354188,88319898'
The db values can be different every time, there can be just one value or 2 or more values
How do I do it.
Using awk
A=`echo $QUERY_RESULT | awk '{ nreg=split($0,reg);for(i=1;i<=nreg;i++){split(reg[i],fld,"|");printf("%s%s",(i==1?"":","),fld[1]);}}'`
echo $A
88371087,88354188,88319898
Using grep -oP
grep -oP '(^| )\K[^|]+' <<< "$QUERY_RESULT"
88371087
88354188
88319898
OR to get comma separated value:
A=$(grep -oP '(^| )\K[^|]+' <<< "$QUERY_RESULT"|sed '$!s/$/,/'|tr -d '\n')
echo "$A"
88371087,88354188,88319898
$ words=( $( grep -oP '\S+(?=\|)' <<< "$QUERY_RESULT") )
$ A=$(IFS=,; echo "${words[*]}")
$ echo "$A"
88371087,88354188,88319898
Bash only.
shopt -s extglob
result=${QUERY_RESULT//|+([^ ]) /,}
result=${result%|*}
echo "$result"
Output:
88371087,88354188,88319898

Need to grab data inbetween tilde character

Can any one advise how to search on linux for some data between a tilde character. I need to get IP data however its been formed like the below.
Details:
20110906000418~118.221.246.17~DATA~DATA~DATA
One more:
echo '20110906000418~118.221.246.17~DATA~DATA~DATA' | sed -r 's/[^~]*~([^~]+)~.*/\1/'
echo "20110906000418~118.221.246.17~DATA~DATA~DATA" | cut -d'~' -f2
This uses the cut command with the delimiter set to ~. The -f2 switch then outputs just the 2nd field.
If the text you give is in a file (called filename), try:
grep "[0-9]*~" filename | cut -d'~' -f2
With cut:
echo "20110906000418~118.221.246.17~DATA~DATA~DATA" | cut -d~ -f2
With awk:
echo "20110906000418~118.221.246.17~DATA~DATA~DATA"
| awk -F~ '{ print $2 }'
In awk:
echo '20110906000418~118.221.246.17~DATA~DATA~DATA' | awk -F~ '{print $2}'
Just use bash
$ string="20110906000418~118.221.246.17~DATA~DATA~DATA"
$ echo ${string#*~}
118.221.246.17~DATA~DATA~DATA
$ string=${string#*~}
$ echo ${string%%~*}
118.221.246.17
one more, using perl:
$ perl -F~ -lane 'print $F[1]' <<< '20110906000418~118.221.246.17~DATA~DATA~DATA'
118.221.246.17
bash:
#!/bin/bash
IFS='~'
while read -a array;
do
echo ${array[1]}
done < ip
If string is constant, the following parameter expansion performs substring extraction:
$ a=20110906000418~118.221.246.17~DATA~DATA~DATA
$ echo ${a:15:14}
118.221.246.17
or using regular expressions in bash:
$ echo $(expr "$a" : '[^~]*~\([^~]*\)~.*')
118.221.246.17
last one, again using pure bash methods:
$ tmp=${a#*~}
$ echo $tmp
118.221.246.17~DATA~DATA~DATA
$ echo ${tmp%%~*}
118.221.246.17

Resources