Bash Script - getting input from standard input or a file - linux

I have a bash script that prints columns by name taken from the command line. It works well if I give the script the file as one of the arguments. It does not work well if I pipe input to the script and use /dev/stdin as the file. Does anyone know how I can modify the script to accept standard input from a pipe correctly? Here is my script.
#!/bin/bash
insep=" "
outsep=" "
while [[ ${#} > 0 ]]
do
option="$1"
if [ -f $option ] || [ $option = /dev/stdin ];
then
break;
fi
case $option in
-s|--in_separator)
insep="$2"
shift # past argument
shift # past argument
;;
-o|--out_separator)
outsep="$2"
shift # past argument
shift # past argument
;;
*)
echo "unknown option $option"
exit 1;
;;
esac
done
headers="${#:2}"
grep_headers=$(echo "${headers[#]}" | sed 's/ /|/g')
file=$1
columns=$(awk -F: 'NR==FNR{b[($2)]=tolower($1);next}{print $1,b[$1]}' \
<(head -1 $file | sed "s/$insep/\n/g" | egrep -iwn "$grep_headers" | awk '{s=tolower($0);print s}') \
<(awk -F: -v header="$headers" 'BEGIN {n=split(tolower(header),a," ");for(i=1;i<=n;i++) print a[i]}' $file ) \
| awk '{print "$"$2}' ORS='OFS' | sed "s/OFS\$//")
awk -v insep="$insep" -v outsep="$outsep" "BEGIN{FS=insep;OFS=outsep}{print $columns}" $file
exit;
Sample Input:
col_1 col_2 col_3 col_4 col_5 col_6 col_7 col_8 col_9 col_10
10000 10010 10020 10030 10040 10050 10060 10070 10080 10090
10001 10011 10021 10031 10041 10051 10061 10071 10081 10091
10002 10012 10022 10032 10042 10052 10062 10072 10082 10092
10003 10013 10023 10033 10043 10053 10063 10073 10083 10093
10004 10014 10024 10034 10044 10054 10064 10074 10084 10094
10005 10015 10025 10035 10045 10055 10065 10075 10085 10095
10006 10016 10026 10036 10046 10056 10066 10076 10086 10096
10007 10017 10027 10037 10047 10057 10067 10077 10087 10097
10008 10018 10028 10038 10048 10058 10068 10078 10088 10098
Running with file as an argument (works as expected):
> ./shell_scripts/print_columns.sh file1.txt col_1 col_4 col_6 col_2 | head
col_1 col_4 col_6 col_2
10000 10030 10050 10010
10001 10031 10051 10011
10002 10032 10052 10012
10003 10033 10053 10013
Piping from standard in (does not work as expected):
> head file1.txt | ./shell_scripts/print_columns.sh /dev/stdin col_1 col_4 col_6 col_2 | head
0185 10215 10195
10136 10166 10186 10146
10137 10167 10187 10147
10138 10168 10188 10148
10139 10169 10189 10149

An example:
script.sh:
#!/bin/bash
if [[ -f "$1" ]]; then
file="$1"
cat "$file"
shift
else
while read -r file; do echo "$file"; done
fi
echo "${#}"
Test with:
./script.sh file1.txt abc 123 456
and with UUOC:
cat file1.txt | ./script.sh abc 123 456

Related

Lines in File 1 not present in File 2 based on a column

I have 2 files
File 1 - IN.txt
08:43:22 IN 0xabc
08:43:31 IN 0xdef
08:54:45 IN 0xghi
08:54:45 IN 0xjkl
File 2 - OUT.txt
08:43:32 OUT 0xdef
08:54:45 OUT 0xghi
08:54:45 OUT 0xjkl
Basically I am troubleshooting a network issue, IN.txt is packets coming in, OUT.txt is packets going out and column 3 is the packet code so it should match for the packet in the same transaction.
I want to know all IN packets that do not have a matching OUT packet.
Desired output:
08:43:22 IN 0xabc
#!/bin/bash
IN=$(awk -F " " '{print $3}' in.txt)
OUT=$(awk -F " " '{print $3}' out.txt)
for i in $IN
do
flag=false
for o in $OUT
do
if [[ "$i" == "$o" ]]; then
flag=true
break
fi
done
if [[ $flag == false ]]; then
echo "Cannot find packet: $i in out"
fi
done
Result:
dingrui#gdcni:~/onie$ ./filter.sh
Cannot find packet: 0xabc in out
you can use a for.
for i in $(cat IN.txt| awk '{print $3}'); do grep -i $i OUT.txt | wc -l; done
Or more readable:
for i in $(cat IN.txt| awk '{print $3}'); do result=$(grep -i $i OUT.txt | wc -l);echo $i "|" $result; done
OUTPUT:
0xabc | 0
0xdef | 1
0xghi | 1
0xjkl | 1
NOTE: Only matches the packets, I didn't look at the time which doesn't seem important since you want to check packets
You can use fgrep for this:
$ cut -d' ' -f3 < OUT.txt > OUT.txt2
$ fgrep -v IN.txt -f OUT.txt2
08:43:22 IN 0xabc

What is the difference can be tested between mapfile and mapfile -t?

Sample 1 with option t:
f(){
mapfile -t aaa < /dev/stdin
echo -e ${aaa[#]}
}
echo -e "a\nb\nc" | f
Sample 2 without option t:
f(){
mapfile aaa < /dev/stdin
echo -e ${aaa[#]}
}
echo -e "a\nb\nc" | f
The outputs are same:
a b c
EDIT:
Why are there two whitespaces in #cdarke's and #heemayl's answer?
echo -e "a\nb\nc" | f2 | od -b
0000000 141 012 040 142 012 040 143 012 012
0000011
Then I tried this: echo -e "a\tb\tc" | f2 | od -b
0000000 141 011 142 011 143 012 012
0000007
And there are no whitespaces.
Thx in advance!
To see the difference quote ${aaa[#]}:
echo -e "${aaa[#]}"
Without quoting the expansion is being subjected to word splitting according to IFS (and pathname expansion too). As the elements (strings) contain newlines, these will go through word splitting.
Example:
$ f() { mapfile -t aaa < /dev/stdin; echo -e "${aaa[#]}" ;}
$ echo -e "a\nb\nc" | f
a b c
$ g() { mapfile aaa < /dev/stdin; echo -e "${aaa[#]}" ;}
$ echo -e "a\nb\nc" | g
a
b
c
The problem is with the echo:
f1(){
mapfile -t aaa < /dev/stdin
echo -e "${aaa[#]}" # note the quotes
}
echo -e "a\nb\nc" | f1
f2(){
mapfile aaa < /dev/stdin
echo -e "${aaa[#]}" # note the quotes
}
echo -e "a\nb\nc" | f2
Gives:
a b c
a
b
c
In the second case, without the quotes, the newlines are seen as whitespace.

Extract substring in BASH script whit characters treated as numbers

I need to extract a substring of string in bash script.
This is the code with "echos":
echo "number:"
echo "$number"
echo "bb"
registers3=$(echo $number | grep -o -E '[0-9]+')
registers2="$(grep -oE '[0-9]+' <<< "$number")"
registers="${number//[^0-9]/}"
valor=$(grep -o "[0-9]" <<<"$number")
echo "valor:"
echo $valor
echo "reg:"
echo "$registers"
echo "reg2:"
echo "$registers2"
echo "reg3:"
echo "$registers3"
And this the output:
number:
/ > -------
420
/ >
bb
valor:
1 0 3 4 4 2 0
reg:
1034420
reg2:
1034
420
reg3:
1034
420
the problem is the special characters of $number.
can you help me to extract only the number. in this case is "421"
Thanks!!!
EDIT:
If i put $number in file ($number> file.txt) and open with vi and :set list i get:
^[[?1034h/ > -------$
420$
/ > $
Instead of this:
registers=$(echo $number | grep -o -E '[0-9]+')
try this:
registers="$(grep -oE '[0-9]+' <<< "$number")"
or even better, this one:
registers="${number//[^0-9]/}"

Can I use hexdump in a shell script?

Can I use hexdump in a shell script?
When I use it I keep getting an error .
syntax error near unexpected token 'hexdump'
#!/bin/bash
#bash-hexdump
# Quick script to check delay of the shotpoints
echo " please enter the complete line name as mentioned in the RAID2 "
read $line
cd /argus/raid2/"$line"
echo
echo " Entering the directory "
echo
for file in /argus/raid2/"$line"/*.ffid
hexdump -e "16 \"%_p\" \"\\n\"" $FFID | sed -n '68,73p' > list1
done
for filename in 'cat list1'
do
sed -n 6p | awk '{print $1}' = $wd
cat list.txt | sed -n 1p | cut -c13-14 = $hh
cat list.txt | sed -n 1p | cut -c15-16 = $mm
cat list.txt | sed -n 2p | cut -c1-2 = $ss
done
while [ true ]
do
$FFID=`ls -1rt $1 | grep -i ffid | tail -1`
echo " FFID value is : $FFID"
while [ $FFID = `ls -1rt $1 | grep -i ffid | tail -1` ]
do
hexdump -e "16 \"%_p\" \"\\n\"" $FFID | sed -n '68,73p' > list
done
for filename in 'cat list'
do
cat list.txt | sed -n 1p | cut -c13-14 = $hh1
cat list.txt | sed -n 1p | cut -c15-16 = $mm1
cat list.txt | sed -n 2p | cut -c1-2 = $ss1
done
$time1 = "$hh"":""$mm"":""$ss" ;
$time2 = "$hh1"":""$mm1"":""$ss1" ;
$former_seconds = $(date --date= "$time1" +%s);
$later_seconds = $(date --date= "$time2" +%s);
$time_delay = ( "$later_seconds" - "$former_seconds" )
$wb_time = ( "$wd" * 1.33 )
if
(("$wb_time" + "$time_delay")) < 12.0
then
echo "please slow down"
fi
if [ -e EOL.ffid ]
then
echo "EOL.ffid detected, exiting script"
exit
fi
done
I am not able to figure out why the hexdump code is giving me an error . Please help .
You are missing the do in your for loop:
for file in /argus/raid2/"$line"/*.ffid
do
hexdump -e "16 \"%_p\" \"\\n\"" $FFID | sed -n '68,73p' > list1
done

ubuntu bash printing result in system with extra ""

Why do my result for A have "" and only capture first word while my B is fine?
File: sample.txt
Amos Tan:Sunny Day:22.5:3:2
Jason Ong:Rainy Day:20.5:3:2
Bryan Sing:Cloudy Day:29.5:3:2
Code in terminal:
cat ./sample.txt | while read A B
do
title=`echo “$A” | cut -f 1 -d ":"`
echo "Found $title"
author=`echo “$B” | cut -f 2 -d ":"`
echo "Found $author
done
Results:
Found “Amos”
Found Sunny Day
Found “Jason”
Found Rainy Day
Found “Bryan”
Found Cloudy Day
This line is the problem:
cat ./sample.txt | while read A B
It is reading first word into A and rest of the line in variable B.
You can better use:
while read -r line
do
title=$(echo "$line" | cut -f 1 -d ":")
echo "Found title=$title"
author=$(echo "$line" | cut -f 2 -d ":")
echo "Found author=$author"
done < ./sample.txt
Or simply use awk:
awk -F : '{printf "title=%s, author=%s\n", $1, $2}' sample.txt

Resources