Keep zero after Hexdecimal conversion in bash - linux

I'm writing a very simple script in bash:
for i in {000..1024} ; do
echo $i ;
echo "obase=16; $i" | bc
done
I want to print all numbers with minimum three digits as 000, 001, 002 .. until 099. With integer numbers it works good, but after obase = 16; $ i '| bcs number return with one or two digits.
How can I solve it in the easiest way?

You might want to use printf to format the number, for example:
for i in {0..1024} ; do
echo $i
printf '%03X\n' $i
done
Or j=$(printf '%03X' $i) and then echo $j.
For more on formatting check the Format strings subsection of Syntax section here.

Not sure it's "easiest", but generally printf is good for formatting issues. eg
printf "%03s\n" "$(echo "obase=16; $i" | bc)"

Related

compare float numbers with integers to get output in shell programing? [duplicate]

I have this Bash script and I had a problem in line 16.
How can I take the previous result of line 15 and add
it to the variable in line 16?
#!/bin/bash
num=0
metab=0
for ((i=1; i<=2; i++)); do
for j in `ls output-$i-*`; do
echo "$j"
metab=$(cat $j|grep EndBuffer|awk '{sum+=$2} END { print sum/120}') (line15)
num= $num + $metab (line16)
done
echo "$num"
done
For integers:
Use arithmetic expansion: $((EXPR))
num=$((num1 + num2))
num=$(($num1 + $num2)) # Also works
num=$((num1 + 2 + 3)) # ...
num=$[num1+num2] # Old, deprecated arithmetic expression syntax
Using the external expr utility. Note that this is only needed for really old systems.
num=`expr $num1 + $num2` # Whitespace for expr is important
For floating point:
Bash doesn't directly support this, but there are a couple of external tools you can use:
num=$(awk "BEGIN {print $num1+$num2; exit}")
num=$(python -c "print $num1+$num2")
num=$(perl -e "print $num1+$num2")
num=$(echo $num1 + $num2 | bc) # Whitespace for echo is important
You can also use scientific notation (for example, 2.5e+2).
Common pitfalls:
When setting a variable, you cannot have whitespace on either side of =, otherwise it will force the shell to interpret the first word as the name of the application to run (for example, num= or num)
num= 1 num =2
bc and expr expect each number and operator as a separate argument, so whitespace is important. They cannot process arguments like 3+ +4.
num=`expr $num1+ $num2`
Use the $(( )) arithmetic expansion.
num=$(( $num + $metab ))
See Chapter 13. Arithmetic Expansion for more information.
There are a thousand and one ways to do it. Here's one using dc (a reverse Polish desk calculator which supports unlimited precision arithmetic):
dc <<<"$num1 $num2 + p"
But if that's too bash-y for you (or portability matters) you could say
echo $num1 $num2 + p | dc
But maybe you're one of those people who thinks RPN is icky and weird; don't worry! bc is here for you:
bc <<< "$num1 + $num2"
echo $num1 + $num2 | bc
That said, there are some unrelated improvements you could be making to your script:
#!/bin/bash
num=0
metab=0
for ((i=1; i<=2; i++)); do
for j in output-$i-* ; do # 'for' can glob directly, no need to ls
echo "$j"
# 'grep' can read files, no need to use 'cat'
metab=$(grep EndBuffer "$j" | awk '{sum+=$2} END { print sum/120}')
num=$(( $num + $metab ))
done
echo "$num"
done
As described in Bash FAQ 022, Bash does not natively support floating point numbers. If you need to sum floating point numbers the use of an external tool (like bc or dc) is required.
In this case the solution would be
num=$(dc <<<"$num $metab + p")
To add accumulate possibly-floating-point numbers into num.
In Bash,
num=5
x=6
(( num += x ))
echo $num # ==> 11
Note that Bash can only handle integer arithmetic, so if your AWK command returns a fraction, then you'll want to redesign: here's your code rewritten a bit to do all math in AWK.
num=0
for ((i=1; i<=2; i++)); do
for j in output-$i-*; do
echo "$j"
num=$(
awk -v n="$num" '
/EndBuffer/ {sum += $2}
END {print n + (sum/120)}
' "$j"
)
done
echo "$num"
done
I always forget the syntax so I come to Google Search, but then I never find the one I'm familiar with :P. This is the cleanest to me and more true to what I'd expect in other languages.
i=0
((i++))
echo $i;
I really like this method as well. There is less clutter:
count=$[count+1]
#!/bin/bash
read X
read Y
echo "$(($X+$Y))"
You should declare metab as integer and then use arithmetic evaluation
declare -i metab num
...
num+=metab
...
For more information, see 6.5 Shell Arithmetic.
Use the shell built-in let. It is similar to (( expr )):
A=1
B=1
let "C = $A + $B"
echo $C # C == 2
Source: Bash let builtin command
Another portable POSIX compliant way to do in Bash, which can be defined as a function in .bashrc for all the arithmetic operators of convenience.
addNumbers () {
local IFS='+'
printf "%s\n" "$(( $* ))"
}
and just call it in command-line as,
addNumbers 1 2 3 4 5 100
115
The idea is to use the Input-Field-Separator(IFS), a special variable in Bash used for word splitting after expansion and to split lines into words. The function changes the value locally to use word-splitting character as the sum operator +.
Remember the IFS is changed locally and does not take effect on the default IFS behaviour outside the function scope. An excerpt from the man bash page,
The shell treats each character of IFS as a delimiter, and splits the results of the other expansions into words on these characters. If IFS is unset, or its value is exactly , the default, then sequences of , , and at the beginning and end of the results of the previous expansions are ignored, and any sequence of IFS characters not at the beginning or end serves to delimit words.
The "$(( $* ))" represents the list of arguments passed to be split by + and later the sum value is output using the printf function. The function can be extended to add scope for other arithmetic operations also.
#!/usr/bin/bash
#integer numbers
#===============#
num1=30
num2=5
echo $(( num1 + num2 ))
echo $(( num1-num2 ))
echo $(( num1*num2 ))
echo $(( num1/num2 ))
echo $(( num1%num2 ))
read -p "Enter first number : " a
read -p "Enter second number : " b
# we can store the result
result=$(( a+b ))
echo sum of $a \& $b is $result # \ is used to espace &
#decimal numbers
#bash only support integers so we have to delegate to a tool such as bc
#==============#
num2=3.4
num1=534.3
echo $num1+$num2 | bc
echo $num1-$num2 | bc
echo $num1*$num2 |bc
echo "scale=20;$num1/$num2" | bc
echo $num1%$num2 | bc
# we can store the result
#result=$( ( echo $num1+$num2 ) | bc )
result=$( echo $num1+$num2 | bc )
echo result is $result
##Bonus##
#Calling built in methods of bc
num=27
echo "scale=2;sqrt($num)" | bc -l # bc provides support for calculating square root
echo "scale=2;$num^3" | bc -l # calculate power
#!/bin/bash
num=0
metab=0
for ((i=1; i<=2; i++)); do
for j in `ls output-$i-*`; do
echo "$j"
metab=$(cat $j|grep EndBuffer|awk '{sum+=$2} END { print sum/120}') (line15)
let num=num+metab (line 16)
done
echo "$num"
done
Works on MacOS. bc is a command line calculator
#!/bin/bash
sum=0
for (( i=1; i<=5; i++ )); do
sum=$(echo "$sum + 1.1" | bc) # bc: if you want to use decimal
done
echo "Total: $sum"

How to test for certain characters in a file

I am currently running a script with an if statement. Before I run the script, I want to make sure the file provided as the first argument has certain characters.
If the file does not have those certain characters in certain spots then the output would be else "File is Invalid" on the command line.
For the if statement to be true, the file needs to have at least one hyphen in Field 1 line 1 and at least one comma in Field one Line one.
How would I create an if statement with perhaps a test command to validate those certain characters are present?
Thanks
Im new to Linux/Unix, this is my homework so I haven't really tried anything, only brain storming possible solutions.
function usage
{
echo "usage: $0 filename ..."
echo "ERROR: $1"
}
if [ $# -eq 0 ]
then
usage "Please enter a filename"
else
name="Yaroslav Yasinskiy"
echo $name
date
while [ $# -gt 0 ]
do
if [ -f $1 ]
then
if <--------- here is where the answer would be
starting_data=$1
echo
echo $1
cut -f3 -d, $1 > first
cut -f2 -d, $1 > last
cut -f1 -d, $1 > id
sed 's/$/:/' last > last1
sed '/last:/ d' last1 > last2
sed 's/^ *//' last2 > last3
sed '/first/ d' first > first1
sed 's/^ *//' first1 > first2
sed '/id/ d' id > id1
sed 's/-//g' id1 > id2
paste -d\ first2 last3 id2 > final
cat final
echo ''
else
echo
usage "Coult not find file $1"
fi
shift
done
fi
In answer to your direct question:
For the if statement to be true, the file needs to have at least one
hyphen in Field 1 line 1 and at least one comma in Field one Line one.
How would I create an if statement with perhaps a test command to
validate those certain characters are present?
Bash provides all the tools you need. While you can call awk, you really just need to read the first line of the file into two-variable (say a and b) and then use the [[ $a =~ regex ]] to where the regex is an extended regular expression that verifies that the first field (contained in $a) contains both a '-' and ','.
For details on the [[ =~ ]] expression, see bash(1) - Linux manual page under the section labeled [[ expression ]].
Let's start with read. When you provide two variables, read will read the first field (based on normal word-splitting given by IFS (the Internal Field Separator, default $'[ \t\n]' - space, tab, newline)). So by doing read -r a b you read the first field into a and the rest of the line into b (you don't care about b for your test)
Your regex can be ([-]+.*[,]+|[,]+.*[-]+) which is an (x|y), e.g. x OR y expression where x is [-]+.*[,]+ (one or more '-' and one or more ','), your y is [,]+.*[-]+ (one or more ',' and one or more '-'). So by using the '|' your regex will accept either a comma then zero-or-more characters and a hyphen or a hyphen and zero-or-more characters and then a comma in the first field.
How do you read the line? With simple redirection, e.g.
read -r a b < "$1"
So your conditional test in your script would look something like:
if [ -f $1 ]
then
read -r a b < "$1"
if [[ $a =~ ([-]+.*[,]+|[,]+.*[-]+) ]] # <-- here is where the ...
then
starting_data=$1
...
else
echo "File is Invalid" >&2 # redirection to 2 (stderr)
fi
else
echo
usage "Coult not find file $1"
fi
shift
...
Example Test Files
$ cat valid
dog-food, cat-food, rabbit-food
50lb 16lb 5lb
$ cat invalid
dogfood, catfood, rabbitfood
50lb 16lb 5lb
Example Use/Output
$ read -r a b < valid
if [[ $a =~ ([-]+.*[,]+|[,]+.*[-]+) ]]; then
echo "file valid"
else
echo "file invalid"
fi
file valid
and for the file without the certain characters:
$ read -r a b < invalid
if [[ $a =~ ([-]+.*[,]+|[,]+.*[-]+) ]]; then
echo "file valid"
else
echo "file invalid"
fi
file invalid
Now you really have to concentrate on eliminating the spawning of at least a dozen subshells where you call cut 3-times, sed 7-times, paste once and then cat. While it is good you are thinking through what you need to do, and getting it working, as mentioned in my comment, any time you are looping, you want to eliminate the number of subshells spawned to the greatest extent possible. I suspect as #Mig answered, awk will be the proper tool that can likely eliminate all 12 subshells are replace it with a single call to awk.
I personally would use awk for this all part since you want to test fields and create a string with concatenated fields. Awk is perfect for that.
But here is a small script which shows how you could just test your file's first line:
if [[ $(head -n 1 file.csv | awk '$1~/-/ && $1~/,/ {print "MATCH"}') == 'MATCH' ]]; then
echo "yes"
else
echo "no"
fi
It looks overkill when not doing the whole thing in awk but it works. I am sure there is a way to test only one regex, but that would involve knowing which flavour of awk you have because I think they don't all use the same regex engine. Therefore I left this out for the sake of simplicity.

Linux "echo -n" not being flushed

I have the following code:
while ...
echo -n "some text"
done | while read; do
echo "$REPLY" >> file
done
but echo works only when used without "-n" flag.
looks like when using -n, the output is not flushed/read by next while loop
How can I make sure that "some text" will be read even when not followed by EOL?
You can't distinguish between
echo -n "some text"
and
echo -n "some t"
echo -n "ext"
so you need some kind of delimiting rule. Usually EOL is used for that. read supports custom delimiter via -d or can split based on number of chars via -n or -N. For example you can make read fire on each symbol:
echo -n qwe | while read -N 1 ch; do echo $ch; done
The workaround would be (following original example):
while ...
echo -n "some text"
done | (cat && echo) | while read; do
echo "$REPLY" >> file
done
This will append EOL to the test stream & allow read to read it.
The side effect will be an additional EOL at the end of stream.
You can start with defining your own delimiter:
while :; do
echo -n "some text"
sleep 2
done | while read -d' ' reply; do
echo "-$reply-"
done
This prints:
-some-
-textsome-
-textsome-
For an email perhaps it makes sense to use . as a delimiter, but you need to decide on some tokenization scheme.
You can make read read one char a time, but should add something for reading special characters (newlines, spaces): IFS=.
I want to show that I really capture the characters, so I will uppercase the replies.
i=0
while (( i++<5 )) ; do
echo -n "some text $i. "
sleep 1;
done | while IFS= read -rn1 reply; do
printf "%s" "${reply^^}"
done
This solution has one feature: You will not see any newlines.
When you want to see them too, you need to fix this with
i=1
while (( i++<5 )) ; do
echo -n "some text $i.
second line."
sleep 1;
done | while IFS= read -rn1 reply; do
if (( ${#reply} == 0 )); then
echo
else
printf "%s" "${reply^^}"
fi
done

Give file and its line number to convert the hexadecimal number in decimal in bash

I have a text file of more than 10,000 lines. In this file I have a line number 713 contains three hexadecimal number like, 7E 42 C0. I need to convert this to decimal. My code works well if I give only this number in a text file. But, I want to give the file and the line number of this file
Like in the following:
while read p q r;
do
printf "%d %d %d\n" 0x$p 0x$q 0x$r;
done < $(sed -n '713p' run21.txt);
But it didn't work and give me an error of ambiguous redirect. Please, let me know any solution.
You are on the right track, to convert from hex to dec use bc with defined ibase and obase, e.g.
$ echo "obase=10; ibase=16; $(sed -n 713p run21.txt)" | bc
Whatever hex value is at line 713 will be output as decimal. You can use read -r a b c to read the 3 values and convert each in a similar manner. Or you can remove the whitespace between the 3 hex values and convert it into a single decimal value depending on your requirements. For example:
hexval=""
for i in $(sed -n 713p run21.txt); do
hexval="$hexval$i"
done
echo "obase=10; ibase=16; $hexval" | bc
Output
8274624
To convert each separately, you can do:
for i in $(sed -n 713p run21.txt); do
printf " %d" "$(echo "obase=10; ibase=16; $i" | bc)"
done
printf "\n"
Output
126 66 192
As another step in making the conversion more robust, you can (and should) add validation of all values read as input. You can do much more validation as to the type of characters read into each variable, but you should at least validate you read AND filled the number of values you expected. Putting that together, and utilizing read to read and validate all 3 values, you could do something like the following.
The script takes the line_to_read as its first mandatory argument and the filename to read from as its second optional argument (it will read from stdin by default):
#!/bin/bash
test -z "$1" && { ## validate line number given
printf "error: insufficient input.\nusage: %s line [file (stdin)]\n" "${0##*/}"
exit 1
}
test "$1" -eq "$1" &>/dev/null || { ## test $1 is an integer value
printf "error: first parameter not an integer '%s'\n" "$1"
exit 1
}
fn="$2" ## read from filename as 2nd parameter (or by default from stdin)
test -f "$fn" || fn=/dev/stdin
read -r a b c < <(sed -n "${1}p" "$fn") ## read each of the values
test -n "$a" -a -n "$b" -a -n "$c" || { ## validate all 3 vars filled
printf "error: less than 3 values on line '%s',\n" "$1"
exit 1
}
printf "input : %3x %3x %3x\n" "0x$a" "0x$b" "0x$c" ## output original line
## output line converted from hex to decimal
printf "output: %3d %3d %3d\n" $(echo "obase=10; ibase=16; $a" | bc) \
$(echo "obase=10; ibase=16; $b" | bc) $(echo "obase=10; ibase=16; $c" | bc)
Example Input
$ cat dat/3hex.txt
7E 42 C0
Example Use/Output
$ bash 3hexcvt.sh 1 <dat/3hex.txt
input : 7e 42 c0
output: 126 66 192
perhaps more straightforward this way...
read p q r < <(sed -n '713{p;q}' file); printf "%d %d %d\n" 0x$p 0x$q 0x$r;
#David
I did this
while read -r p q r;
do
#printf "%d %d %d\n" 0x$p 0x$q 0x$r;
#printf "%d %d %d\n"
echo "obase=10; ibase=16; $(sed -n 713p run21.txt)" | bc ;
done
But there is no output.

bash: convert string to int & if int > #

I would like to have a bash script that checks if a file has more than # amount of lines but i have not yet got it working right and I'm not so sure on how to do it.
I've never used bash before.
right now i use: linesStr=$(cat log | wc -l) to get the amount of lines in the file (expect it to be a string). when echo'ing it gives me the number 30 which is correct.
but since its most likely a string it doesnt do the if-statement, so i need to have linesStr converted into a int called linesInt.
I also have the feeling the if-statement itself is not done correctly either.
#!/bin/bash
linesStr=$(cat log | wc -l)
echo $linesStr
if [$linesStr > 29]
then echo "log file is bigger than 29 lines"
#sed -i 1d log
fi
I would appreciate if anyone can give me a simple beginners solution.
No need for cat.
Lack of spaces around [ and ].
Use a numeric comparison operator instead of the redirect operator.
Here is a working script.
#!/bin/bash
linesStr=$( wc -l < log )
if [[ "$linesStr" -gt "29" ]]; then
echo Foo
fi
your if block of code is wrong if [$linesStr > 29] there should be a space after [ and before ]
#!/bin/bash
linesStr=$(wc -l < log )
echo $linesStr
if [[ $lineStr -gt 29 ]];then
echo "log file is bigger than 29 lines"
fi
it is advisable that you always use [[ ]] with an if statement rather than using [ ]. Whenever you want to compare integers dont use > or <, use -gt -ge -lt -le. And if you want to do any form of mathematical comparison it is advisable that you use (( )).
(( lineStr > 29 )) && {
# do stuff
}
you should also note that you don't need the bash comparison operators or getting the value of a variable with $ when using (( ))
There are no string or integer types to convert. The problem is that you're using the wrong comparison operator. For numeric comparison use if [ $linesStr -gt 29 ]. Read man bash section CONDITIONAL EXPRESSIONS for available operators.
(( $(wc -l < log) > 29 )) && echo too long

Resources