Bash IFS, getting zapped in function call - linux

Platform CentOS Linux release 7.6.1810, working in bash.
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
This is an idiom I've seen recommended for parsing text in bash in general and in particular for returning multiple values from a function.
IFS=":" read A B <<< $(echo ONE:TWO)
I'm getting unexpected behaviour when I call a function, yyy in the example here
IFS=":" read Y1 Y2 <<< $(yyy)
where yyy itself also wants to do a similar call.
The effect is that that within yyy() even though I explicitly specify the IFS
IFS=":" read C1 C2 <<< $( echo "A:B" )
The fields are parsed, but both values are assigned to C1, it gets the value "A B". If the function is called in isolation it works as expected.
This is a test case, distilled down from a much larger script. I want to know what is happening with IFS here. In the failure case (the second example below) setting IFS=":" in the caller somehow cause the result fields to be aggregated. The first and third calls to yyy() below work as expected, output shown after the code.
#!/bin/bash
debug() { echo "$1" 1>&2 ; }
yyy() {
debug "in yyy"
# why are the two values assigned to A here if the caller specified IFS?
IFS=":" read A B <<< $(echo ONE:TWO)
debug "A=$A"
debug "B=$B"
echo "$A:$B"
}
# this works as expected
read Y1 Y2 <<< $(yyy)
echo -e "===\n"
# this cause the read in yyy() to aggregate
IFS=":" read Y1 Y2 <<< $(yyy)
echo -e "===\n"
# This is a workaround that enables yyy() to work correctly
# But why do I need to do this?
OUT="$(yyy)"
IFS=":" read Y1 Y2 <<< $(echo $OUT)
This is the output
in yyy
A=ONE B=TWO
===
in yyy
A=ONE TWO B=
===
in yyy
A=ONE B=TWO
Note that in the second case A gets the value ONE TWO

This seems to be a bug in bash-4.2 as discussed here, IFS incorrectly splitting herestrings in bash 4.2. Should work on the versions above that.
These are the results on the same version as you have - GNU bash, version 4.2.46(2). When I ran the function yyy in debug mode ( by setting set -x in prompt ).
++ IFS=:
++ read A B
+++ echo ONE:TWO
++ debug 'A=ONE TWO'
++ echo 'A=ONE TWO'
A=ONE TWO
++ debug B=
++ echo B=
B=
++ echo 'ONE TWO:'
The above is snippet of the output from the debug mode output. As you can see when the echo ONE:TWO is printed as a result of the command substitution, no word splitting is expected to happen because the line doesn't contain any character of the default IFS value (space/tab or a newline)
So you would expect reading the the whole string with IFS=: expected to split the string and put the values in the constituent variables A and B, but somehow the : character is lost and a string ONE TWO is stored as the first variable value.
Look at the output of the function execution in GNU bash, version 4.4.12(1) which exhibits the right behavior.
++ IFS=:
++ read A B
+++ echo ONE:TWO
++ debug A=ONE
++ echo A=ONE
A=ONE
++ debug B=TWO
++ echo B=TWO
B=TWO
++ echo ONE:TWO
There have been lot of IFS related bugs up to version 4.4.0 bash/CHANGES. So a personal recommendation is to upgrade your bash version to a more recent stable one. Also see Trying to split a string into two variables
Similar bug on version 4.4.0(1)-release
You would expect the ONE:TWO to be unmodified when the $(..) is expanded because for reasons mentioned earlier. But here too the delimit character is lost and the variable A is set to ONE TWO
IFS=":" read A B <<< $(echo ONE:TWO)
echo "$A"
ONE TWO
Surprisingly the above code works on 4.2.46(2), which means the 4.4.0(1) broke a functionality which used to work in the earlier releases.

Related

Passing string with a space between words through various function layers

I have two functions in Bash. One is a generic run function, that accepts an input and evaluates it, while printing the command, and testing the exit code. This is used in a large script to ensure each command executes successfully before continuing.
The second one is a complex function, that is doing some Git history parsing. The problematic line is the only one shown.
I am calling this function from a for-loop, that iterates over a list of terms to search. The issue is that spaces are not being handled correctly, when between other words. I have tried running my script though shell-checking websites, and all of the suggestions seem to break my code.
function run() {
echo "> ${1}"
eval "${1}"
# Test exit code of the eval, and exit if non-zero
}
function searchCommitContents() {
run 'result=$(git log -S'"${1}"' --format=format:%H)'
# Do something with result, which is a list of matching SHA1 hashes for the commits
echo "${result}"
}
# Main
declare -a searchContents=('foo' 'bar' ' foo ' 'foo bar')
for i in "${searchContents[#]}"
do
searchCommitContents "${i}"
done
Here is the output I get:
> result=$(git log -Sfoo --format=format:%H)
<results>
> result=$(git log -Sbar --format=format:%H)
<results>
> result=$(git log -S foo --format=format:%H)
<results>
> result=$(git log -Sfoo bar --format=format:%H)
fatal: ambiguous argument 'bar': unknown revision of path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
I tried to add additional single and double-quotes to various areas of the code, such that the 'foo bar' string would not resolve to two different words. I also tried adding an escape to the dollar sign, like so: -s'"\${1}"' based on other questions on this site.
Why are you printing result=$(? It's an internal variable, it can be anything, there is no need for it in logs.
Print the command that you are executing, not the variable name.
run() {
echo "+ $*" >&2
"$#"
}
searchCommitContents() {
local result
result=$(run git log -s"${1}" --format=format:%H)
: do stuff to "${result}"
echo "$result"
}
issue with an input that has a space in the middle.
If you want quoted string, use printf "%q" or ${...#Q} for newer Bash, but I don't really enjoy both quoting methods and just use $*. I really like /bin/printf from GNU coreutils, but it's a separate process... while ${..#Q} is the fastest, it's (still) not enough portable for me (I have some old Bash around).
# compare
$ set -- a 'b c' d
$ echo "+ $*" >&2
+ a b c d
$ echo "+$(printf " %q" "$#")" >&2
+ a b\ \ c d
$ echo "+" "${##Q}" >&2
+ 'a' 'b c' 'd'
$ echo "+$(/bin/printf " %q" "$#")" >&2
+ a 'b c' d
See these lines:
> result=$(git log -Sfoo bar --format=format:%H)
fatal: ambiguous argument 'bar': unknown revision of path not in the working tree.
Specifically this: -Sfoo bar. It should be -S"foo bar" or -S "foo bar". Because to pass an argument with spaces, we need to quote the argument. But, each time the argument pass through a command/function layer, one layer of quote ('', "") is extracted. So, we need to nest the quote.
So in this line:
declare -a searchContents=('foo' 'bar' ' foo ' 'foo bar')
change 'foo bar' to '"foo bar"' or "'foo bar'" or "\"foo bar\"".
This is a case of 2 layers nested quotes. The more the layer, the trickier it gets. Here's an example of 4 layers quotes I once did.

Shell Script: value too great for base (error token is "16#?")

I am new to shell script. I am working with Hex values and writing a simple script for substraction. Here is my script:
#!/bin/bash
var1=“0x0001”
var2=“0x0005”
var3=“$(( 16#$var2 - 16#$var1 ))”
echo “Diference $var3”
I am getting this error :
line 6: 16#?: value too great for base (error token is "16#?")
Could you please let me know where my mistake is?
$ var1=0x0001
$ var2=0x0005
$ var3=$(( $var2 - $var1 ))
$ echo "Diference $var3"
Diference 4
Assign the hex values without double quotes(i.e not as strings).
Since you have already put a 0x there is no need for 16#
To conver the answer back to hex you can use:
printf '%x' $num
Here is an example:
$ var1=0x19
$ var2=0xA
$ var3=$(( $var1 - $var2 ))
$ echo $var3
15
$ printf '%x\n' $var3
f
$ var3=$(printf '%x' $var3)
$ echo $var3
f
16# and 0x are redundant, and mutually exclusive. The problem is that, due to the 16#, Bash thinks the x is trying to be a digit in a base-16 number (whereas it's only valid in base 34 or higher). Just drop either the 16# or the 0x, and it'll work.

Printing IFS values from file

Script:
#!/bin/ksh
FILENAME=$1
while read RECORD VALUE
do
echo ${RECORD} ${VALUE} "X"
done <"$FILENAME"
input file:
A 1
B 2
The output of script:
X1
X2
If I remove from echo "x", e.g.
echo ${RECORD} ${VALUE}
I am getting
A 1
B 2
what is wrong?
Update:
If I do
echo "X" ${RECORD} ${VALUE}
it prints correctly:
X A 1
X B 2
and :
echo ${RECORD} "X"
also prints correctly, so i am guessing the issues is with VALUE that maybe contains return carriage symbol (as input file was created on windows)
adding this inside the loop:
VALUE=`echo $VALUE| tr -d '\r'`
solved the issue, if you have a better solution you are more than welcome.
There is a parameter expansion operator you can use to remove a character from the end of a value, if it is present.
VALUE=${VALUE%$'\r'}
This is handled in-shell, without needing to start a new process.

String Concatenation error in Cygwin bash shell

I am having problems concatenate two strings in BASH (I am using Cygwin)
When I am doing it step by step in the cygwin window, it works.
i.e by defining dt=2012-12-31 and c=.txt explicitly and then concatenating in filename=${dt}${c}.
It doesn't seem to work when i am running it through my script where these variables are defined by cutting and assigning values from content of a file.
Though the variables are assigned with the same values as above, the concatenation in this case doesn't work.
instead of 2012-12-31.txt i am getting .txt-12-31 as result.
The code is:
for x in {0..11}
do
IFS=$'\n'
filename=date_list.txt
file=($(<"$filename"))
IFS=$'\t\n'
dt=${file[$x]}
echo $dt
for y in {0..85}
do
IFS=$'\n'
filename=SQL_Mnemonics.txt
file=($(<"$filename"))
IFS=$'\t\n'
Mn=${file[$y]}
for k in {3..502}
do
IFS=$'\n'
c=.txt
filename=${dt}${c}
file=($(<"$filename"))
IFS=$'\t\n'
echo ${file[$k]} > temp_file.txt
cusip=`cut -c11-19 temp_file.txt`
result=$(sh ctest.sh $Mn, $dt, $cusip)
echo "$result" > tmp1.txt
t1=`cut -c18-40 tmp1.txt`
echo $t1 | sed 's/[[:space:]]//g' > temp_file.txt
cat tst.txt | sed 's/-----//g' >> ForFame/${Mn}.${dt}.txt
done
done
done

KornShell Printf - Padding a string

I'm attempting to write a KornShell (ksh) function that uses printf to pad a string to a certain width.
Examples:
Call
padSpaces Hello 10
Output
'Hello '
I currently have:
padSpaces(){
WIDTH=$2
FORMAT="%-${WIDTH}.${WIDTH}s"
printf $FORMAT $1
}
Edit: This seems to be working, in and of itself, but when I assign this in the script it seems to lose all but the first space.
TEXT=`padSpaces "TEST" 10`
TEXT="${TEXT}A"
echo ${TEXT}
Output:
TEST A
I'm also open to suggestions that don't use printf. What I'm really trying to get at is a way to make a fixed width file from ksh.
Your function works fine for me. Your assignment won't work with spaces around the equal sign. It should be:
SOME_STRING=$(padSpaces TEST 10)
I took the liberty of replacing the backticks, too.
You don't show how you are using the variable or how you obtain the output you showed. However, your problem may be that you need to quote your variables. Here's a demonstration:
$ SOME_STRING=$(padSpaces TEST 10)
$ sq=\'
$ echo $sq$SOME_STRING$sq
'TEST '
$ echo "$sq$SOME_STRING$sq"
'TEST '
Are you aware that you define a function called padSpaces, yet call one named padString? Anyway, try this:
padString() {
WIDTH=$2
FORMAT="%-${WIDTH}s"
printf $FORMAT $1
}
Or, the more compact:
padString() {
printf "%-${2}s" $1
}
The minus sign tells printf to left align (instead of the default right alignment). As the manpage states about the command printf format [ arg ... ],
The arguments arg are printed on standard output in accordance with the
ANSI-C formatting rules associated with the format string format.
(I just installed ksh to test this code; it works on my machineTM.)

Resources