I have a bash script that arrives like:
SCRIPT=$(curl .... | parsing...)
echo $SCRIPT > myfile
But when I try to echo it in a file, some parts get evaluated. (Variables are substituted if any are defined, the * character is replaced by all files in the working directory, etc...)
Can I prevent bash from evaluating any content of a variable, while still echoing?
Yes, use double quotes for that. I'll demonstrate:
$ x='*'
$ echo $x
..list of files..
$ echo '$x'
$x
$ echo "$x"
*
Related
At the command prompt ($) I execute the commands:
$ stupid="-a hello"
$ echo $stupid
Echo produces:
-a hello
At the command prompt ($) I execute the commands:
$ stupid="-e hello"
$ echo $stupid
Echo produces:
hello
Why did the "-e" disappear?
Since $stupid is unquoted, it gets processed as flag of echo and enables interpretation of backslash escapes.
If you did:
$ stupid="-e hello"
$ echo "$stupid"
You would see value of $stupid echoed in its entirety:
-e hello
Because the resulting command after variable expansion would be
echo "-e hello"
In your case however, $stupid is first expanded and then then the command is executed as:
echo -e hello
It may become even more obvious if your variable value actually included an escaped character such as: foo="-e \ttext", try both echo $foo and echo "$foo" and see what happens.
Bottom line: double quoting your strings and or variable is usually the prudent thing to do.
I’ve noticed that we can use curly braces to make some of the commands much shorter as it is evaluated into list of arguments.
Input:
echo a{,b,c}
Output:
a ab ac
How do I force the same behaviour when the arguments are passed from the file?
Input:
cat file.txt | xargs echo
Output:
a{,b,c}
Expected output - same as in the previous example.
That {} expansion is a bash / zsh feature, as such then you need to explicitly run it thru any of these shells, in your case would be (using -I<STRING> to let xargs replace it in the string before running it):
cat file.txt |xargs -I# bash -c 'echo #'
xargs calls the echo as found in the $PATH, not the shell's builtin echo.
check the list of bash expansions: brace expansion happens first, so it won't get a chance to expand in that pipeline.
You'll have to do something like
while read -r line; do eval echo "$line"; done < file.txt
which exposes you to all kinds of nasty attacks if someone puts something malicious in that file.
Other than asking why would you want to do this... I offer the following:
add the string to a file:
echo 'a{,b,c}' > /tmp/foo
put the string in a variable:
export thing=`cat /tmp/foo`
eval the string:
eval $thing
If you had a bunch of these in a file then run the file through a loop and eval the loop value:
echo 'a{,b,c}' >> /tmp/foo
echo 'a{,b,c}' >> /tmp/foo
echo 'a{,b,c}' >> /tmp/foo
for i in `cat /tmp/foo`; do eval echo $i; done
I am creating a Bash script which reads some other environment variables:
echo "Exporting configuration variables..."
while IFS="=" read -r k v; do
key=$k
value=$v
if [[ ${#key} > 0 && ${#value} > 0 ]]; then
export $key=$value
fi
done < $HOME/myfile
and have the variable:
$a=$b/c/d/e
and want to call $a as in:
cp myOtherFile $a
The result for the destination folder for the copy is "$b/c/d/e", and an error is shown:
"$b/c/d/e" : No such file or directory
because it is interpreted literally as a folder path.
Can this path be reinterpreted before being used in the cp command?
You need eval to do this :
$ var=foo
$ x=var
$ eval $x=another_value
$ echo $var
another_value
I recommend you this doc before using eval : http://mywiki.wooledge.org/BashFAQ/048
And a safer approach is to use declare instead of eval:
declare "$x=another_value"
Thanks to chepner 2 for the latest.
It sounds like you want $HOME/myfile to support Bash notations, such as parameter-expansion. I think the best way to do that is to modify $HOME/myfile to be, in essence, a Bash script:
export a=$b/c/d/e
and use the source builtin to run it as part of the current Bash script:
source $HOME/myfile
... commands ...
cp myOtherFile "$a"
... commands ...
try this
cp myOtherFile `echo $a`
I am trying to work out how to make bash (force?) expand variables in a string (which was loaded from a file).
I have a file called "something.txt" with the contents:
hello $FOO world
I then run
export FOO=42
echo $(cat something.txt)
this returns:
hello $FOO world
It didn't expand $FOO even though the variable was set. I can't eval or source the file - as it will try and execute it (it isn't executable as it is - I just want the string with the variables interpolated).
Any ideas?
I stumbled on what I think is THE answer to this question: the envsubst command:
echo "hello \$FOO world" > source.txt
export FOO=42
envsubst < source.txt
This outputs: hello 42 world
If you would like to continue work on the data in a file destination.txt, push this back to a file like this:
envsubst < source.txt > destination.txt
In case it's not already available in your distro, it's in the
GNU package gettext.
#Rockallite
I wrote a little wrapper script to take care of the '$' problem.
(BTW, there is a "feature" of envsubst, explained at
https://unix.stackexchange.com/a/294400/7088
for expanding only some of the variables in the input, but I
agree that escaping the exceptions is much more convenient.)
Here's my script:
#! /bin/bash
## -*-Shell-Script-*-
CmdName=${0##*/}
Usage="usage: $CmdName runs envsubst, but allows '\$' to keep variables from
being expanded.
With option -sl '\$' keeps the back-slash.
Default is to replace '\$' with '$'
"
if [[ $1 = -h ]] ;then echo -e >&2 "$Usage" ; exit 1 ;fi
if [[ $1 = -sl ]] ;then sl='\' ; shift ;fi
sed 's/\\\$/\${EnVsUbDolR}/g' | EnVsUbDolR=$sl\$ envsubst "$#"
Many of the answers using eval and echo kind of work, but break on various things, such as multiple lines, attempting to escaping shell meta-characters, escapes inside the template not intended to be expanded by bash, etc.
I had the same issue, and wrote this shell function, which as far as I can tell, handles everything correctly. This will still strip only trailing newlines from the template, because of bash's command substitution rules, but I've never found that to be an issue as long as everything else remains intact.
apply_shell_expansion() {
declare file="$1"
declare data=$(< "$file")
declare delimiter="__apply_shell_expansion_delimiter__"
declare command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"
eval "$command"
}
For example, you can use it like this with a parameters.cfg which is really a shell script that just sets variables, and a template.txt which is a template that uses those variables:
. parameters.cfg
printf "%s\n" "$(apply_shell_expansion template.txt)" > result.txt
In practice, I use this as a sort of lightweight template system.
you can try
echo $(eval echo $(cat something.txt))
You don't want to print each line, you want to evaluate it so that Bash can perform variable substitutions.
FOO=42
while read; do
eval echo "$REPLY"
done < something.txt
See help eval or the Bash manual for more information.
Another approach (which seems icky, but I am putting it here anyway):
Write the contents of something.txt to a temp file, with an echo statement wrapped around it:
something=$(cat something.txt)
echo "echo \"" > temp.out
echo "$something" >> temp.out
echo "\"" >> temp.out
then source it back in to a variable:
RESULT=$(source temp.out)
and the $RESULT will have it all expanded. But it seems so wrong !
Single line solution that doesn't need temporary file :
RESULT=$(source <(echo "echo \"$(cat something.txt)\""))
#or
RESULT=$(source <(echo "echo \"$(<something.txt)\""))
If you only want the variable references to be expanded (an objective that I had for myself) you could do the below.
contents="$(cat something.txt)"
echo $(eval echo \"$contents\")
(The escaped quotes around $contents is key here)
If something.txt has only one line, a bash method, (a shorter version of Michael Neale's "icky" answer),
using process & command substitution:
FOO=42 . <(echo -e echo $(<something.txt))
Output:
hello 42 world
Note that export isn't needed.
If something.txt has one or more lines, a GNU sed evaluate method:
FOO=42 sed 's/"/\\\"/g;s/.*/echo "&"/e' something.txt
Following solution:
allows replacing of variables which are defined
leaves unchanged variables placeholders which are not defined. This is especially useful during automated deployments.
supports replacement of variables in following formats:
${var_NAME}
$var_NAME
reports which variables are not defined in environment and returns error code for such cases
TARGET_FILE=someFile.txt;
ERR_CNT=0;
for VARNAME in $(grep -P -o -e '\$[\{]?(\w+)*[\}]?' ${TARGET_FILE} | sort -u); do
VAR_VALUE=${!VARNAME};
VARNAME2=$(echo $VARNAME| sed -e 's|^\${||g' -e 's|}$||g' -e 's|^\$||g' );
VAR_VALUE2=${!VARNAME2};
if [ "xxx" = "xxx$VAR_VALUE2" ]; then
echo "$VARNAME is undefined ";
ERR_CNT=$((ERR_CNT+1));
else
echo "replacing $VARNAME with $VAR_VALUE2" ;
sed -i "s|$VARNAME|$VAR_VALUE2|g" ${TARGET_FILE};
fi
done
if [ ${ERR_CNT} -gt 0 ]; then
echo "Found $ERR_CNT undefined environment variables";
exit 1
fi
foo=45
file=something.txt # in a file is written: Hello $foo world!
eval echo $(cat $file)
$ eval echo $(cat something.txt)
hello 42 world
$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.
envsubst is a great solution (see LenW's answer) if the content you're substituting is of "reasonable" length.
In my case, I needed to substitute in a file's content to replace the variable name. envsubst requires that the content be exported as environment variables and bash has a problem when exporting environment variables that are more than a megabyte or so.
awk solution
Using cuonglm's solution from a different question:
needle="doc1_base64" # The "variable name" in the file. (A $ is not needed.)
needle_file="doc1_base64.txt" # Will be substituted for the needle
haystack=$requestfile1 # File containing the needle
out=$requestfile2
awk "BEGIN{getline l < \"${needle_file}\"}/${needle}/{gsub(\"${needle}\",l)}1" $haystack > $out
This solution works for even large files.
expenv () {
LF=$'\n'
echo "cat <<END_OF_TEXT${LF}$(< "$1")${LF}END_OF_TEXT" | bash
return $?
}
expenv "file name"
The following works: bash -c "echo \"$(cat something.txt)"\"
#!/bin/bash
hello()
{
SRC=$1
DEST=$2
for IP in `cat /opt/ankit/configs/machine.configs` ; do
echo $SRC | grep '*' > /dev/null
if test `echo $?` -eq 0 ; then
for STAR in $SRC ; do
echo -en "$IP"
echo -en "\n\t ARG1=$STAR ARG2=$2\n\n"
done
else
echo -en "$IP"
echo -en "\n\t ARG1=$SRC ARG2=$DEST\n\n"
fi
done
}
hello $1 $2
The above is the shell script which I provide source (SRC) & desitnation (DEST) path. It worked fine when I did not put in a SRC path with wild card ''. When I run this shell script and give ''.pdf or '*'as follows:
root#ankit1:~/as_prac# ./test.sh /home/dev/Examples/*.pdf /ankit_test/as
I get the following output:
192.168.1.6
ARG1=/home/dev/Examples/case_Contact.pdf ARG2=/home/dev/Examples/case_howard_county_library.pdf
The DEST is /ankit_test/as but DEST also get manupulated due to '*'. The expected answer is
ARG1=/home/dev/Examples/case_Contact.pdf ARG2=/ankit_test/as
So, if you understand what I am trying to do, please help me out to solve this BUG.
I'll be grateful to you.
Thanks in advance!!!
I need to know exactly how I use '*.pdf' in my program one by one without disturbing DEST.
Your script needs more work.
Even after escaping the wildcard, you won't get your expected answer. You will get:
ARG1=/home/dev/Examples/*.pdf ARG2=/ankit__test/as
Try the following instead:
for IP in `cat /opt/ankit/configs/machine.configs`
do
for i in $SRC
do
echo -en "$IP"
echo -en "\n\t ARG1=$i ARG2=$DEST\n\n"
done
done
Run it like this:
root#ankit1:~/as_prac# ./test.sh "/home/dev/Examples/*.pdf" /ankit__test/as
The shell will expand wildcards unless you escape them, so for example if you have
$ ls
one.pdf two.pdf three.pdf
and run your script as
./test.sh *.pdf /ankit__test/as
it will be the same as
./test.sh one.pdf two.pdf three.pdf /ankit__test/as
which is not what you expect. Doing
./test.sh \*.pdf /ankit__test/as
should work.
If you can, change the order of the parameters passed to your shell script as follows:
./test.sh /ankit_test/as /home/dev/Examples/*.pdf
That would make your life a lot easier since the variable part moves to the end of the line. Then, the following script will do what you want:
#!/bin/bash
hello()
{
SRC=$1
DEST=$2
for IP in `cat /opt/ankit/configs/machine.configs` ; do
echo -en "$IP"
echo -en "\n\t ARG1=$SRC ARG2=$DEST\n\n"
done
}
arg2=$1
shift
while [[ "$1" != "" ]] ; do
hello $1 $arg2
shift
done
You are also missing a final "done" to close your outer for loop.
OK, this appears to do what you want:
#!/bin/bash
hello() {
SRC=$1
DEST=$2
while read IP ; do
for FILE in $SRC; do
echo -e "$IP"
echo -e "\tARG1=$FILE ARG2=$DEST\n"
done
done < /tmp/machine.configs
}
hello "$1" $2
You still need to escape any wildcard characters when you invoke the script
The double quotes are necessary when you invoke the hello function, otherwise the mere fact of evaluating $1 causes the wildcard to be expanded, but we don't want that to happen until $SRC is assigned in the function
Here's what I came up with:
#!/bin/bash
hello()
{
# DEST will contain the last argument
eval DEST=\$$#
while [ $1 != $DEST ]; do
SRC=$1
for IP in `cat /opt/ankit/configs/machine.configs`; do
echo -en "$IP"
echo -en "\n\t ARG1=$SRC ARG2=$DEST\n\n"
done
shift || break
done
}
hello $*
Instead of passing only two parameters to the hello() function, we'll pass in all the arguments that the script got.
Inside the hello() function, we first assign the final argument to the DEST var. Then we loop through all of the arguments, assigning each one to SRC, and run whatever commands we want using the SRC and DEST arguments. Note that you may want to put quotation marks around $SRC and $DEST in case they contain spaces. We stop looping when SRC is the same as DEST because that means we've hit the final argument (the destination).
For multiple input files using a wildcard such as *.txt, I found this to work perfectly, no escaping required. It should work just like a native bash app like "ls" or "rm." This was not documented just about anywhere so since I spent a better part of 3 days trying to figure it out I decided I should post it for future readers.
Directory contains the following files (output of ls)
file1.txt file2.txt file3.txt
Run script like
$ ./script.sh *.txt
Or even like
$ ./script.sh file{1..3}.txt
The script
#!/bin/bash
# store default IFS, we need to temporarily change this
sfi=$IFS
#set IFS to $'\n\' - new line
IFS=$'\n'
if [[ -z $# ]]
then
echo "Error: Missing required argument"
echo
exit 1
fi
# Put the file glob into an array
file=("$#")
# Now loop through them
for (( i=0 ; i < ${#file[*]} ; i++ ));
do
if [ -w ${file[$i]} ]; then
echo ${file[$i]} " writable"
else
echo ${file[$i]} " NOT writable"
fi
done
# Reset IFS to its default value
IFS=$sfi
The output
file1.txt writable
file2.txt writable
file3.txt writable
The key was switching the IFS (Internal Field Separator) temporarily. You have to be sure to store this before switching and then switch it back when you are done with it as demonstrated above.
Now you have a list of expanded files (with spaces escaped) in the file[] array which you can then loop through. I like this solution the best, easiest to program for and easiest for the users.
There's no need to spawn a shell to look at the $? variable, you can evaluate it directly.
It should just be:
if [ $? -eq 0 ]; then
You're running
./test.sh /home/dev/Examples/*.pdf /ankit_test/as
and your interactive shell is expanding the wildcard before the script gets it. You just need to quote the first argument when you launch it, as in
./test.sh "/home/dev/Examples/*.pdf" /ankit_test/as
and then, in your script, quote "$SRC" anywhere where you literally want the things with wildcards (ie, when you do echo $SRC, instead use echo "$SRC") and leave it unquoted when you want the wildcards expanded. Basically, always put quotes around things which might contain shell metacharacters unless you want the metacharacters interpreted. :)