I somehow can't figure out how to start a self-written programm with arguments from a shell script. If I'm in a folder whose parent folder contains the binary, then I can start the program with
$ ../binary --opt1 arg1 --opt2 arg2
Now, say the arguments and options are listed in a file args in the current folder.
args.txt:
--opt1 arg1 --opt2 arg2
If I'm trying to execute the binary from a shell script in the current folder like:
$ ./script.sh args.txt
script.sh:
#!/bin/bash
if [ $# != 1 ]
then
exit 1;
fi
params=$(<"$1")
../binary "$params"
# ../binary <<<"$params" doesn't work either.
How can I make this work?
Edit (updated script):
#!/bin/bash
if [ $# != 1 ]
then
exit 1;
fi
params=$(<"$1")
START=$(date +%s)
../binary "$params"
# ../binary <<<"$params" doesn't work either.
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test took $DIFF seconds"
Use command substituion:
$ ./script.sh args.txt
content of ./script.sh
#!/bin/bash
../binary $(< "$1")
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting. The command substitution $(cat file) can be replaced by the equivalent but faster $(< file).
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or \. The first back‐quote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.
Command substitutions may be nested. To nest when using the backquoted form, escape the inner backquotes with backslashes. If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.
Related
Hi… Need a little help here…
I tried to emulate the DOS' dir command in Linux using bash script. Basically it's just a wrapped ls command with some parameters plus summary info. Here's the script:
#!/bin/bash
# default to current folder
if [ -z "$1" ]; then var=.;
else var="$1"; fi
# check file existence
if [ -a "$var" ]; then
# list contents with color, folder first
CMD="ls -lgG $var --color --group-directories-first"; $CMD;
# sum all files size
size=$(ls -lgGp "$var" | grep -v / | awk '{ sum += $3 }; END { print sum }')
if [ "$size" == "" ]; then size="0"; fi
# create summary
if [ -d "$var" ]; then
folder=$(find $var/* -maxdepth 0 -type d | wc -l)
file=$(find $var/* -maxdepth 0 -type f | wc -l)
echo "Found: $folder folders "
echo " $file files $size bytes"
fi
# error message
else
echo "dir: Error \"$var\": No such file or directory"
fi
The problem is when the argument contains an asterisk (*), the ls within the script acts differently compare to the direct ls command given at the prompt. Instead of return the whole files list, the script only returns the first file. See the video below to see the comparation in action. I don't know why it behaves like that.
Anyone knows how to fix it? Thank you.
Video: problem in action
UPDATE:
The problem has been solved. Thank you all for the answers. Now my script works as expected. See the video here: http://i.giphy.com/3o8dp1YLz4fIyCbOAU.gif
The asterisk * is expanded by the shell when it parses the command line. In other words, your script doesn't get a parameter containing an asterisk, it gets a list of files as arguments. Your script only works with $1, the first argument. It should work with "$#" instead.
This is because when you retrieve $1 you assume the shell does NOT expand *.
In fact, when * (or other glob) matches, it is expanded, and broken into segments by $IFS, and then passed as $1, $2, etc.
You're lucky if you simply retrieved the first file. When your first file's path contains spaces, you'll get an error because you only get the first segment before the space.
Seriously, read this and especially this. Really.
And please don't do things like
CMD=whatever you get from user input; $CMD;
You are begging for trouble. Don't execute arbitrary string from the user.
Both above answers already answered your question. So, i'm going a bit more verbose.
In your terminal is running the bash interpreter (probably). This is the program which parses your input line(s) and doing "things" based on your input.
When you enter some line the bash start doing the following workflow:
parsing and lexical analysis
expansion
brace expansion
tidle expansion
variable expansion
artithmetic and other substitutions
command substitution
word splitting
filename generation (globbing)
removing quotes
Only after all above the bash
will execute some external commands, like ls or dir.sh... etc.,
or will do so some "internal" actions for the known keywords and builtins like echo, for, if etc...
As you can see, the second last is the filename generation (globbing). So, in your case - if the test* matches some files, your bash expands the willcard characters (aka does the globbing).
So,
when you enter dir.sh test*,
and the test* matches some files
the bash does the expansion first
and after will execute the command dir.sh with already expanded filenames
e.g. the script get executed (in your case) as: dir.sh test.pas test.swift
BTW, it acts exactly with the same way for your ls example:
the bash expands the ls test* to ls test.pas test.swift
then executes the ls with the above two arguments
and the ls will print the result for the got two arguments.
with other words, the ls don't even see the test* argument - if it is possible - the bash expands the wilcard characters. (* and ?).
Now back to your script: add after the shebang the following line:
echo "the $0 got this arguments: $#"
and you will immediatelly see, the real argumemts how your script got executed.
also, in such cases is a good practice trying to execute the script in debug-mode, e.g.
bash -x dir.sh test*
and you will see, what the script does exactly.
Also, you can do the same for your current interpreter, e.g. just enter into the terminal
set -x
and try run the dir.sh test* = and you will see, how the bash will execute the dir.sh command. (to stop the debug mode, just enter set +x)
Everbody is giving you valuable advice which you should definitely should follow!
But here is the real answer to your question.
To pass unexpanded arguments to any executable you need to single quote them:
./your_script '*'
The best solution I have is to use the eval command, in this way:
#!/bin/bash
cmd="some command \"with_quetes_and_asterisk_in_it*\""
echo "$cmd"
eval $cmd
The eval command takes its arguments and evaluates them into the command as the shell does.
This solves my problem when I need to call a command with asterisk '*' in it from a script.
I want to add the string %%% to the beginning of some specific lines in a text file.
This is my script:
#!/bin/bash
a="c:\Temp"
sed "s/$a/%%%$a/g" <File.txt
And this is my File.txt content:
d:\Temp
c:\Temp
e:\Temp
But nothing changes when I execute it.
I think the 'sed' command is not finding the pattern, possibly due to the \ backslashes in the variable a.
I can find the c:\Temp line if I use grep with -F option (to not interpret strings):
cat File.txt | grep -F "$a"
But sed seems not to implement such '-F` option.
Not working neither:
sed 's/$a/%%%$a/g' <File.txt
sed 's/"$a"/%%%"$a"/g' <File.txt
I have found similar threads about replacing with sed, but they don't refer to variables.
How can I replace the desired lines by using a variable adding them the %%% char string?
EDIT: It would be fine that the $a variable could be entered via parameter when calling the script, so it will be assigned like:
a=$1
Try it like this:
#!/bin/sh
a='c:\\Temp' # single quotes
sed "s/$a/%%%$a/g" <File.txt # double quotes
Output:
Johns-MacBook-Pro:sed jcreasey$ sh x.sh
d:\Temp
e:\Temp
%%%c:\Temp
You need the double slash '\' to escape the '\'.
The single quotes won't expand the variables.
So you escape the slash in single quotes and pass it into the double quotes.
Of course you could also just do this:
#!/bin/sh
sed 's/\(.*Temp\)/%%%&/' <File.txt
If you want to get input from the command line you have to allow for the fact that \ is an escape character there too. So the user needs to type 'c:\\' or the interpreter will just wait for another character. Then once you get it, you will need to escape it again. (printf %q).
#!/bin/sh
b=`printf "%q" $1`
sed "s/\($b\)/%%% &/" < File.txt
The issue you are having has to do with substitution of your variable providing a regular expression looking for a literal c:Temp with the \ interpreted as an escape by the shell. There are a number of workarounds. Seeing the comments and having worked through the possibilities, the following will allow the unquoted entry of the search term:
#!/bin/bash
## validate that needed input is given on the command line
[ -n "$1" -a "$2" ] || {
printf "Error: insufficient input. Usage: %s <term> <file>\n" "${0//*\//}" >&2
exit 1
}
## validate that the filename given is readable
[ -r "$2" ] || {
printf "Error: file not readable '%s'\n" "$2" >&2
exit 1
}
a="$1" # assign a
filenm="$2" # assign filename
## test and fix the search term entered
[[ "$a" =~ '/' ]] || a="${a/:/:\\}" # test if \ removed by shell, if so replace
a="${a/\\/\\\\}" # add second \
sed -e "s/$a/%%%$a/g" "$filenm" # call sed with output to stdout
Usage:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Note: This allows both single-quoted or unquoted entry of the dos path search term. To edit in place use sed -i. Additionally, the [[ operator and =~ operator are limited to bash.
I could have sworn the original question said replace, but to append, just as you suggest in the comments. I have updated the code with:
sed -e "s/$a/%%%$a/g" "$filenm"
Which provides the new output:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Remember: If you want to edit the file in place use sed -i or sed -i.bak which will edit the actual file (and if -i.bak is given create a backup of the original in originalname.bak). Let me know if that is not what you intended and I'm happy to edit again.
Creating your script with a positional parameter of $1
#!/bin/bash
a="$1"
cat <file path>|sed "s/"$1"/%%%"$1"/g" > "temporary file"
Now whenever you want sed to find "c:\Temp" you need to use your script command line as follows
bash <my executing script> c:\\\\Temp
The first backslash will make bash interpret any backslashes that follows therefore what will be save in variable "a" in your executing script is "c:\\Temp". Now substituting this variable in sed will cause sed to interpret 1 backlash since the first backslash in this variable will cause sed to start interpreting the other backlash.
when you Open your temporary file you will see:
d:\Temp
%%%c:\Temp
e:\Temp
I want to print code into a file using cat <<EOF >>:
cat <<EOF >> brightup.sh
!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
curr=$((curr+406));
echo $curr > /sys/class/backlight/intel_backlight/brightness;
fi
EOF
but when I check the file output, I get this:
!/bin/bash
curr=1634
if [ -lt 4477 ]; then
curr=406;
echo > /sys/class/backlight/intel_backlight/brightness;
fi
I tried putting single quotes but the output also carries the single quotes with it. How can I avoid this issue?
You only need a minimal change; single-quote the here-document delimiter after <<.
cat <<'EOF' >> brightup.sh
or equivalently backslash-escape it:
cat <<\EOF >>brightup.sh
Without quoting, the here document will undergo variable substitution, backticks will be evaluated, etc, like you discovered.
If you need to expand some, but not all, values, you need to individually escape the ones you want to prevent.
cat <<EOF >>brightup.sh
#!/bin/sh
# Created on $(date # : <<-- this will be evaluated before cat;)
echo "\$HOME will not be evaluated because it is backslash-escaped"
EOF
will produce
#!/bin/sh
# Created on Fri Feb 16 11:00:18 UTC 2018
echo "$HOME will not be evaluated because it is backslash-escaped"
As suggested by #fedorqui, here is the relevant section from man bash:
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion,
or pathname expansion is performed on word. If any characters in word
are quoted, the delimiter is the result of quote removal on word, and
the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion. In the
latter case, the character sequence \<newline> is ignored, and \
must be used to quote the characters \, $, and `.
This should work, I just tested it out and it worked as expected: no expansion, substitution, or what-have-you took place.
cat <<< '
#!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
curr=$((curr+406));
echo $curr > /sys/class/backlight/intel_backlight/brightness;
fi' > file # use overwrite mode so that you don't keep on appending the same script to that file over and over again, unless that's what you want.
Using the following also works.
cat <<< ' > file
... code ...'
Also, it's worth noting that when using heredocs, such as << EOF, substitution and variable expansion and the like takes place. So doing something like this:
cat << EOF > file
cd "$HOME"
echo "$PWD" # echo the current path
EOF
will always result in the expansion of the variables $HOME and $PWD. So if your home directory is /home/foobar and the current path is /home/foobar/bin, file will look like this:
cd "/home/foobar"
echo "/home/foobar/bin"
instead of the expected:
cd "$HOME"
echo "$PWD"
Or, using your EOF markers, you need to quote the initial marker so expansion won't be done:
#-----v---v------
cat <<'EOF' >> brightup.sh
#!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
curr=$((curr+406));
echo $curr > /sys/class/backlight/intel_backlight/brightness;
fi
EOF
IHTH
I know this is a two year old question, but this is a quick answer for those searching for a 'how to'.
If you don't want to have to put quotes around anything you can simply write a block of text to a file, and escape variables you want to export as text (for instance for use in a script) and not escape one's you want to export as the value of the variable.
#!/bin/bash
FILE_NAME="test.txt"
VAR_EXAMPLE="\"string\""
cat > ${FILE_NAME} << EOF
\${VAR_EXAMPLE}=${VAR_EXAMPLE} in ${FILE_NAME}
EOF
Will write '"${VAR_EXAMPLE}="string" in test.txt' into test.txt
This can also be used to output blocks of text to the console with the same rules by omitting the file name
#!/bin/bash
VAR_EXAMPLE="\"string\""
cat << EOF
\${VAR_EXAMPLE}=${VAR_EXAMPLE} to console
EOF
Will output '"${VAR_EXAMPLE}="string" to console'
cat with <<EOF>> will create or append the content to the existing file, won't overwrite. whereas cat with <<EOF> will create or overwrite the content.
cat test.txt
hello
cat <<EOF>> test.txt
> hi
> EOF
cat test.txt
hello
hi
cat <<EOF> test.txt
> haiiiii
> EOF
cat test.txt
haiiiii
Say, I have a file foo.txt specifying N arguments
arg1
arg2
...
argN
which I need to pass to the command my_command
How do I use the lines of a file as arguments of a command?
If your shell is bash (amongst others), a shortcut for $(cat afile) is $(< afile), so you'd write:
mycommand "$(< file.txt)"
Documented in the bash man page in the 'Command Substitution' section.
Alterately, have your command read from stdin, so: mycommand < file.txt
As already mentioned, you can use the backticks or $(cat filename).
What was not mentioned, and I think is important to note, is that you must remember that the shell will break apart the contents of that file according to whitespace, giving each "word" it finds to your command as an argument. And while you may be able to enclose a command-line argument in quotes so that it can contain whitespace, escape sequences, etc., reading from the file will not do the same thing. For example, if your file contains:
a "b c" d
the arguments you will get are:
a
"b
c"
d
If you want to pull each line as an argument, use the while/read/do construct:
while read i ; do command_name $i ; done < filename
command `< file`
will pass file contents to the command on stdin, but will strip newlines, meaning you couldn't iterate over each line individually. For that you could write a script with a 'for' loop:
for line in `cat input_file`; do some_command "$line"; done
Or (the multi-line variant):
for line in `cat input_file`
do
some_command "$line"
done
Or (multi-line variant with $() instead of ``):
for line in $(cat input_file)
do
some_command "$line"
done
References:
For loop syntax: https://www.cyberciti.biz/faq/bash-for-loop/
You do that using backticks:
echo World > file.txt
echo Hello `cat file.txt`
If you want to do this in a robust way that works for every possible command line argument (values with spaces, values with newlines, values with literal quote characters, non-printable values, values with glob characters, etc), it gets a bit more interesting.
To write to a file, given an array of arguments:
printf '%s\0' "${arguments[#]}" >file
...replace with "argument one", "argument two", etc. as appropriate.
To read from that file and use its contents (in bash, ksh93, or another recent shell with arrays):
declare -a args=()
while IFS='' read -r -d '' item; do
args+=( "$item" )
done <file
run_your_command "${args[#]}"
To read from that file and use its contents (in a shell without arrays; note that this will overwrite your local command-line argument list, and is thus best done inside of a function, such that you're overwriting the function's arguments and not the global list):
set --
while IFS='' read -r -d '' item; do
set -- "$#" "$item"
done <file
run_your_command "$#"
Note that -d (allowing a different end-of-line delimiter to be used) is a non-POSIX extension, and a shell without arrays may also not support it. Should that be the case, you may need to use a non-shell language to transform the NUL-delimited content into an eval-safe form:
quoted_list() {
## Works with either Python 2.x or 3.x
python -c '
import sys, pipes, shlex
quote = pipes.quote if hasattr(pipes, "quote") else shlex.quote
print(" ".join([quote(s) for s in sys.stdin.read().split("\0")][:-1]))
'
}
eval "set -- $(quoted_list <file)"
run_your_command "$#"
If all you need to do is to turn file arguments.txt with contents
arg1
arg2
argN
into my_command arg1 arg2 argN then you can simply use xargs:
xargs -a arguments.txt my_command
You can put additional static arguments in the xargs call, like xargs -a arguments.txt my_command staticArg which will call my_command staticArg arg1 arg2 argN
Here's how I pass contents of a file as an argument to a command:
./foo --bar "$(cat ./bar.txt)"
None of the answers seemed to work for me or were too complicated. Luckily, it's not complicated with xargs (Tested on Ubuntu 20.04).
This works with each arg on a separate line in the file as the OP mentions and was what I needed as well.
cat foo.txt | xargs my_command
One thing to note is that it doesn't seem to work with aliased commands.
The accepted answer works if the command accepts multiple args wrapped in a string. In my case using (Neo)Vim it does not and the args are all stuck together.
xargs does it properly and actually gives you separate arguments supplied to the command.
I suggest using:
command $(echo $(tr '\n' ' ' < parameters.cfg))
Simply trim the end-line characters and replace them with spaces, and then push the resulting string as possible separate arguments with echo.
In my bash shell the following worked like a charm:
cat input_file | xargs -I % sh -c 'command1 %; command2 %; command3 %;'
where input_file is
arg1
arg2
arg3
As evident, this allows you to execute multiple commands with each line from input_file, a nice little trick I learned here.
Both solutions work even when lines have spaces:
readarray -t my_args < foo.txt
my_command "${my_args[#]}"
if readarray doesn't work, replace it with mapfile, they're synonyms.
I formerly tried this one below, but had problems when my_command was a script:
xargs -d '\n' -a foo.txt my_command
After editing #Wesley Rice's answer a couple times, I decided my changes were just getting too big to continue changing his answer instead of writing my own. So, I decided I need to write my own!
Read each line of a file in and operate on it line-by-line like this:
#!/bin/bash
input="/path/to/txt/file"
while IFS= read -r line
do
echo "$line"
done < "$input"
This comes directly from author Vivek Gite here: https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/. He gets the credit!
Syntax: Read file line by line on a Bash Unix & Linux shell:
1. The syntax is as follows for bash, ksh, zsh, and all other shells to read a file line by line
2. while read -r line; do COMMAND; done < input.file
3. The -r option passed to read command prevents backslash escapes from being interpreted.
4. Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
5. while IFS= read -r line; do COMMAND_on $line; done < input.file
And now to answer this now-closed question which I also had: Is it possible to `git add` a list of files from a file? - here's my answer:
Note that FILES_STAGED is a variable containing the absolute path to a file which contains a bunch of lines where each line is a relative path to a file I'd like to do git add on. This code snippet is about to become part of the "eRCaGuy_dotfiles/useful_scripts/sync_git_repo_to_build_machine.sh" file in this project, to enable easy syncing of files in development from one PC (ex: a computer I code on) to another (ex: a more powerful computer I build on): https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles.
while IFS= read -r line
do
echo " git add \"$line\""
git add "$line"
done < "$FILES_STAGED"
References:
Where I copied my answer from: https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/
For loop syntax: https://www.cyberciti.biz/faq/bash-for-loop/
Related:
How to read contents of file line-by-line and do git add on it: Is it possible to `git add` a list of files from a file?
In a BASH script, I'm trying to detect whether a file exists. The filename is in a variable but the -e command seems to be unable to detect the file. The following code always outputs "~/misc/tasks/drupal_backup.sh does not exist"
filename="~/misc/tasks/drupal_backup.sh"
if [ -e "$filename" ]; then
echo "$filename exists"
else
echo "$filename does not exist"
fi
On the other hand, the following code detects the file correctly:
if [ -e ~/misc/tasks/drupal_backup.sh ]; then
echo "$filename exists"
else
echo "$filename does not exist"
fi
Why would this be? How can I get it to detect the file when the filename is in a variable?
That's an interesting one. Substituting $HOME for ~ works as does removing the quotes from the assignment.
If you put a set -x at the top of that script, you'll see that the version with quotes sets filename to ~/... which is what's given to -e. If you remove the quotes, filename is set to the expanded /home/somebody/.... So in the first case, you see:
+ [ -e ~/... ]
and it doesn't like it. In the second case, you see:
+ [ -e /home/somebody/... ]
and it does work.
If you do it without the variable, you see:
+ [ -e /home/somebody/... ]
and, yes, it works.
After a bit of investigation, I've found that it's actually the order in which bash performs its expansions. From the bash man page:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
That's why it's not working, the variable is substituted after the tilde expansion. In other words, at the point where bash wants to expand ~, there isn't one. It's only after variable expansion does the word get changed into ~/... and there will be no tilde expansion after that.
One thing you could do is to change your if statement to:
if [[ -e $(eval echo $filename) ]]; then
This will evaluate the $filename argument twice. The first time (with eval), there will be no ~ during the tilde expansion phase but $filename will be changed to ~/... during the variable expansion phase.
Then, on the second evaluation (the one being done as part of the if itself), the ~ will be there during the tilde expansion phase.
I've tested that on my .profile file and it seems to work, I suggest you confirm in your particular case.