I am writing a Bourne Script and am looking to select files that match certain regular expressions. I am doing this with an if test structure, and it identifies a file ending in ".o". However, when there are two files in a directory which I am searching that end in ".o" I get the following error: "expr: syntax error". How could this be possible?
if test "`expr \"$file\" : ${SPECIFIED_DIRECTORY}/*.o`" != "0"; then
do something
fi
A regular expression test is almost certainly the wrong tool for the job here, but let's assume that specified_directory (lower-case by convention to avoid conflicts with environment variables and builtins) contained a regex to match a directory name, as opposed to a literal name, and thus actually was the right tool. If that were the case, you'd want to write...
if expr "$file" : "$specified_directory"/'.*[.]o' >/dev/null; then
...
fi
No test command, no subshell. Keep it simple.
If you don't escape the . (in this case, by making it a character class, [.]), it has its normal regular-expression meaning, of matching exactly one character. Similarly, .* is the way to match zero-or-more of any character in a regex, not bare * (which is the fnmatch syntax).
This approach works on any POSIX shell, and calls no tools not available built-in to the shell, thus making it efficient at runtime.
contains_any_files() {
set -- "$1"/*
[ "$#" -gt 0 ] && [ -f "$1" ]
}
if contains_any_files "$dir"; then
...
fi
If I understand correctly, if there are any .o files in the directory, then do something:
if find "$dir"/*.o >/dev/null 2>&1; then
# do something
fi
The find command will exit with success only if there are any matching files. Otherwise it will exit with failure, and the then block won't be executed. The >/dev/null 2>&1 is to hide stdout and stderr.
Related
I'm writing a simple script to check some repositories updates and, if needed, I'm making new packages from these updates to install new versions of those programs it refers to (in Arch Linux). So I made some testing before executing the real script.
The problem is that I'm getting the error [: excessive number of arguments (but I think the proper translation would be [: too many arguments) from this piece of code:
# Won't work despite the double quoted $r
if [ "$r" == *"irt"* ]; then
echo "TEST"
fi
The code is fixed by adding double square brackets which I did thanks to this SO answer made by #user568458:
# Makes the code works
if [[ "$r" == *"irt"* ]]; then
echo "TEST"
fi
Note that $r is defined by:
# Double quotes should fix it, right? Those special characters/multi-lines
r="$(ls)"
Also note that everything is inside a loop and the loop progress with success. The problems occurs every time the if comparison matches, not printing the "TEST" issued, jumping straight to the next iteration of the loop (no problem: no code exists after this if).
My question is: why would the error happens every time the string matches? By my understanding, the double quotes would suffice to fix it. Also, If I count on double square brackets to fix it, some shells won't recognize it (refers to the answer mentioned above). What's the alternative?
Shell scripting seems a whole new programming paradigm.. I never quite grasp the details and fail to secure a great source for that.
The single bracket is a shell builtin, as opposed to the double bracket which is a shell keyword. The difference is that a builtin behaves like a command: word splitting, file pattern matching, etc. occur when the shell parses the command. If you have files that match the pattern *irt*, say file1irt.txt and file2irt.txt, then when the shell parses the command
[ "$r" = *irt* ]
it expands $r, matches all files matching the pattern *irt*, and eventually sees the command:
[ expansion_of_r = file1irt.txt file2irt.txt ]
which yields an error. No quotes can fix that. In fact, the single bracket form can't handle pattern matching at all.
On the other hand, the double brackets are not handled like commands; Bash will not perform any word splitting nor file pattern matching, so it really sees
[[ "expansion_of_r" = *irt* ]]
In this case, the right hand side is a pattern, so Bash tests whether the left hand side matches that pattern.
For a portable alternative, you can use:
case "$r" in
(*irt*) echo "TEST" ;;
esac
But now you have a horrible anti-pattern here. You're doing:
r=$(ls)
if [[ "$r" = *irt* ]]; then
echo "TEST"
fi
What I understand is that you want to know whether there are files matching the pattern *irt* in the current directory. A portable possibility is:
for f in *irt*; do
if [ -e "$f" ]; then
echo "TEST"
break
fi
done
Since you're checking for files with a certain file name, I'd suggest to use find explicitly. Something like
r="$(find . -name '*irt*' 2> /dev/null)"
if [ ! -z "$r" ]; then
echo "found: $r"
fi
I am having a problem where cmd1 works, but not cmd2 in my Bash script ending in .sh. I have made the Bash script executable.
Additionally, I can execute cmd2 just fine from my Bash terminal. I have tried to make a minimally reproducible example, but my larger goal is to run a complicated executable with command line arguments and pass output to a file that may or may not exist (rather than displaying the output in the terminal).
Replacing > with >> also gives the same error in the script, but not the terminal.
My Bash script:
#!/bin/bash
cmd1="cat test.txt"
cmd2="cat test.txt > a"
echo $cmd1
$cmd1
echo $cmd2
$cmd2
test.txt has the words "dog" and "cat" on two separate lines without quotes.
Short answer: see BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.
Long answer: the shell expands variable references (like $cmd1) toward the end of the process of parsing a command line, after it's done parsing redirects (like > a is supposed to be) and quotes and escapes and... In fact, the only thing it does with the expanded value is word splitting (e.g. treating cat test.txt > a as "cat" followed by "test.txt", ">", and finally "a", rather than a single string) and wildcard expansion (e.g. if $cmd expanded to cat *.txt, it'd replace the *.txt part with a list of matching files). (And it skips word splitting and wildcard expansion if the variable is in double-quotes.)
Partly as a result of this, the best way to store commands in variables is: don't. That's not what they're for; variables are for data, not commands. What you should do instead, though, depends on why you were storing the command in a variable.
If there's no real reason to store the command in a variable, then just use the command directly. For conditional redirects, just use a standard if statement:
if [ -f a ]; then
cat test.txt > a
else
cat test.txt
fi
If you need to define the command at one point, and use it later; or want to use the same command over and over without having to write it out in full each time, use a function:
cmd2() {
cat test.txt > a
}
cmd2
It sounds like you may need to be able to define the command differently depending on some condition, you can actually do that with a function as well:
if [ -f a ]; then
cmd() {
cat test.txt > a
}
else
cmd() {
cat test.txt
}
fi
cmd
Alternately, you can wrap the command (without redirect) in a function, then use a conditional to control whether it redirects:
cmd() {
cat test.txt
}
if [ -f a ]; then
cmd > a
else
cmd
fi
It's also possible to wrap a conditional redirect into a function itself, then pipe output to it:
maybe_redirect_to() {
if [ -f "$1" ]; then
cat > "$1"
else
cat
fi
}
cat test.txt | maybe_redirect_to a
(This creates an extra cat process that isn't really doing anything useful, but if it makes the script cleaner, I'd consider that worth it. In this particular case, you could minimize the stray cats by using maybe_redirect_to a < test.txt.)
As a last resort, you can store the command string in a variable, and use eval to parse it. eval basically re-runs the shell parsing process from the beginning, meaning that it'll recognize things like redirects in the string. But eval has a well-deserved reputation as a bug magnet, because it's easy for it to treat parts of the string you thought were just data as command syntax, which can cause some really weird (& dangerous) bugs.
If you must use eval, at least double-quote the variable reference, so it runs through the parsing process just once, rather than sort-of-once-and-a-half as it would unquoted. Here's an example of what I mean:
cmd3="echo '5 * 3 = 15'"
eval "$cmd3"
# prints: 5 * 3 = 15
eval $cmd3
# prints: 5 [list of files in the current directory] 3 = 15
# ...unless there are any files with shell metacharacters in their names, in
# which case something more complicated might happen.
BashFAQ #50 discusses some other possible reasons and solutions. Note that the array approach will not work here, since arrays also get expanded after redirects are parsed.
If you pop an 'eval' in front of $cmd2 it should work as expected:
#!/bin/bash
cmd2="cat test.txt > a"
eval $cmd2
If you're not sure about the operation of a script you could always use the debug mode to see if you can determine the error.
bash -x scriptname
This will run the command and display the output of variable evaluations. Hopefully this will reveal any issues with syntax.
Hi… Need a little help here…
I tried to emulate the DOS' dir command in Linux using bash script. Basically it's just a wrapped ls command with some parameters plus summary info. Here's the script:
#!/bin/bash
# default to current folder
if [ -z "$1" ]; then var=.;
else var="$1"; fi
# check file existence
if [ -a "$var" ]; then
# list contents with color, folder first
CMD="ls -lgG $var --color --group-directories-first"; $CMD;
# sum all files size
size=$(ls -lgGp "$var" | grep -v / | awk '{ sum += $3 }; END { print sum }')
if [ "$size" == "" ]; then size="0"; fi
# create summary
if [ -d "$var" ]; then
folder=$(find $var/* -maxdepth 0 -type d | wc -l)
file=$(find $var/* -maxdepth 0 -type f | wc -l)
echo "Found: $folder folders "
echo " $file files $size bytes"
fi
# error message
else
echo "dir: Error \"$var\": No such file or directory"
fi
The problem is when the argument contains an asterisk (*), the ls within the script acts differently compare to the direct ls command given at the prompt. Instead of return the whole files list, the script only returns the first file. See the video below to see the comparation in action. I don't know why it behaves like that.
Anyone knows how to fix it? Thank you.
Video: problem in action
UPDATE:
The problem has been solved. Thank you all for the answers. Now my script works as expected. See the video here: http://i.giphy.com/3o8dp1YLz4fIyCbOAU.gif
The asterisk * is expanded by the shell when it parses the command line. In other words, your script doesn't get a parameter containing an asterisk, it gets a list of files as arguments. Your script only works with $1, the first argument. It should work with "$#" instead.
This is because when you retrieve $1 you assume the shell does NOT expand *.
In fact, when * (or other glob) matches, it is expanded, and broken into segments by $IFS, and then passed as $1, $2, etc.
You're lucky if you simply retrieved the first file. When your first file's path contains spaces, you'll get an error because you only get the first segment before the space.
Seriously, read this and especially this. Really.
And please don't do things like
CMD=whatever you get from user input; $CMD;
You are begging for trouble. Don't execute arbitrary string from the user.
Both above answers already answered your question. So, i'm going a bit more verbose.
In your terminal is running the bash interpreter (probably). This is the program which parses your input line(s) and doing "things" based on your input.
When you enter some line the bash start doing the following workflow:
parsing and lexical analysis
expansion
brace expansion
tidle expansion
variable expansion
artithmetic and other substitutions
command substitution
word splitting
filename generation (globbing)
removing quotes
Only after all above the bash
will execute some external commands, like ls or dir.sh... etc.,
or will do so some "internal" actions for the known keywords and builtins like echo, for, if etc...
As you can see, the second last is the filename generation (globbing). So, in your case - if the test* matches some files, your bash expands the willcard characters (aka does the globbing).
So,
when you enter dir.sh test*,
and the test* matches some files
the bash does the expansion first
and after will execute the command dir.sh with already expanded filenames
e.g. the script get executed (in your case) as: dir.sh test.pas test.swift
BTW, it acts exactly with the same way for your ls example:
the bash expands the ls test* to ls test.pas test.swift
then executes the ls with the above two arguments
and the ls will print the result for the got two arguments.
with other words, the ls don't even see the test* argument - if it is possible - the bash expands the wilcard characters. (* and ?).
Now back to your script: add after the shebang the following line:
echo "the $0 got this arguments: $#"
and you will immediatelly see, the real argumemts how your script got executed.
also, in such cases is a good practice trying to execute the script in debug-mode, e.g.
bash -x dir.sh test*
and you will see, what the script does exactly.
Also, you can do the same for your current interpreter, e.g. just enter into the terminal
set -x
and try run the dir.sh test* = and you will see, how the bash will execute the dir.sh command. (to stop the debug mode, just enter set +x)
Everbody is giving you valuable advice which you should definitely should follow!
But here is the real answer to your question.
To pass unexpanded arguments to any executable you need to single quote them:
./your_script '*'
The best solution I have is to use the eval command, in this way:
#!/bin/bash
cmd="some command \"with_quetes_and_asterisk_in_it*\""
echo "$cmd"
eval $cmd
The eval command takes its arguments and evaluates them into the command as the shell does.
This solves my problem when I need to call a command with asterisk '*' in it from a script.
I want to know how I can use an if statement in a shell script to check the existence of multiple directories.
For example, if /tmp has subdirectories test1, test2, test3, I want to move them to another directory.
I am using if [ -d /tmp/test* ]; then mv test* /pathOfNewDir
but it does not work on the if statement part.
The -d test only accepts one argument, so you'll need to test each directory individually. I would also not recommend moving test* as it may match more than you intended.
Use the double-bracket syntax test syntax (e.g. if [[ -d...), which is bash-specific but tends to be clearer and have fewer gotchas than the single-bracket syntax. If you just need to check a few directories, you can do it with a simple statement like if [[ -d /tmp/test1 && -d /tmp/test2 && -d /tmp/test3 ]]; then...
Unfortunately, the shell's file-testing operators (such as -d and -f) operate on a single, literal path only:
A conditional such as [ -d /tmp/test* ] won't work, because if /tmp/test* expands to multiple matches, you'll get a syntax error (only 1 argument accepted).
The bash variant [[ -d /tmp/test* ]] doesn't work either, because no globbing (pathname expansion) is performed inside [[ ... ]].
To test whether a globbing pattern matches anything, the cleanest approach is to define an auxiliary function (this solution is POSIX-compliant):
exists() { [ -e "$1" ]; }
Invoke it with an [unquoted] pattern, e.g.:
exists foo* && echo 'HAVE MATCHES'
# or, in an `if` statement:
if exists foo*; then # ...
The only caveat is that if shopt -s failglob is in effect in bash, an error message will be printed to stderr if there's no match, and the rest of the command will not be executed.
See below for an explanation of the function.
Applied to your specific scenario, we get (using bash syntax):
# Define aux. function
exists() { [[ -e $1 ]]; }
exists /tmp/test*/ && mv /tmp/test*/ /path/to/new/dir
Note the trailing / in /tmp/test*/ to ensure that only directories match, if any.
&& ensures that the following command is only executed if the function's exit code indicates true.
mv /tmp/test*/ ... moves all matching directories at once to the new target directory.
Alternatively, capture globbing results in an helper array variable:
if matches=(/tmp/test*/) && [[ -e ${matches[0]} ]]; then
mv "${matches[#]}" /path/to/new/dir
fi
Or, process matches individually:
for d in /tmp/test*/; do
[[ -e $d ]] || break # exit, if no actual match
# Process individual match.
mv "$d" /path/to/new/dir
done
Explanation of auxiliary function exists() { [ -e "$1" ]; }:
It takes advantage of several shell features:
If you invoke it with a[n unquoted] pattern such as exists foo*, the shell will expand foo* to all matching files/directories and pass their names as individual arguments to the function.
If there are no matches, the pattern will be passed as is to the function - this behavior is mandated by POSIX.
Caveat: bash has configuration items that allow changing this behavior (shell options failglob and nullglob) - though by default it acts as mandated by POSIX in this case. (zsh, sadly, by default fails if there's no match.)
Inside the function, it's sufficient to examine the 1st argument ($1) to determine whether any matches were found:
If the 1st argument, $1 refers to an actual, existing filesystem item (as indicated by the exit code of the -e file-test operator), the implication is that the pattern indeed matched something (at least one, possibly more items).
Otherwise, the implication is that the pattern was passed as is, implying that no matches were found.
Note that the exit code of the -e test - due to being the last command in the function - implicitly serves as the exit code of the function as a whole.
It looks like you may want to use find:
find /tmp -name "test*" -maxdepth 1 -type d -exec mv \{\} /target/directory \;
This finds all test* directories directly under /tmp without recursion and moves them to /target/directory.
This approach uses ls and grep to create a list of matching directories or write an error in case no such directories are found:
IFS="
" # input is separated with newlines
if dirs=$( ls -1 -F | grep "^test.*/" | tr -d "/" )
then
# directories found - move them:
for d in $dirs
do
mv "$d" "$target_directory"/
done
else
# no directories found - send error
fi
While it would seem feasible to use find for such a task, find does not directly provide feedback on the number of matches as required by the OP according to the comments.
Note: Using ls for the task introduces a few limitations on filenames. This approach will not work with filenames containing newlines or wildcard characters.
I'm new to shell programming. I intend to get directory name after zip file was extracted. The print statement of it is
$test.sh helloworld.zip
helloworld
Let's take a look at test.sh:
#! /bin/sh
length=echo `expr index "$1" .zip`
a=$1
echo $(a:0:length}
However I got the Bad substitution error from the compiler.
And when I mention about 'shell'.I just talking about shell for I don't know the difference between bash or the others.I just using Ubuntu 10.04 and using the terminal. (I am using bash.)
If your shell is a sufficiently recent version of bash, that parameter expansion notation should work.
In many other shells, it will not work, and a bad substitution error is the way the shell says 'You asked for a parameter substitution but it does not make sense to me'.
Also, given the script:
#! /bin/sh
length=echo `expr index "$1" .zip`
a=$1
echo $(a:0:length}
The second line exports variable length with value echo for the command that is generated by running expr index "$1" .zip. It does not assign to length. That should be just:
length=$(expr index "${1:?}" .zip)
where the ${1:?} notation generates an error if $1 is not set (if the script is invoked with no arguments).
The last line should be:
echo ${a:0:$length}
Note that if $1 holds filename.zip, the output of expr index $1 .zip is 2, because the letter i appears at index 2 in filename.zip. If the intention is to get the base name of the file without the .zip extension, then the classic way to do it is:
base=$(basename $1 .zip)
and the more modern way is:
base=${1%.zip}
There is a difference; if the name is /path/to/filename.zip, the classic output is filename and the modern one is /path/to/filename. You can get the classic output with:
base=${1%.zip}
base=${base##*/}
Or, in the classic version, you can get the path with:
base=$(dirname $1)/$(basename $1 .zip)`.)
If the file names can contain spaces, you need to think about using double quotes, especially in the invocations of basename and dirname.
Try running it with bash.
bash test.sh helloworld.zip
-likewise-
"try changing the first line to #!/bin/bash" as comment-answered by – #shellter
Try that in bash :
echo $1
len=$(wc -c <<< "$1")
a="${1}.zip"
echo ${a:0:$len}
Adapt it to fit your needs.