I'm trying to write a bash script that allows the user to pass a directory path using wildcards.
For example,
bash show_files.sh *
when executed within this directory
drw-r--r-- 2 root root 4.0K Sep 18 11:33 dir_a
-rw-r--r-- 1 root root 223 Sep 18 11:33 file_b.txt
-rw-rw-r-- 1 root root 106 Oct 18 15:48 file_c.sql
would output:
dir_a
file_b.txt
file_c.sql
The way it is right now, it outputs:
dir_a
contents of show_files.sh:
#!/bin/bash
dirs="$1"
for dir in $dirs
do
echo $dir
done
The parent shell, the one invoking bash show_files.sh *, expands the * for you.
In your script, you need to use:
for dir in "$#"
do
echo "$dir"
done
The double quotes ensure that multiple spaces etc in file names are handled correctly.
See also How to iterate over arguments in a bash shell script.
Potentially confusing addendum
If you're truly sure you want to get the script to expand the *, you have to make sure that * is passed to the script (enclosed in quotes, as in the other answers), and then make sure it is expanded at the right point in the processing (which is not trivial). At that point, I'd use an array.
names=( $# )
for file in "${names[#]}"
do
echo "$file"
done
I don't often use $# without the double quotes, but this is one time when it is more or less the correct thing to do. The tricky part is that it won't handle wild cards with spaces in very well.
Consider:
$ > "double space.c"
$ > "double space.h"
$ echo double\ \ space.?
double space.c double space.h
$
That works fine. But try passing that as a wild-card to the script and ... well, let's just say it gets to be tricky at that point.
If you want to extract $2 separately, then you can use:
names=( $1 )
for file in "${names[#]}"
do
echo "$file"
done
# ... use $2 ...
Quote the wild-card:
bash show_files.sh '*'
or make your script accept a list of arguments, not just one:
for dir in "$#"
do
echo "$dir"
done
It's better to iterate directly over "$#' rather than assigning it to another variable, in order to preserve its special ability to hold elements that themselves contain whitespace.
Related
Let's say I have a directory path in variable DIR and I want to list this directory. If I care only about spaces in the path, then I could do
ls "$DIR"
What should I write if I want to support also single and double quotes and other weird stuff in the directory path? Example:
DIR="/Users/Mick O'Neil (the \"Terminator\")/Desktop"
echo $DIR # prints /Users/Mick O'Neil (the "Terminator")/Desktop
ls <what should I write here?>
Quotes are not just for spaces but for everything, so using the double quotes is the safety level you need here.
From Bash Reference Manual on Quoting:
3.1.2.3 Double Quotes
Enclosing characters in double quotes (‘"’) preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘`’, ‘\’, and, when history expansion is enabled, ‘!’.
Let's store this string into a file for later usage.
$ cat file
Mick O'Neil (the "Terminator")
Read into a var:
$ filename=$(<file)
Check its value:
$ echo "$filename"
Mick O'Neil (the "Terminator")
Create a file with this value:
$ touch "$filename"
Check it has been created successfully:
$ lt "$filename"
-rw-r--r-- 1 me me 0 Mar 1 15:09 Mick O'Neil (the "Terminator")
Hi… Need a little help here…
I tried to emulate the DOS' dir command in Linux using bash script. Basically it's just a wrapped ls command with some parameters plus summary info. Here's the script:
#!/bin/bash
# default to current folder
if [ -z "$1" ]; then var=.;
else var="$1"; fi
# check file existence
if [ -a "$var" ]; then
# list contents with color, folder first
CMD="ls -lgG $var --color --group-directories-first"; $CMD;
# sum all files size
size=$(ls -lgGp "$var" | grep -v / | awk '{ sum += $3 }; END { print sum }')
if [ "$size" == "" ]; then size="0"; fi
# create summary
if [ -d "$var" ]; then
folder=$(find $var/* -maxdepth 0 -type d | wc -l)
file=$(find $var/* -maxdepth 0 -type f | wc -l)
echo "Found: $folder folders "
echo " $file files $size bytes"
fi
# error message
else
echo "dir: Error \"$var\": No such file or directory"
fi
The problem is when the argument contains an asterisk (*), the ls within the script acts differently compare to the direct ls command given at the prompt. Instead of return the whole files list, the script only returns the first file. See the video below to see the comparation in action. I don't know why it behaves like that.
Anyone knows how to fix it? Thank you.
Video: problem in action
UPDATE:
The problem has been solved. Thank you all for the answers. Now my script works as expected. See the video here: http://i.giphy.com/3o8dp1YLz4fIyCbOAU.gif
The asterisk * is expanded by the shell when it parses the command line. In other words, your script doesn't get a parameter containing an asterisk, it gets a list of files as arguments. Your script only works with $1, the first argument. It should work with "$#" instead.
This is because when you retrieve $1 you assume the shell does NOT expand *.
In fact, when * (or other glob) matches, it is expanded, and broken into segments by $IFS, and then passed as $1, $2, etc.
You're lucky if you simply retrieved the first file. When your first file's path contains spaces, you'll get an error because you only get the first segment before the space.
Seriously, read this and especially this. Really.
And please don't do things like
CMD=whatever you get from user input; $CMD;
You are begging for trouble. Don't execute arbitrary string from the user.
Both above answers already answered your question. So, i'm going a bit more verbose.
In your terminal is running the bash interpreter (probably). This is the program which parses your input line(s) and doing "things" based on your input.
When you enter some line the bash start doing the following workflow:
parsing and lexical analysis
expansion
brace expansion
tidle expansion
variable expansion
artithmetic and other substitutions
command substitution
word splitting
filename generation (globbing)
removing quotes
Only after all above the bash
will execute some external commands, like ls or dir.sh... etc.,
or will do so some "internal" actions for the known keywords and builtins like echo, for, if etc...
As you can see, the second last is the filename generation (globbing). So, in your case - if the test* matches some files, your bash expands the willcard characters (aka does the globbing).
So,
when you enter dir.sh test*,
and the test* matches some files
the bash does the expansion first
and after will execute the command dir.sh with already expanded filenames
e.g. the script get executed (in your case) as: dir.sh test.pas test.swift
BTW, it acts exactly with the same way for your ls example:
the bash expands the ls test* to ls test.pas test.swift
then executes the ls with the above two arguments
and the ls will print the result for the got two arguments.
with other words, the ls don't even see the test* argument - if it is possible - the bash expands the wilcard characters. (* and ?).
Now back to your script: add after the shebang the following line:
echo "the $0 got this arguments: $#"
and you will immediatelly see, the real argumemts how your script got executed.
also, in such cases is a good practice trying to execute the script in debug-mode, e.g.
bash -x dir.sh test*
and you will see, what the script does exactly.
Also, you can do the same for your current interpreter, e.g. just enter into the terminal
set -x
and try run the dir.sh test* = and you will see, how the bash will execute the dir.sh command. (to stop the debug mode, just enter set +x)
Everbody is giving you valuable advice which you should definitely should follow!
But here is the real answer to your question.
To pass unexpanded arguments to any executable you need to single quote them:
./your_script '*'
The best solution I have is to use the eval command, in this way:
#!/bin/bash
cmd="some command \"with_quetes_and_asterisk_in_it*\""
echo "$cmd"
eval $cmd
The eval command takes its arguments and evaluates them into the command as the shell does.
This solves my problem when I need to call a command with asterisk '*' in it from a script.
I have n files, named f1, f2, ..., fn. For each of these files, I have to execute a sed command, and name the new files as file1, file2, ..., filen.
I need the new files to keep the same number as their original ones. Can anyone help?
Here's what I've tried so far:
#!/bin/sh
for element in *
do
echo "$element" sed -n '/Col3/p' $element > Quest $element
done
If we assume that all the your files are in the form in your question...
$ ls -l
total 2
-rw-r--r-- 1 ghoti wheel 0 Jan 3 13:20 f1
-rw-r--r-- 1 ghoti wheel 0 Jan 3 13:20 f2
-rw-r--r-- 1 ghoti wheel 0 Jan 3 13:20 f3
-rw-r--r-- 1 ghoti wheel 0 Jan 3 13:20 f4
then you're on the right track with a for loop. But you probably want to narrow your search to only the files that are important to you.
In bash, you can use extglob to control this sort of thing. For example:
#!/usr/bin/env bash
shopt -s extglob
for file in +([a-z])+([0-9]); do
echo "Old: $file / New: file${file##[a-z]}"
done
This matches any files whose names consist of letters followed by numbers.
If, on the other hand, you want to make this portable, so that it will work in a POSIX shell (since in your question you've specified /bin/sh), you might put the detection into the loop itself:
#!/bin/sh
for file in *; do
if ! expr "$file" : '[a-z][a-z]*[0-9][0-9]*$' >/dev/null; then
continue
fi
echo "Old: $file / New: file${file##[a-z]}"
done
In both of these examples, we use POSIX "Parameter Expansion" to strip off the letters at the beginning of the filename.
#!/bin/env bash
for FILE in *
do
[[ "$FILE" =~ [0-9]+$ ]] && mv "$FILE" file${BASH_REMATCH[0]}
done
The expression within [[ ]] is a test, which tests for a match against a regular expression, which looks for a string ending in a number. If the match is successful, the matched number can be found in the bash array variable BASH_REMATCH at index 0. The part after && is executed if the test succeeds, and renames the file to fileNN,
It has been a long time since I did much bash script writing.
This is a bash script to copy and rename files by deleting all before the first period delimiter:
#!/bin/bash
mkdir fullname
mv *.audio fullname
cd fullname
for x in * ;
do
cp $x ../`echo $x | cut -d "." -f 2-`
done
cd ..
ls
It works well for file names with no embedded spaces but not for those with spaces.
How can I change the code to fix this simple Linux bash script? Any suggestions for improving the code for other reasons would also be welcome.
Example filenames, some with embedded spaces and some not (from link)
http://www.homenetvideo.com/demo/index.php?/Radio%20%28VLC%29
Ambient.A6.SOMA Space Station.audio
Blues.B9.Blues Radio U.K.audio
Classical.K3.Radio Stephansdom - Vienna.audio
College.CI.KDVS U of California, Davis.audio
Country.Q1.K-FROG.audio
Easy.G4.WNYU.audio
Eclectic.M2.XPN.audio
Electronica.E2.Rinse.audio
Folk.F1.Radionomy.audio
Hiphop.H1.NPR.audio
Indie.I4.WAUG.audio
Jazz.J6.KCSM.audio
Latin.L3.Mega.audio
Misc.X7.Gaydio.audio
News.N9.KQED.audio
Oldies.O1.Lonestar.audio
OldTime.Y1.Roswell.audio
Progressive.P1.Aural Moon.audio
Rock.R8.WXRT.audio
Scanner.Z3.Montreal.audio
Soul.S1.181.FM.audio
Talk.T2.TWiT.audio
World.W3.Persian.audio
http://lh5.googleusercontent.com/-QjLEiAtT4cw/U98_UFcWvvI/AAAAAAAABv8/gyPhbg8s7Bw/w681-h373-no/homenet-radio.png
Whenever you deal with file names that might have spaces in them, you must reference them as "$x" rather than just $x. That's what's causing your cp command to fail.
Your echo command is also problematic. Although echo does the right thing for simple spaces - it echoes a file named A B C as A B C - it will still fail if you have more than one consecutive space in the name, or whitespace that isn't a simple space character.
Instead of passing the file names to external programs for processing, which always requires getting them through the whitespace-hostile command line, you should use bash built-in functions for string manipulations wherever possible, e.g. ${x%%foo}, ${x#bar} and similar functions. The man page describes them under "Parameter expansion".
Here's my suggestion:
#!/bin/bash
shopt -s nullglob
mkdir fullname
mv *.audio fullname
(
cd fullname || exit
for x in *; do
cp "$x" "../${x#*.}"
done
)
ls
nullglob prevents * from presenting itself if no file matches it. Just optional.
() summons a subshell and saves you from changing back to another directory.
|| exit terminates the subshell if cd fails to change directory.
${x#*.} removes the <first>. from $x and expands it.
The script goal is simple.
I have many directory which contains some captured traffic files.
I want to run a command for each directory. So I came up with a script. But I don't know why the script is run only with the first match.
#!/bin/bash
# Collect throughput from a group of directory containing capture files
# Group of directory can be specify by pattern
# Usage: ./collectThroughputList [regex]
# [regex] is the name pattern of the group of directory
for DIR in $( ls -d $1 ); do
if test -d "$DIR"; then
echo Collecting throughputs from directory: "$DIR"
( sh collectThroughput.sh $DIR > $DIR.txt )
fi
done
echo Done\!
I try it with:
for DIR in $1; do
or
for DIR in `ls -d $1`; do
or
for DIR in $( ls -d "$1" ); do
or
for DIR in $( ls -d $1 ); do
But the result is the same. The for loop runs only one time.
Finally I found this one and did some tricks for it to work. However, I would like to know why my first script doesn't work.
find *Delay50ms* -type d -exec bash -c "cd '{}' && echo enter '{}' && ../collectThroughput.sh ../'{}' > ../'{}'.txt" \;
"*Delay*" is the directory pattern name that I want to run the command with.
Thanks for pointing out the issues.
Since you want to find all sub-directories under $1, use it like this:
for DIR in $(find $1 -type d)
Problem
Most probably the problem you are encountering is due to the fact that you are trying to use some kind of pattern like * as argument to your script.
Running it with something like:
my_script *
What's happening here is, that the shell will expand * prior to calling your script.
Thus after word splitting has been performed $1 in your script will just reference the first entry returned by ls.
Example
Given the following directory layout:
directory_a
directory_b
directory_c
Calling my_script * will result in:
my_script directory_a directory_b directory_c
being called thus your loop just iterating over $(ls -d directory_a) which in fact is nothing else but directory_a alone.
Solution
To have the program run with $1=* you would have to escape the * prior to calling your script.
Try running:
my_script \*
To see it effectively does what it is intended to do then. This way $1 in your script will contain * instead of directory_a which most probably is the way you wanted your script to work.
as mikyra has pointed out, the shell expands your argument * to all entries in your directory prior to passing it to your script.
if you want shell-expansion of your wildcards (e.g. * matches all but hidden files), you could simply leave the expansion to the shell and use the result, by iterating over all arguments, rather than just the first one:
for DIR in $#; do
# ...
done
if you want to do the expansion yourself (e.g. because the pattern should be applied only to a pre-filtered list or to files in a different directory, or because you want regex-expansion rather than shell globbing), you have to protect the argument from being expanded by the shell, either using backslash notation (like mikyra's \*) or by using quotes (which is often easier to use):
my_script "*"