Some of my files are separated into different directories such as /apps, /games, /docs etc....
Within each directory, is a subdirectory called _CHECKSUM. Inside this directory, is a file called openssl.sh.
For example:
openssl sha1 /path/to/apps/*.iso | sed 's/\/.*.\///' > /path/to/apps/_CHECKSUM/sum.sha1
This outputs to a file called sum.sha1 within the _CHECKSUM directory, of which the contents could look like this:
SHA1(anApp.iso)= b398c8b175411e6174942d7b4acbc5c90473a852
SHA1(anotherApp.iso)= cc150483feed3d4b607749f31eddccefd0ba5478
SHA1(yetAnotherApp.iso)= d9682a2eca25b70dddf7a906374c27ee35614c7d
However, some directories contain multiple filetypes, so the script would have to look like this:
openssl sha1 /path/to/games/*.{7z,iso} | sed 's/\/.*.\///' > /path/to/games/_CHECKSUM/sum.sha1
producing something like this:
SHA1(myFaveGame.7z)= b398c8b175411e6174942d7b4acbc5c90473a852
SHA1(anotherGoodGame.iso)= cc150483feed3d4b607749f31eddccefd0ba5478
I don't want to always run these scripts manually, so I created the following script, /path/to/scripts/openssl_recursive.sh:
#!/bin/bash
# finds every openssl.sh recursively and executes it.
IFS=$'\n'
for file in $(find /path/to -name "openssl.sh" | sort -n)
do
echo "executing $file ..."
sh $file
echo "done.";
done
This seems to work fine for all directories where just one file type exists. However, for the openssl.sh scripts that contain multiple extensions, an empty sum.sha1 file is created.
Why is it that if I run the openssl.sh directly, it will create the correct result in sum.sha1 for directories with multiple filetypes, yet if I run the openssl_recursive.sh, this results in an empty sum.sha1?
as stated here, modern Debian and Ubuntu systems symlink sh to dash by default, which is a lighter version and lacks some advanced features.
So this may not be the same shell, and probably doesn't like "rich" wildcard constructs like *.{7z,iso}. You must have fallen into that category.
On the other hand, bash accepts those wildcards happily.
So a working solution is forcing the use of /bin/bash env variable:
#!/bin/bash
# finds every openssl.sh recursively and executes it.
IFS=$'\n'
for file in $(find /path/to -name "openssl.sh" | sort -n)
do
echo "executing $file ..."
/bin/bash $file
echo "done.";
done
Related
I have a large folder of jpegs, which I would like to rename sequentially to image01.jpg, image02.jpg...image533jpg etc.
I have tried using the following
find ‘/myImages/‘ -maxdepth 1 -name ‘*.jpg’ | sort -n | awk 'BEGIN{ x=1 }{printf "mv \"%s\" \”/myImages/image%04d.jpg\”\n”, $0, x++ }' | bash
which I got from here: http://www.algissalys.com/how-to/how-to-quickly-rename-modify-and-scale-all-images-in-a-directory-using-linux
However, this is only returning
>
And then nothing happens, any suggestions would be great.
The easiest way to do that is with rename which you can install with homebrew using:
brew install rename
Then, you can go into your directory containing the images and run:
rename --dry-run -X -e '$_ = "$N"' *jpg
Sample Output
'a.jpg' would be renamed to '1.jpg'
'article.jpg' would be renamed to '2.jpg'
'blob-0.jpg' would be renamed to '3.jpg'
'blob-1.jpg' would be renamed to '4.jpg'
'blob-2.jpg' would be renamed to '5.jpg'
'blob-3.jpg' would be renamed to '6.jpg'
If that looks correct, you can run it again without the --dry-run to actually do it, rather than just telling you what it will do.
If you want your names zero-padded, the easiest is to let rename work out how much padding you need automatically like this:
rename --dry-run -X -N ...01 -e '$_ = "$N"' *jpg
The benefits of using rename are that:
it is simple and powerful
it will warn you before overwriting any files
it can do a dry run and tell you what would happen without actually doing anything
If you want an explanation of the command '$_ = "$N"' then read on...
The rename command is actually a Perl script, so the part I mention above is just a Perl script enclosed in single quotes. The $N is just a Perl variable that expands to be a sequentially increasing number. The Perl special variable $_ is filled with the name of the current file before your little Perl script is executed, and crucially, you are expected to set it to the name you want that input file renamed as.
You could do that with a bash script. Say you have the following in a file called rename_images.
#!/bin/bash
declare -a FILESERIES
FILESERIES=(`ls $1`)
NUM=${#FILESERIES[#]}
NEWNAME=$2
EXT=$3
for (( i=0; i<$NUM ; i++))
do
FI=${FILESERIES[$i]}
NEWFILENAME=`echo $NEWNAME$i$EXT`
mv $FI $NEWFILENAME
done
To do what you need, run the script from within the folder with all the images as follows:
./rename_images '*.jpg' image .jpg
And you should be sorted.
I have this question after quite a day of searching the net, perhaps I'm doing something wrong , here is my script:
#!/bin/bash
shopt -s extglob
FILE_EXTENSIONS=properties\|xml\|sh\|sql\|ksh
SOURCE_FOLDER=$1
if [ -z "$SOURCE_FOLDER" ]; then
SOURCE_FOLDER=$(pwd)
fi # Set directory to current working folder if no input parameter.
for file in $SOURCE_FOLDER/**/*.*($FILE_EXTENSIONS)
do
echo Working with file: $file
done
Basically, I want to recursively get all the files filtered by a list of extensions within folders from a directory that is passed as an argument including the directory itself.
I would like to know if there is a way of doing this and how without the use of the find command.
Imagine I have this file tree:
bin/props.properties
bin/xmls.xml
bin/source/sources.sh
bin/config/props.properties
bin/config/folders/moreProps.xml
My script, as it is right now and running from /bin, would echo:
bin/source/sources.sh
bin/config/props.properties
bin/config/folders/moreProps.xml
Leaving the ones in the working path aside.
P.S. I know this can be done with find but I really want to know if there's another way for the sake of learning.
Thanks!
You can use find with grep, just like this:
#!/bin/bash
SOURCE_FOLDER=$1
EXTENSIONS="properties|xml|sh|sql|ksh"
find $SOURCE_FOLDER | grep -E ".(${EXTENSIONS})"
#or even better
find $SOURCE_FOLDER -regextype posix-egrep -regex ".*(${EXTENSIONS})"
I am teaching myself more (l)unix skills and wanted to see if I could begin to write a program that will eventually read all .gz files and expand them. However, I want it to be super dynamic.
#!/bin/bash
dir=~/derp/herp/path/goes/here
for file in $(find dir -name '*gz')
do
echo $file
done
So when I excute this file, I simply go
bash derp.sh.
I don't like this. I feel the script is too brittle.
How can I rework my for loop so that I can say
bash derp.sh ~/derp/herp/path/goes/here (1)
I tried re-coding it as follows:
for file in $*
However, I don't want to have to type in bash
derp.sh ~/derp/herp/path/goes/here/*.gz.
How could I rewrite this so I could simply type what is in (1)? I feel I must be missing something simple?
Note
I tried
for file in $*/*.gz and that obviously did not work. I appreciate your assistance, my sources have been a wrox unix text, carpentry v5, and man files. Unfortunately, I haven't found anything that will what I want.
Thanks,
GeekyOmega
for dir in "$#"
do
for file in "$dir"/*.gz
do
echo $file
done
done
Notes:
In the outer loop, dir is assigned successively to each argument given on the command line. The special form "$#" is used so that the directory names that contain spaces will be processed correctly.
The inner loop runs over each .gz file in the given directory. By placing $dir in double-quotes, the loop will work correctly even if the directory name contains spaces. This form will also work correctly if the gz file names have spaces.
#!/bin/bash
for file in $(find "$#" -name '*.gz')
do
echo $file
done
You'll probably prefer "$#" instead of $*; if you were to have spaces in filenames, like with a directory named My Documents and a directory named Music, $* would effectively expand into:
find My Documents Music -name '*.gz'
where "$#" would expand into:
find "My Documents" "Music" -name '*.gz'
Requisite note: Using for file in $(find ...) is generally regarded as a bad practice, because it does tend to break if you have spaces or newlines in your directory structure. Using nested for loops (as in John's answer) is often a better idea, or using find -print0 and read as in this answer.
The script goal is simple.
I have many directory which contains some captured traffic files.
I want to run a command for each directory. So I came up with a script. But I don't know why the script is run only with the first match.
#!/bin/bash
# Collect throughput from a group of directory containing capture files
# Group of directory can be specify by pattern
# Usage: ./collectThroughputList [regex]
# [regex] is the name pattern of the group of directory
for DIR in $( ls -d $1 ); do
if test -d "$DIR"; then
echo Collecting throughputs from directory: "$DIR"
( sh collectThroughput.sh $DIR > $DIR.txt )
fi
done
echo Done\!
I try it with:
for DIR in $1; do
or
for DIR in `ls -d $1`; do
or
for DIR in $( ls -d "$1" ); do
or
for DIR in $( ls -d $1 ); do
But the result is the same. The for loop runs only one time.
Finally I found this one and did some tricks for it to work. However, I would like to know why my first script doesn't work.
find *Delay50ms* -type d -exec bash -c "cd '{}' && echo enter '{}' && ../collectThroughput.sh ../'{}' > ../'{}'.txt" \;
"*Delay*" is the directory pattern name that I want to run the command with.
Thanks for pointing out the issues.
Since you want to find all sub-directories under $1, use it like this:
for DIR in $(find $1 -type d)
Problem
Most probably the problem you are encountering is due to the fact that you are trying to use some kind of pattern like * as argument to your script.
Running it with something like:
my_script *
What's happening here is, that the shell will expand * prior to calling your script.
Thus after word splitting has been performed $1 in your script will just reference the first entry returned by ls.
Example
Given the following directory layout:
directory_a
directory_b
directory_c
Calling my_script * will result in:
my_script directory_a directory_b directory_c
being called thus your loop just iterating over $(ls -d directory_a) which in fact is nothing else but directory_a alone.
Solution
To have the program run with $1=* you would have to escape the * prior to calling your script.
Try running:
my_script \*
To see it effectively does what it is intended to do then. This way $1 in your script will contain * instead of directory_a which most probably is the way you wanted your script to work.
as mikyra has pointed out, the shell expands your argument * to all entries in your directory prior to passing it to your script.
if you want shell-expansion of your wildcards (e.g. * matches all but hidden files), you could simply leave the expansion to the shell and use the result, by iterating over all arguments, rather than just the first one:
for DIR in $#; do
# ...
done
if you want to do the expansion yourself (e.g. because the pattern should be applied only to a pre-filtered list or to files in a different directory, or because you want regex-expansion rather than shell globbing), you have to protect the argument from being expanded by the shell, either using backslash notation (like mikyra's \*) or by using quotes (which is often easier to use):
my_script "*"
Is it possible to get, using Bash, a list of commands starting with a certain string?
I would like to get what is printed hitting <tab> twice after typing the start of the command and, for example, store it inside a variable.
You should be able to use the compgen command, like so:
compgen -A builtin [YOUR STRING HERE]
For example, "compgen -A builtin l" returns
let
local
logout
You can use other keywords in place of "builtin" to get other types of completion. Builtin gives you shell builtin commands. "File" gives you local filenames, etc.
Here's a list of actions (from the BASH man page for complete which uses compgen):
alias Alias names. May also be specified as -a.
arrayvar Array variable names.
binding Readline key binding names.
builtin Names of shell builtin commands. May also be specified as -b.
command Command names. May also be specified as -c.
directory Directory names. May also be specified as -d.
disabled Names of disabled shell builtins.
enabled Names of enabled shell builtins.
export Names of exported shell variables. May also be specified as -e.
file File names. May also be specified as -f.
function Names of shell functions.
group Group names. May also be specified as -g.
helptopic Help topics as accepted by the help builtin.
hostname Hostnames, as taken from the file specified by the HOSTFILE shell
variable.
job Job names, if job control is active. May also be specified as
-j.
keyword Shell reserved words. May also be specified as -k.
running Names of running jobs, if job control is active.
service Service names. May also be specified as -s.
setopt Valid arguments for the -o option to the set builtin.
shopt Shell option names as accepted by the shopt builtin.
signal Signal names.
stopped Names of stopped jobs, if job control is active.
user User names. May also be specified as -u.
variable Names of all shell variables. May also be specified as -v.
A fun way to do this is to hit M-* (Meta is usually left Alt).
As an example, type this:
$ lo
Then hit M-*:
$ loadkeys loadunimap local locale localedef locale-gen locate
lockfile-create lockfile-remove lockfile-touch logd logger login
logname logout logprof logrotate logsave look lorder losetup
You can read more about this in man 3 readline; it's a feature of the readline library.
If you want exactly how bash would complete
COMPLETIONS=$(compgen -c "$WORD")
compgen completes using the same rules bash uses when tabbing.
JacobM's answer is great. For doing it manually, i would use something like this:
echo $PATH | tr : '\n' |
while read p; do
for i in $p/mod*; do
[[ -x "$i" && -f "$i" ]] && echo $i
done
done
The test before the output makes sure only executable, regular files are shown. The above shows all commands starting with mod.
Interesting, I didn't know about compgen. Here a script I've used to do it, which doesn't check for non-executables:
#!/bin/bash
echo $PATH | tr ':' '\0' | xargs -0 ls | grep "$#" | sort
Save that script somewhere in your $PATH (I named it findcmd), chmod u+w it, and then use it just like grep, passing your favorite options and pattern:
findcmd ^foo # finds all commands beginning with foo
findcmd -i -E 'ba+r' # finds all commands matching the pattern 'ba+r', case insensitively
Just for fun, another manual variant:
find -L $(echo $PATH | tr ":" " ") -name 'pattern' -type f -perm -001 -print
where pattern specifies the file name pattern you want to use. This will miss commands that are not globally executable, but which you have permission for.
[tested on Mac OS X]
Use the -or and -and flags to build a more comprehensive version of this command:
find -L $(echo $PATH | tr ":" " ") -name 'pattern' -type f
\( \
-perm -001 -or \
\( -perm -100 -and -user $(whoami)\) \
\) -print
will pick up files you have permission for by virtue of owning them. I don't see a general way to get all those you can execute by virtue of group affiliation without a lot more coding.
Iterate over the $PATH variable and do ls beginningofword* for each directory in the path?
To get it exactly equivalent, you would need to filter out only executable files and sort by name (should be pretty easy with ls flags and the sort command).
What is listed when you hit are the binary files in your PATH that start with that string. So, if your PATH variable contains:
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/java/bin:/usr/lib/java/jre/bin:/usr/lib/qt/bin:/usr/share/texmf/bin:.
Bash will look in each of those directories to show you the suggestions once you hit . Thus, to get the list of commands starting with "ls" into a variable you could do:
MYVAR=$(ls /usr/local/bin/ls* /usr/bin/ls* /bin/ls*)
Naturally you could add all the other directories I haven't.