I have such a bash script. I want to gzip all .ppm files under a directory to another directory. For this reason I have written such a bash script:
cd /export/students/sait/12-may
for next_file in $(find . -type f ! -name *.ppm )
do
/bin/gzip -f -c $next_file > /export/students/sait/12-may-yedek/$next_file.gz
done
When I execute this script, I get such error:
/usr/bin/find: Argument list too long
How can I fix this problem?
Quote this part : *.ppm to prevent filename globbing, and also remove the ! as you want to find files with .ppm extension, not the other way around.
find . -type f -name '*.ppm'
Instead of running a loop you could do it with single find command which would provide white space safety:
find /export/students/sait/12-may -type f -name '*.ppm' -exec sh -c '/bin/gzip -f -c "$0" > "$0".gz' {} \;
Related
Variable causing issue while doing the test command in unix Command is::
find $PWD -type d -exec sh -c 'test "{}" ">" "$PWD/$VersionFolders"' \; -print|wc -l`
Input Values-
Here $PWD- Current Directory
b1_v.1.0
b1_v.1.2
b1_v.1.3
b1_v.1.4
Given Version folder as $VersionFolders
b1_v.1.2
The Command should check if any folders exist in current directory which is greater than the give version folder and it should count or display.
This approach has to be consider with out date or time created of folders.
Expected Output-
b1_v.1.3
b1_v.1.4
If I give hard code Directories its working fine. But when I pass it like as variable.it give all folders.
working fine this commend-
find $PWD -type d -exec sh -c 'test "{}" ">" "$PWD/b1_v.1.2"' \; -print|wc -l`
Not working this command with variable-
find $PWD -type d -exec sh -c 'test "{}" ">" "$PWD/$VersionFolders"' ; -print|wc -l`
The variable $VersionFolders won't get expanded inside the single quotes, and apparently you did not export it to make it visible to subprocesses.
An obscure but common hack is to put it in $0 (which nominally should contain the name of the shell itself, and is the first argument after sh -c '...') because that keeps the code simple.
find . -type d \
-exec sh -c 'test "$1" ">" "$0"' \
"$VersionFolders" {} \; \
-print | wc -l
But as #chepner remarks, you can run -exec test {} ">" "$VersionFolders" \; directly.
The shell already does everything in the current directory, so you don't need to spell out $PWD. Perhaps see also What exactly is current working directory?
I've seen a few answers regarding this, but as a newbie, I don't really understand how to implement that in my script.
it should be pretty easy (for those who can stuff like this)
I'm using a simple
for f in "/drive1/"images*.{jpg,png}; do
but this is simply overloading and giving me
Argument list too long
How is this easiest solved?
Argument list too long workaroud
Argument list length is something limited by your config.
getconf ARG_MAX
2097152
But after discuss around differences between bash specifics and system (os) limitations (see comments from that other guy), this question seem wrong:
Regarding discuss on comments, OP tried something like:
ls "/simple path"/image*.{jpg,png} | wc -l
bash: /bin/ls: Argument list too long
This happen because of OS limitation, not bash!!
But tested with OP code, this work finely
for file in ./"simple path"/image*.{jpg,png} ;do echo -n a;done | wc -c
70980
Like:
printf "%c" ./"simple path"/image*.{jpg,png} | wc -c
Reduce line length by reducing fixed part:
First step: you could reduce argument length by:
cd "/drive1/"
ls images*.{jpg,png} | wc -l
But when number of file will grow, you'll be buggy again...
More general workaround:
find "/drive1/" -type f \( -name '*.jpg' -o -name '*.png' \) -exec myscript {} +
If you want this to NOT be recursive, you may add -maxdepth as 1st option:
find "/drive1/" -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) \
-exec myscript {} +
There, myscript will by run with filenames as arguments. The command line for myscript is built up until it reaches a system-defined limit.
myscript /drive1/file1.jpg '/drive1/File Name2.png' /drive1/...
From man find:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca‐
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of `{}'
Inscript sample
You could create your script like
#!/bin/bash
target=( "/drive1" "/Drive 2/Pictures" )
[ "$1" = "--run" ] && exec find "${target[#]}" -type f \( -name '*.jpg' -o \
-name '*.png' \) -exec $0 {} +
for file ;do
echo Process "$file"
done
Then you have to run this with --run as argument.
work with any number of files! (Recursively! See maxdepth option)
permit many target
permit spaces and special characters in file and directrories names
you could run same script directly on files, without --run:
./myscript hello world 'hello world'
Process hello
Process world
Process hello world
Using pure bash
Using arrays, you could do things like:
allfiles=( "/drive 1"/images*.{jpg,png} )
[ -f "$allfiles" ] || { echo No file found.; exit ;}
echo Number of files: ${#allfiles[#]}
for file in "${allfiles[#]}";do
echo Process "$file"
done
There's also a while read loop:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) |
while IFS= read -r file; do
or with zero terminated files:
find "/drive1/" -maxdepth 1 -mindepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) -print0 |
while IFS= read -r -d '' file; do
I am attempting to move some video files of mine into new subdirectories while also renaming them on my Unraid system. The files all follow a similar naming convention:
featurette name-featurette.mkv
I would like to move these files from their current directory to a subdirectory and rename them like this:
featurettes/featurette name.mkv
I am able to create the directories and relocate the files using find and execdir:
find . -type f -name *-featurette.mkv -maxdepth 2 -execdir mkdir ./featurettes/ \;
find . -type f -name *-featurette.mkv -maxdepth 2 -execdir mv {} ./featurettes/ \;
I am struggling with the renaming piece. I've tried the rename command but am unable to get it to work within the featurettes directory, let alone from two directories above, which is where I'd like to execute the command. I've tried the following command within the featurettes directory:
rename \-featurette.mkv .mkv *
However I get the error:
invalid option -- 'f'
I thought by escaping the dash I could avoid that issue, but it doesn't appear to work. Any advice on how to remove this pattern from all files within subdirectories matching it would be very much appreciated.
From man rename you see this command gets options and 3 positional parameters:
SYNOPSIS
rename [options] expression replacement file...
So in your case the first parameter is being interpreted as an option. You may use this syntax:
rename -- '-featurette' '' *-featurette.mkv
to rename the files. -- indicates that any options are over and what follows are only positional parameters.
Totally, to copy the files with one mv process and rename them:
mkdir -p target/dir
find . -maxdepth 2 -type f -name "*-featurette.mkv" -exec mv -t target/dir {} +
cd target/dir && rename -- '-featurette' '' *-featurette.mkv
If you want to rename many files located into different subdirectories, you can use this syntax:
find . -name "*-featurette.mkv" -print0 | xargs -r0 rename -- '-featurette' ''
find . \
-maxdepth 2 \
-type f \
-name '*-featurette.mkv' \
-execdir sh -c '
echo mkdir -p ./featurettes/
echo mv -- "$#" ./featurettes/
' _ {} \+
Issue with your implementations I fixed or improved:
-maxdepth 2 must precede -type f
-name '*-featurette.mkv' must have the pattern quoted to prevent the shell to expand globb it.
-execdir is best used with an inline shell, so it can also process multiple arguments from the same directory
Also keep in mind that while running a command with -execdir, find will cd to that directory. It means that mv -- "$#" ./featurettes/' will move files into the ./featurettes/' directory relative to were -execdir has just cd.
Version which also rename files while moving:
( has no echo dry-run protection, so use only if you are sure it does what you want )
#!/usr/bin/env sh
find . \
-maxdepth 2 \
-depth \
-name '*-featurette.mkv' \
-type f \
-execdir sh -c '
mkdir -p featurettes
for arg
do
basename=${arg##*/}
mv -- "$basename" "./featurettes/${basename%-featurette.mkv}.mkv"
done
' _ {} +
You can use Bash's shell parameter expansion feature to get the part of the file name, for example:
$> filename=name-featurette.mkv
$> echo ${filename%-*} #To print first part before '-'
name
$> echo ${filename##*.} #To get the extension
mkv
$> echo ${filename#*-} #To print the part after '-' with extension
featurette.mkv
With this and slightly modifying your find command, you should be able to move+rename your files:
find . -type f -name '*-featurette.mkv' -maxdepth 2 -execdir sh -c 'f="{}"; mv -- "$f" ./featurettes/"${f%-*}.mkv"' \;
In fact you should be able to combine both the find command into one to create_dir, move and rename file.
find . -type f -name '*-featurette.mkv' -maxdepth 2 -execdir sh -c 'f="{}"; mkdir ./featurettes; mv -- "$f" ./featurettes/"${f%-*}.mkv"' \;
This question already has answers here:
Recursively change file extensions in Bash
(6 answers)
Closed 6 years ago.
The user inputs a file type they are looking for; it is stored in $arg1; the file type they would like to change them is stored as $arg2. I'm able to find what I'm looking for, but I'm unsure of how to keep the filename the same but just change the type... ie., file1.txt into file1.log.
find . -type f -iname "*.$arg1" -exec mv {} \;
To enable the full power of shell parameter expansions, you can call bash -c in your exec action:
find . -type f -iname "*.$arg1" \
-exec bash -c 'echo mv "$1" "${1/%.*/$1}"' _ {} "$arg2" \;
We add {} and "$arg2" as a parameters to bash -c, so they become accessible within the command as $0 and $1. ${0%.*} removes the extension, to be replaced by whatever $arg2 expands to.
As it is, the command just prints the mv commands it would execute; to actually rename the files, the echo has to be removed.
The quoting is relevant: the argument to bash -c is in single quotes to prevent $0 and $1 from being expanded prematurely, and the two arguments to mv, and arg2 are also quoted to deal with file names with spaces in them.
Combining the find -exec bash idea with the bash loop idea, you can use the + terminator on the -exec to tell find to pass multiple filenames to a single invocation of the bash command. Pass the new type as the first argument - which shows up in $0 and so is conveniently skipped by a for loop over the rest of the command-line arguments - and you have a pretty efficient solution:
find . -type f -iname "*.$arg1" -exec bash -c \
'for arg; do mv "$arg" "${arg%.*}.$0"; done' "$arg2" {} +
Alternatively, if you have either version of the Linux rename command, you can use that. The Perl one (a.k.a. prename, installed by default on Ubuntu and other Debian-based distributions; also available for OS X from Homebrew via brew install rename) can be used like this:
find . -type f -iname "*.$arg1" -exec rename 's/\Q'"$arg1"'\E$/'"$arg2"'/' {} +
That looks a bit ugly, but it's really just the s/old/new/ substitution command familiar from many UNIX tools. The \Q and \E around $arg1 keep any weird characters inside the suffix from being interpreted as regular expression metacharacters that might match something unexpected; the $ after the \E makes sure the pattern only matches at the end of the filename.
The pattern-based version installed by default on Red Hat-based Linux distros (Fedora, CentOS, etc) is simpler:
find . -type f -iname "*.$arg1" -exec rename ".$arg1" ".$arg2" {} +
but it's also dumber: if you rename .com .exe stackoverflow.com_scanner.com, you'll get a file named stackoverflow.exe_scanner.exe.
I would do it like so:
find . -type f -iname "*.$arg1" -print0 |\
while IFS= read -r -d '' file; do
mv -- "$file" "${file%$arg1}$arg2"
done
I took your find command and fed its output to a while loop. Within that loop, I am doing the actual renaming. This way I have the name of the file as a variable that I can manipulate using bash's string manipulation operations.
If you have perl based rename command
Sample directory:
$ find
.
./a"bn.txt
./t2.abc
./abc
./abc/t1.txt
./abc/list.txt
./a bc.txt
Sample args:
$ arg1='txt'
$ arg2='log'
Dry run:
$ find -type f -iname "*.$arg1" -exec rename -n "s/$arg1$/$arg2/" {} +
rename(./a"bn.txt, ./a"bn.log)
rename(./abc/t1.txt, ./abc/t1.log)
rename(./abc/list.txt, ./abc/list.log)
rename(./a bc.txt, ./a bc.log)
Remove -n option once it is okay:
$ find -type f -iname "*.$arg1" -exec rename "s/$arg1$/$arg2/" {} +
$ find
.
./a bc.log
./t2.abc
./abc
./abc/list.log
./abc/t1.log
./a"bn.log
I was helped out today with a command, but it doesn't seem to be working. This is the command:
find /home/me/download/ -type f -name "*.rm" -exec ffmpeg -i {} -sameq {}.mp3 && rm {}\;
The shell returns
find: missing argument to `-exec'
What I am basically trying to do is go through a directory recursively (if it has other directories) and run the ffmpeg command on the .rm file types and convert them to .mp3 file types. Once this is done, remove the .rm file that has just been converted.
A -exec command must be terminated with a ; (so you usually need to type \; or ';' to avoid interpretion by the shell) or a +. The difference is that with ;, the command is called once per file, with +, it is called just as few times as possible (usually once, but there is a maximum length for a command line, so it might be split up) with all filenames. See this example:
$ cat /tmp/echoargs
#!/bin/sh
echo $1 - $2 - $3
$ find /tmp/foo -exec /tmp/echoargs {} \;
/tmp/foo - -
/tmp/foo/one - -
/tmp/foo/two - -
$ find /tmp/foo -exec /tmp/echoargs {} +
/tmp/foo - /tmp/foo/one - /tmp/foo/two
Your command has two errors:
First, you use {};, but the ; must be a parameter of its own.
Second, the command ends at the &&. You specified “run find, and if that was successful, remove the file named {};.“. If you want to use shell stuff in the -exec command, you need to explicitly run it in a shell, such as -exec sh -c 'ffmpeg ... && rm'.
However you should not add the {} inside the bash command, it will produce problems when there are special characters. Instead, you can pass additional parameters to the shell after -c command_string (see man sh):
$ ls
$(echo damn.)
$ find * -exec sh -c 'echo "{}"' \;
damn.
$ find * -exec sh -c 'echo "$1"' - {} \;
$(echo damn.)
You see the $ thing is evaluated by the shell in the first example. Imagine there was a file called $(rm -rf /) :-)
(Side note: The - is not needed, but the first variable after the command is assigned to the variable $0, which is a special variable normally containing the name of the program being run and setting that to a parameter is a little unclean, though it won't cause any harm here probably, so we set that to just - and start with $1.)
So your command could be something like
find -exec bash -c 'ffmpeg -i "$1" -sameq "$1".mp3 && rm "$1".mp3' - {} \;
But there is a better way. find supports and and or, so you may do stuff like find -name foo -or -name bar. But that also works with -exec, which evaluates to true if the command exits successfully, and to false if not. See this example:
$ ls
false true
$ find * -exec {} \; -and -print
true
It only runs the print if the command was successfully, which it did for true but not for false.
So you can use two exec statements chained with an -and, and it will only execute the latter if the former was run successfully.
Try putting a space before each \;
Works:
find . -name "*.log" -exec echo {} \;
Doesn't Work:
find . -name "*.log" -exec echo {}\;
I figured it out now. When you need to run two commands in exec in a find you need to actually have two separate execs. This finally worked for me.
find . -type f -name "*.rm" -exec ffmpeg -i {} -sameq {}.mp3 \; -exec rm {} \;
You have to put a space between {} and \;
So the command will be like:
find /home/me/download/ -type f -name "*.rm" -exec ffmpeg -i {} -sameq {}.mp3 && rm {} \;
Just for your information:
I have just tried using "find -exec" command on a Cygwin system (UNIX emulated on Windows), and there it seems that the backslash before the semicolon must be removed:
find ./ -name "blabla" -exec wc -l {} ;
For anyone else having issues when using GNU find binary in a Windows command prompt. The semicolon needs to be escaped with ^
find.exe . -name "*.rm" -exec ffmpeg -i {} -sameq {}.mp3 ^;
You need to do some escaping I think.
find /home/me/download/ -type f -name "*.rm" -exec ffmpeg -i {} \-sameq {}.mp3 \&\& rm {}\;
Just in case anyone sees a similar "missing -exec args" in Amazon Opsworks Chef bash scripts, I needed to add another backslash to escape the \;
bash 'remove_wars' do
user 'ubuntu'
cwd '/'
code <<-EOH
find /home/ubuntu/wars -type f -name "*.war" -exec rm {} \\;
EOH
ignore_failure true
end
Also, if anyone else has the "find: missing argument to -exec" this might help:
In some shells you don't need to do the escaping, i.e. you don't need the "\" in front of the ";".
find <file path> -name "myFile.*" -exec rm - f {} ;
Both {} and && will cause problems due to being expanded by the command line. I would suggest trying:
find /home/me/download/ -type f -name "*.rm" -exec ffmpeg -i \{} -sameq \{}.mp3 \; -exec rm \{} \;
In my case I needed to execute "methods" from by bash script, which does not work when using -exec bash -c, so I add another solution I found here, as well:
UploadFile() {
curl ... -F "file=$1"
}
find . | while read file;
do
UploadFile "$file"
done
This thread pops up first when searching for solutions to execute commands for each file from find, so I hope it's okay that this solution does not use the -exec argument
I got the same error when I left a blank space after the ending ; of an -exec command.So, remove blank space after ;
If you are still getting "find: missing argument to -exec" try wrapping the execute argument in quotes.
find <file path> -type f -exec "chmod 664 {} \;"