Regarding zip, I'm looking for a way to ensure a file is added to an archive independent of what has been passed to the archive creation exclusion list.
In my instance, I've developed an application that allows users to specify their own exclusion filter when creating a zip.
However, I need to ensure a couple custom files are always added to the archive, independent of what was specified in the filter.
For example:
Filters Specified: "*.bar"
File I need to add: foo.bar
So if I execute
zip -rq -i foo.bar -x "*.bar"
foo.bar will not be included in the archive.
So it comes down to:
How can one override what is in the exclusion list for the few files I need added?
Alternatively, could one develop an exclusion expression that combines what the user specified something that effectively says omit it all but foo.bar?
A solution I've come up with is to take two passes at it - first create an archive with the exclusion list and then grow the archive by foo.bar, but I'm looking for a way to do it in a single shot.
Thanks
The -i option is used to include only the specified files. With pattern "foo.bar" we only include the file named foo.bar leaving others.
The -x option is used to exclude files. With pattern *.bar we exclude the foo.bar file that we included before.
Also you must consider that the patterns will be resolved as pathname expansion not as POSIX or Perl regular expresion. If you want do patterns more complex you can use extended patterns.
By example:
shopt -s extglob
zip -rq foo.zip . -i ?(foo.bar|!(*.bar))
shopt -u extglob
Or:
shopt -s extglob
zip -rq foo.zip . -x !(!(*.bar)|foo.bar)
shopt -u extglob
But the last commands aren't recursive because the patterns doesn't contemplate subdirectories. Although we use -r option we include only the files in pattern. The manual page tell us about write \ before of * for apply recursively, but not work with all patterns.
Also you can use other commands to get list of files to include.
By example:
zip -rq foo.zip . -i `find ./ -not -name "*.bar" -o -name "foo.bar"`
Or:
zip -r foo.zip . -x `find ./ -name "*.bar" -a -not -name "foo.bar"`
Related
This question already has answers here:
Bash move * to subfolder fail: cannot move to a subdirectory of itself
(5 answers)
Closed 1 year ago.
How to move content of a folder to its subfolder.
When I run this:
mv xyz/* xyz/archive
I get this notice:
mv: cannot move 'xyz/archive' to a subdirectory of itself,
'xyz/archive/archive'
Is is possible to exclude archive folder from xyz/* selection ?
You can use extended globs to exclude archive from your pattern as follows:
shopt -s extglob
mv !(archive) archive
This will move everything in the current folder, except for hidden files, to archive.
If you want to move hidden files starting with a . to the archive folder too, you need to set the dotglob option as described in bash manual under Filename Expansion:
shopt -s dotglob
Here is another method using find:
find . -maxdepth 1 -type f ! -name "archive" -exec mv -t archive {} +
Although the first method is preferred, IMO, since bash builtins are perfectly capable of what find is doing here.
Yes, it is possible to exclude archive from the selection.
Enable the extended globbing option
shopt -s extglob
Run the following command that will move everything but archive
mv !(archive) archive
It will now move everything from xyz to archive
PS: I did run the mv command when I was inside the xyz folder.
I'm looking to search for files of a specific name, modify the name to the full path and then copy the results to another folder.
Is it possible to update each find result with the full path as the file name; i.e.
./folder/subfolder/my-file.csv
becomes
folder_subfolder_my-file.csv
I am listing the files using the following and would like to script it.
find . -name my-file.csv -exec ls {} \;
Since you're using bash, you can take advantage of globstar and use a for loop:
shopt -s globstar # set globstar option
for csv in **/my-file.csv; do
echo "$csv" "${csv//\//_}"
done
shopt -u globstar # unset the option if you don't want it any more
With globstar enabled, ** does a recursive search (similar to the basic functionality of find).
"${csv//\//_}" is an example of ${var//match/replace}, which does a global replacement of all instances of match (here an escaped /) with replace.
If you're happy with the output, then change the echo to mv.
Just to demonstrate how to do this with find;
find . -type f -exec bash -c '
for file; do
f=${file#./}
cp "$file" "./${f//\//_}"
done' _ {} +
The Bash pattern expansion ${f//x/y} replaces x with y throughout. Because find prefixes each found file with the path where it was found (here, ./) we trim that off in order to avoid doing mv "./file" "._file". And because the slash is used in the parameter expansion itself, we need to backslash the slash we want the shell to interpret literally. Finally, because this parameter expansion syntax is a Bash-only extension, we use bash rather than sh.
Obviously, if you want to rename rather than copy, replace cp with mv.
If your find does not support -exec ... + this needs to be refactored somewhat (probably to use xargs); but it should be supported on any reasonably modern platform.
With perl's rename command ...
$ prename
Usage: rename [-v] [-n] [-f] perlexpr [filenames]
... you can rename multiple files by applying a regular expression. rename also accepts file names via stdin:
find ... | rename -n 's#/#_#g'
Check the results and if they are fine, remove -n.
Say I want to copy the contents of a directory excluding files and folders whose names contain the word 'Music'.
cp [exclude-matches] *Music* /target_directory
What should go in place of [exclude-matches] to accomplish this?
In Bash you can do it by enabling the extglob option, like this (replace ls with cp and add the target directory, of course)
~/foobar> shopt extglob
extglob off
~/foobar> ls
abar afoo bbar bfoo
~/foobar> ls !(b*)
-bash: !: event not found
~/foobar> shopt -s extglob # Enables extglob
~/foobar> ls !(b*)
abar afoo
~/foobar> ls !(a*)
bbar bfoo
~/foobar> ls !(*foo)
abar bbar
You can later disable extglob with
shopt -u extglob
The extglob shell option gives you more powerful pattern matching in the command line.
You turn it on with shopt -s extglob, and turn it off with shopt -u extglob.
In your example, you would initially do:
$ shopt -s extglob
$ cp !(*Music*) /target_directory
The full available extended globbing operators are (excerpt from man bash):
If the extglob shell option is enabled using the shopt builtin, several extended
pattern matching operators are recognized.A pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:
?(pattern-list)
Matches zero or one occurrence of the given patterns
*(pattern-list)
Matches zero or more occurrences of the given patterns
+(pattern-list)
Matches one or more occurrences of the given patterns
#(pattern-list)
Matches one of the given patterns
!(pattern-list)
Matches anything except one of the given patterns
So, for example, if you wanted to list all the files in the current directory that are not .c or .h files, you would do:
$ ls -d !(*#(.c|.h))
Of course, normal shell globing works, so the last example could also be written as:
$ ls -d !(*.[ch])
Not in bash (that I know of), but:
cp `ls | grep -v Music` /target_directory
I know this is not exactly what you were looking for, but it will solve your example.
If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to
find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec
find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/
A trick I haven't seen on here yet that doesn't use extglob, find, or grep is to treat two file lists as sets and "diff" them using comm:
comm -23 <(ls) <(ls *Music*)
comm is preferable over diff because it doesn't have extra cruft.
This returns all elements of set 1, ls, that are not also in set 2, ls *Music*. This requires both sets to be in sorted order to work properly. No problem for ls and glob expansion, but if you're using something like find, be sure to invoke sort.
comm -23 <(find . | sort) <(find . | grep -i '.jpg' | sort)
Potentially useful.
You can also use a pretty simple for loop:
for f in `find . -not -name "*Music*"`
do
cp $f /target/dir
done
In bash, an alternative to shopt -s extglob is the GLOBIGNORE variable. It's not really better, but I find it easier to remember.
An example that may be what the original poster wanted:
GLOBIGNORE="*techno*"; cp *Music* /only_good_music/
When done, unset GLOBIGNORE to be able to rm *techno* in the source directory.
My personal preference is to use grep and the while command. This allows one to write powerful yet readable scripts ensuring that you end up doing exactly what you want. Plus by using an echo command you can perform a dry run before carrying out the actual operation. For example:
ls | grep -v "Music" | while read filename
do
echo $filename
done
will print out the files that you will end up copying. If the list is correct the next step is to simply replace the echo command with the copy command as follows:
ls | grep -v "Music" | while read filename
do
cp "$filename" /target_directory
done
One solution for this can be found with find.
$ mkdir foo bar
$ touch foo/a.txt foo/Music.txt
$ find foo -type f ! -name '*Music*' -exec cp {} bar \;
$ ls bar
a.txt
Find has quite a few options, you can get pretty specific on what you include and exclude.
Edit: Adam in the comments noted that this is recursive. find options mindepth and maxdepth can be useful in controlling this.
The following works lists all *.txt files in the current dir, except those that begin with a number.
This works in bash, dash, zsh and all other POSIX compatible shells.
for FILE in /some/dir/*.txt; do # for each *.txt file
case "${FILE##*/}" in # if file basename...
[0-9]*) continue ;; # starts with digit: skip
esac
## otherwise, do stuff with $FILE here
done
In line one the pattern /some/dir/*.txt will cause the for loop to iterate over all files in /some/dir whose name end with .txt.
In line two a case statement is used to weed out undesired files. – The ${FILE##*/} expression strips off any leading dir name component from the filename (here /some/dir/) so that patters can match against only the basename of the file. (If you're only weeding out filenames based on suffixes, you can shorten this to $FILE instead.)
In line three, all files matching the case pattern [0-9]*) line will be skipped (the continue statement jumps to the next iteration of the for loop). – If you want to you can do something more interesting here, e.g. like skipping all files which do not start with a letter (a–z) using [!a-z]*, or you could use multiple patterns to skip several kinds of filenames e.g. [0-9]*|*.bak to skip files both .bak files, and files which does not start with a number.
this would do it excluding exactly 'Music'
cp -a ^'Music' /target
this and that for excluding things like Music?* or *?Music
cp -a ^\*?'complete' /target
cp -a ^'complete'?\* /target
I need to traverse a directory so starting in one directory and going deeper into difference sub directories. However I also need to be able to have access to each individual file to modify the file. Is there already a command to do this or will I have to write a script? Could someone provide some code to help me with this task? Thanks.
The find command is just the tool for that. Its -exec flag or -print0 in combination with xargs -0 allows fine-grained control over what to do with each file.
Example: Replace all foo's by bar's in all files in /tmp and subdirectories.
find /tmp -type f -exec sed -i -e 's/foo/bar/' '{}' ';'
for i in `find` ; do
if [ -d $i ] ; then do something with a directory ; fi
if [ -f $i ] ; then do something with a file etc. ; fi
done
This will return the whole tree (recursively) in the current directory in a list that the loop will go through.
This can be easily achieved by mixing find, xargs, sed (or other file modification command).
For example:
$ find /path/to/base/dir -type f -name '*.properties' | xargs sed -ie '/^#/d'
This will filter all files with file extension .properties.
The xargs command will feed the file path generated by find command into the sed command.
The sed command will delete all lines start with # in the files (feed by xargs).
Command combination in this way is very flexible.
For example, find command have different parameters so you can filter by user name, file size, file path (eg: under /test/ subfolder), file modification time.
Another dimension of flexibility is how and what to change in your file. For ex, sed command allows you to make changes on file in applying substitution (specify via regular expressions). Similarly, you can use gzip to compress the file. And so on ...
You would usually use the find command. On Linux, you have the GNU version, of course. It has many extra (and useful) options. Both will allow you to execute a command (eg a shell script) on the files as they are found.
The exact details of how to make changes to the file depend on the change you want to make to the file. That is probably best scripted, with find running the script:
POSIX or GNU:
find . -type f -exec your_script '{}' +
This will run your script once for a group of files with those names provided as arguments. If you want to do it one file at a time, replace the + with ';' (or \;).
I am assuming SearchMe is the example directory name you need to traverse completely.
I am also assuming, since it was not specified, the files you want to modify are all text file. Is this correct?
In such scenario I would suggest using the command:
find SearchMe -type f -exec vi {} \;
If you are not familiar with vi editor, just use another one (nano, emacs, kate, kwrite, gedit, etc.) and it should work as well.
Bash 4+
shopt -s globstar
for file in **
do
if [ -f "$file" ];then
# do some processing to your file here
# where the find command can't do conveniently
fi
done
Is there a way to use shell globbing to identify nested directories?
so if I have dir/dir1/dir2/dir3/dir4/dir5/.. and I have files under all of them, what is the equivalent globbing pattern to match all files under all directories, similar to - for example - ls -R
In Bash 4, with shopt -s globstar, and zsh you can use **/* which will include everything except hidden files. You can do shopt -s dotglob in Bash 4 or setopt dotglob in zsh to cause hidden files to be included.
In ksh, set -o globstar enables it. I don't think there's a way to include dot files implicitly, but I think **/{.[^.],}* works.
Specifically about git (gitignore, gitattributes, and commands that take filenames): if the pattern contains no slash, * wildcards will match deep. If it does contain a slash, git will call fnmatch with the FNM_PATHNAME flag, and simple wildcards won't match slashes. ** to match deep isn't supported. Maybe this kind of deep matching could be more widely supported with a new FNM_STARSTAR flag, and an implementation in glibc, gnulib and other places.
If you want to act on all the files returned by find, rather than just list them, you can pipe them to xargs:
find <directory> -type f | xargs ls
But this is only for commands that don't have a recursive flag.
You may try:
**/*.*
However it'll ignore hidden files (such as .git files). Sometimes it's a life-saver.
Read more at: What expands to all files in current directory recursively? at SO
You can use tree, it will show all folders recursively.
tree <path>
There is no way to do this with vanilla Bash, however most commands accept a -R or --recursive option to tell them to descend into directories.
If you simply want to list all files located anywhere within a directory or its sub-directories, you can use find.
To recursively find files (-type f) with a given directory:
find <directory> -type f