I am trying to copy an existing file (that is found in across directories) to a new file with a new name in Bash. For example, 'Preview.json' to 'Performance.json'. I have tried using
find * -type f -name 'Preview.json' -exec cp {} {}"QA" \;
But ended up with 'Preview.jsonQA'. (I am new to Bash.) I have tried moving the "QA" in front of the {} but I got errors because of an invalid path.
In an -exec predicate, the symbol {} represents a path that is being considered, starting at one of the starting-point directories designated in the command. Example: start/dir2/Preview.json. You can form other file names by either prepending or appending characters, but whether that makes sense depends on the details. In your case, appending produces commands such as
cp start/dir2/Preview.json start/dir2/Preview.jsonQA
which is a plausible command in the event that start/dir2/Preview.json exists. But cp does not automatically create directories in the destination path, so the result of prepending characters ...
cp start/dir2/Preview.json QAstart/dir2/Preview.json
... is not as likely to be accepted -- it depends on directory QAstart/dir2 existing.
I think what you're actually looking for may be cp commands of the form ...
cp start/dir2/Preview.json start/dir2/QAPreview.json
... but find cannot do this by itself.
For more flexibility in handling the file names discovered by find, pipe its output into another command. If you want to pass them as command-line arguments to another command, then you can interpose the xargs command to achieve that. The command on the receiving end of the pipe can be a shell function or a compound command if you wish.
For example,
# Using ./* instead of * ensures that file names beginning with - will not
# be misinterpreted as options:
find ./* -type f -name 'Preview.json' |
while IFS= read -r name; do # Read one line and store it in variable $name
# the destination name needs to be computed differently if the name
# contains a / character (the usual case) than if it doesn't:
case "${name}" in
*/*) cp "${name}" "${name%/*}/QA${name##*/}" ;;
*) cp "${name}" "QA${name}" ;;
esac
done
Note that that assumes that none of your directory names contain newline characters (the read command would split up newline-containing filenames). That's a reasonably safe assumption, but not absolutely safe.
Of course, you would generally want to have that in a script, not to try to type it on the fly on the command line.
Related
I am using find in a loop to search recursively for files of a specific extension, and then do something with that loop.
cd $DSJobs
jobs=$(find $DSJobs -name "*.dsx")
for j in jobs; do
echo "$j"
done
assuming $DSJobs is a relevent folder, the output of $j is "Jobs" one time. doesn't even repeat.
I want to list all *.dsx files in a folder recursively through subfolders as well.
How do Make this work?
Thanks
The idiomatic way to do this is:
cd "$DSJobs"
find . -name "*.dsx" -print0 | while IFS= read -r -d "" job; do
echo "$job"
done
The complication derives from the fact that space and newline are perfectly valid filename characters, so you get find to output the filenames separated by the null character (which is not allowed to appear in a filename). Then you tell read to use the null character (with -d "") as the delimiter while reading the names.
IFS= read -r var is the way to get bash to read the characters verbatim, without dropping any leading/trailing whitespace or any backslashes.
There are further complications regarding the use of the pipe, which may or may not matter to you depending on what you do inside the loop.
Note: take care to quote your variables, unless you know exactly when to leave the quotes off. Very detailed discussion here.
Having said that, bash can do this without find:
shopt -s globstar
cd "$DSJobs"
for job in **/*.dsx; do
echo "$job"
done
This approach removes all the complications of find | while read.
Incorporating #Gordon's comment:
shopt -s globstar nullglob
for job in "$DSJobs"/**/*.dsx; do
do_stuff_with "$job"
done
The "nullglob" setting is useful when no files match the pattern. Without it, the for loop will have a single iteration where job will have the value job='/path/to/DSJobs/**/*.dsx' (or whatever the contents of the variable) -- including the literal asterisks.
Since all you want is to find files with a specific extension...
find ${DSJobs} -name "*.dsx"
Want to do this for several directories?
for d in <some list of directories>; do
find ${d} -name ""*.dsx"
done
Want to do something interesting with the files?
find ${DSJobs} -name "*.dsx" -exec dostuffwith.sh "{}" \;
I have a question regarding the mv command.
In order to move and rename the file, I did mv file someDir/File2 in my terminal and it moved file into someDir with new name called File2.
However, when I do it with shell script, it sees the File2 part as a directory, instead of the new name for the file.
So I have two variables, NEWDir=newDir, NEWF=newName
for i in *.txt ; do
mv $i $NEWDIR/$NEWF
done
I run this script, it says the following:
mv: target 'newName' is not a directory.
mv requires the destination to be a directory only if more than one source argument is given.
In this case, that can be caused by your variables being split due to lack of quoting. Use double quotes -- as http://shellcheck.net/ directs -- around all expansions.
for i in *.txt ; do
mv "$i" "$NEWDIR/$NEWF"
done
Note that only the last file iterated over will actually be left behind with the given name -- the rest will be overwritten by their successors.
I am using the find command in a shell (bash) script to obtain a list of files matching a pattern and dump those filesnames to a text file.
find ./ -type f -name *.txt >> files.list
Which produces a file containing
helloworld.txt
letter.txt
document-1.txt
document-1.backup.txt
etc.
The find command isn't that important. The one I am actually using contains regex-es, and produces a sensible list of matching files to be used as input file to a program.
I want to tag each filename with a flag to mark whether it is a Type A file or a Type B file. ie, I want the output file files.list to look like this
helloworld.txt type=A
letter.txt type=A
document-1.txt type=B
document-1.backup.txt type=B
if helloworld and letter are Type A files, and document-1 is a Type B file.
I thought perhaps I could write a files.list.tmp file first, and then read it back in using another bash script processing it line by line... But that would make additional file and I don't really want to do that.
Can I use the >> operator to create a variable or something?
I'm not really sure what I should do here.
By the way, when this bash script is called, it already knows whether the files matching the regex are of type A or type B, as that is given as an argument to the script.
I thought perhaps I could write a files.list.tmp file first, and then read it back in using another bash script processing it line by line... But that would make additional file and I don't really want to do that
You can avoid creating temporary file by using process substitution:
./ascript.sh < <(find . -type f -name '*.txt')
And make your script read file names line by line.
You can pipe through the case that contains the choices:
find . -name '*.txt' | while read f; do type=UNKNOWN; case $f in *hello*) type=A;; *letter*) type=A;; *document*) type=B;; esac; echo $f type=$type; done
In the code above the choices are as follows
*hello*: type=A
*letter*: type=A
*document*: type=B
default: type=UNKNOWN
You can add as many *PATTERN*) type=TYPE;; entries as you wish. Given your files the output is:
./document-1.backup.txt type=B
./document-1.txt type=B
./letter.txt type=A
./helloworld.txt type=A
By the way, when this bash script is called, it already knows whether the files matching the regex are of type A or type B, as that is given as an argument to the script.
So, as your find command dumps only filenames of one given type per call, it suffices to append the type=… tag to each output line; this can be done with sed, e. g.:
find ./ -type f -name *.txt | sed "s/$/ type=$type/" >>files.list
I am teaching myself more (l)unix skills and wanted to see if I could begin to write a program that will eventually read all .gz files and expand them. However, I want it to be super dynamic.
#!/bin/bash
dir=~/derp/herp/path/goes/here
for file in $(find dir -name '*gz')
do
echo $file
done
So when I excute this file, I simply go
bash derp.sh.
I don't like this. I feel the script is too brittle.
How can I rework my for loop so that I can say
bash derp.sh ~/derp/herp/path/goes/here (1)
I tried re-coding it as follows:
for file in $*
However, I don't want to have to type in bash
derp.sh ~/derp/herp/path/goes/here/*.gz.
How could I rewrite this so I could simply type what is in (1)? I feel I must be missing something simple?
Note
I tried
for file in $*/*.gz and that obviously did not work. I appreciate your assistance, my sources have been a wrox unix text, carpentry v5, and man files. Unfortunately, I haven't found anything that will what I want.
Thanks,
GeekyOmega
for dir in "$#"
do
for file in "$dir"/*.gz
do
echo $file
done
done
Notes:
In the outer loop, dir is assigned successively to each argument given on the command line. The special form "$#" is used so that the directory names that contain spaces will be processed correctly.
The inner loop runs over each .gz file in the given directory. By placing $dir in double-quotes, the loop will work correctly even if the directory name contains spaces. This form will also work correctly if the gz file names have spaces.
#!/bin/bash
for file in $(find "$#" -name '*.gz')
do
echo $file
done
You'll probably prefer "$#" instead of $*; if you were to have spaces in filenames, like with a directory named My Documents and a directory named Music, $* would effectively expand into:
find My Documents Music -name '*.gz'
where "$#" would expand into:
find "My Documents" "Music" -name '*.gz'
Requisite note: Using for file in $(find ...) is generally regarded as a bad practice, because it does tend to break if you have spaces or newlines in your directory structure. Using nested for loops (as in John's answer) is often a better idea, or using find -print0 and read as in this answer.
How can I use the relative path or absolute path as a single command line argument in a shell script?
For example, suppose my shell script is on my Desktop and I want to loop through all the text files in a folder that is somewhere in the file system.
I tried sh myshscript.sh /home/user/Desktop, but this doesn't seem feasible. And how would I avoid directory names and file names with whitespace?
myshscript.sh contains:
for i in `ls`
do
cat $i
done
Superficially, you might write:
cd "${1:-.}" || exit 1
for file in *
do
cat "$file"
done
except you don't really need the for loop in this case:
cd "${1:-.}" || exit 1
cat *
would do the job. And you could avoid the cd operation with:
cat "${1:-.}"/*
which lists (cats) all the files in the given directory, even if the directory or the file names contains spaces, newlines or other difficult to manage characters. You can use any appropriate glob pattern in place of * — if you want files ending .txt, then use *.txt as the pattern, for example.
This breaks down if you might have so many files that the argument list is too long. In that case, you probably need to use find:
find "${1:-.}" -type f -maxdepth 1 -exec cat {} +
(Note that -maxdepth is a GNU find extension.)
Avoid using ls to generate lists of file names, especially if the script has to be robust in the face of spaces, newlines etc in the names.
Use a glob instead of ls, and quote the loop variable:
for i in "$1"/*.txt
do
cat "$i"
done
PS: ShellCheck automatically points this out.