I have multiple files in multiple directories and i have to rename these files from lowercase to uppercase; the file extension may vary and needs to be in lowercase (should be renamed too for files with extensions in uppercase).
NB: I have rename version from util-linux on CentOS Linux7.
i tried this :
find /mydir -depth | xargs -n 1 rename -v 's/(.*)\/([^\/]*)/$1\/\U$2/' {} \;
find /mydir -depth | xargs -n 1 rename -v 's/(.*)\/([^\/]*)/$2\/\L$2/' {} \;
but it's not working it changes nothing and i have no output.
Itried another solution :
for SRC in `find my_root_dir -depth`
do
DST=`dirname "${SRC}"`/`basename "${SRC}" | tr '[A-Z]' '[a-z]'`
if [ "${SRC}" != "${DST}" ]
then
[ ! -e "${DST}" ] && mv -T "${SRC}" "${DST}" || echo "${SRC} was not renamed"
fi
done
this one partially works but transforms the files extensions to uppercase too.
Any suggestions on how to keep/transform the extensions to lowercase ?
Thank you!
Possible solution with Perl rename:
find /mydir -depth -type f -exec rename -v 's/(.*\/)?([^.]*)/$1\U$2/' {} +
The commands in the question have several problems.
You seem to confuse the syntax of find's -exec action and xargs.
find /mydir -depth -type f -exec rename -v 'substitution_command' {} \;
find /mydir -depth -type f| xargs -n 1 rename -v 'substitution_command'
The xargs version has problems in case a file name contains a space.
If you replace \; with +, multiple file names are passed to one invocation of rename.
The substitution command is only supported by the Perl version of the rename command. You might have to install this version. See Get the Perl rename utility instead of the built-in rename
The substitution did not work in my test. I successfully used
rename -v 's/(.*\/)?([^.]*)/$1\U$2/' file ...
The first group (.*\/)? optionally matches a sequence of characters with a trailing /. This is used to copy the directory unchanged.
The second group ([^.]*) matches a sequence of characters except ..
This is the file name part before the first dot (if any) which will be converted to uppercase. In case the file name has more than one extension, all will remain unchanged, e.g.
Path/To/Foo.Bar.Baz -> Path/To/FOO.Bar.Baz
rename-independent solution (using find together with mv)
You can rename all files in a directory with a following command:
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
First part (for i in $( ls | grep [A-Z] );) looks for all uppercase characters and executes until all files are "scanned".
Second part (``) turns all uppercase characters into lowercase ones.
Perl-based rename dependent solution
rename -f 'y/A-Z/a-z/' *
This command changes uppercase characters to the lowercase ones. -f option allows overwriting of existing files, but it is not necessary.
suggesting a trick with awk that will generate all required mv commands:
awk '{f=$0;split($NF,a,".");$NF=tolower(a[1])"."toupper(a[2]);print "mv "f" "$0}' FS=/ OFS=/ <<< $(find . -type f)
Inspect the result, and run all mv commands together:
bash <<< $(awk '{f=$0;split($NF,a,".");$NF=tolower(a[1])"."toupper(a[2]);print "mv "f" "$0}' FS=/ OFS=/ <<< $(find . -type f))
awk script script.awk explanation
BEGIN { # preprocessing configuration
FS="/"; # set awk field separtor to /
OFS="/"; # set awk output field separtor to /
}
{ # for each line in input list
filePath = $0; # save the whole filePath in variable
# fileName is contained in last field $NF
# split fileName by "." to head: splitedFileNameArr[1] and tail: splitedFileNameArr[2]
split($NF,splitedFileNameArr,".");
# recreate fileName from lowercase(head) "." uppercase(tail)
$NF = tolower(splitedFileNameArr[1]) "." toupper(splitedFileNameArr[2]);
# generate a "mv" command from original filePath and regenerated fileName
print "mv "filePath" "$0;
}
Testing:
mkdir {a1,B2}/{A1,b2} -p; touch {a1,B2}/{A1,b2}/{A,b}{b,C}.{c,D}{d,C}
find . -type f
./a1/A1/Ab.cC
./a1/A1/Ab.cd
./a1/A1/Ab.DC
./a1/A1/Ab.Dd
./B2/b2/AC.DC
./B2/b2/AC.Dd
.....
./B2/b2/bC.cd
./B2/b2/bC.DC
./B2/b2/bC.Dd
awk -f script.awk <<< $(find . -type f)
.....
mv ./a1/b2/Ab.cd ./a1/b2/ab.CD
mv ./a1/b2/Ab.DC ./a1/b2/ab.DC
mv ./a1/b2/Ab.Dd ./a1/b2/ab.DD
mv ./B2/A1/bC.Dd ./B2/A1/bc.DD
.....
mv ./B2/b2/bC.DC ./B2/b2/bc.DC
mv ./B2/b2/bC.Dd ./B2/b2/bc.DD
bash <<< $(awk -f script.awk <<< $(find . -type f))
find . -type f
Related
Given a directory with files with an alphanumeric name:
file45369985.xml
file45793220.xml
file0005461x.xml
Also, given a csv table with a list of files
file45369985.xml,file,45369985,.xml,https://www.tib.eu/de/suchen/id/FILE:45369985/Understanding-terrorism-challenges-perspectives?cHash=16d713678274dd2aa205fc07b2fc5b86
file0005461X.xml,file,0005461X,.xml,https://www.tib.eu/de/suchen/id/FILE:0005461X/The-reality-of-social-construction?cHash=5d8152fbbfae77357c1ec6f443f8c8a4
I would like to match all files in the csv table with the directory's content and move them somewhere else. However, I cannot switch off the case sensitivity in this command:
while read p; do
data_set=$(echo "$p" | cut -f1 -d",")
# do something else
done
How can the "X-Files" be correctly matched as well?
Given the format of the csv file (no quotes around the first field), I show an answer for filenames without newlines.
List all files in current directory
find . -maxdepth 1 -type f -printf "%f\n"
Look for one filename in that list (ignoring case)
grep -Fix file0005461X.xml <(find . -maxdepth 1 -type f -printf "%f\n")
Show first field only from file
cut -d"," -f1 csvfile
Pretend that the output is a file
<(cut -d"," -f1 csvfile)
Tell grep to use that "file" for strings to look for with option f
grep -Fixf <(cut -d"," -f1 csvfile) <(find . -maxdepth 1 -type f -printf "%f\n")
Move to /tmp
grep -Fixf <(cut -d"," -f1 csvfile) <(find . -maxdepth 1 -type f -printf "%f\n") |
xargs -i{} mv "{}" /tmp
You can use join to perform a inner join between the CSV and the file list:
join -i -t, \
<(sort -t, -k1 list.csv) \
<(find given_dir -maxdepth 1 -mindepth 1 -type f -printf "%f\n" | sort) \
-o "2.1"
Explanation:
-i: perform a case insensitive comparison for the join
-t,: use the comma as a field separator
<(sort -t, -k1 list.csv): sort the CSV file on the first field using the comma as a field separator and use the output as a file, and perform a process substitution to "connect the output" to a file and use it as file argument (see Bash manual page)
<(find given_dir -maxdepth 1 -mindepth 1 -type f -printf "%f\n" | sort): list all the file stored in the root of the given directory given_dir (and not in the subdirectories), sort it and perform a process substitution like the above
-o "2.1": list the first column of the second input (the find output) of the join result
Note: this solution relies on GNU find due to printf command
awk -F [,\.] '{ print substr($1,1,length($1)-1)toupper(substr($1,length($1)))"."$2;print substr($1,1,length($1)-1)tolower(substr($1,length($1)))"."$2 }' csvfile | while read line
do
find /path -name "$line" -exec mv '{}' /newpath \;
done
Use awk and set the file delimiter to . and , Take each line and generate both an uppercase and lowercase X version of the file name.
Loop through this output and find the file in a given path. If the file exists, execute the move command to a given path.
You can use grep -i to make case insensitive matches:
while read p; do
data_set=$(echo "$p" | cut -f1 -d",")
match=$(ls $your_dir | grep -i "^$data_set\$")
if [ ! -z match ]; then
mv "$match" $another_dir
fi
done
I want to copy some files from directory A to directory B on basis of partial filenames which are in a text file.
I tried the command below, but it's not working
for fn in $(cat filename.txt); do find . -type -f -name '$fn*' \
-exec rsync -aR '{}' /tmp/test2 \;
File names are in format abcd-1234-opi.txt, rety-4567-yuui.txt. I have to grep numbers from file names and then copy those files to another folder, I have numbers in a text file.
Can anybody guide me?
If I understand your question correct:
for file in $(<./example); do cp "${file}" /tmp/test2/; done
for file in $(<./example); do find . -type f -name ${file} -exec rsync -aR {} /tmp/test2/ \; ; done
If understand correctly, the pattern to search for is the first 3 digits of the numbers after the first -, like this:
abd-12312-xyz
^^^
88-45644-oio
^^^
qwe-78908-678
^^^
Here's one way to write that:
while read line; do
pattern=${line#*-}
pattern=${pattern:0:3}
find . -type f -name "*$pattern*" -exec rsync -aR {} /tmp/test2 \;
done < patterns.txt
If your inputs are like "veeram_20171004-104805_APOLLO_9004060859-all.txt",
and you want to extract the "9004060859",
then you can try to find a different logic to extract that.
For example,
"cut off every after the last '-', and everything before the last '_'".
You can write like this:
pattern=${line%-*}
pattern=${pattern##*_}
Running find repeatedly should probably be avoided. Instead, factor the expressions into the find command line programmatically.
awk 'BEGIN { printf "find . -type -f \\("; sep="" }
{ printf "%s -name \047*%s*\047", sep, $0; sep=" -o" }
END { printf " \\) -exec rsync -aR '{}' /tmp/test2 \\;\n" }' filename.txt |
sh
Leave off the pipe to sh to test.
I have lots of .ini files which configures certain properties for various projects.
I want to filter for PROJECT_A in the ini file first and if that matches, then i want to filter the second pattern CONFIG_B. Since CONFIG_B is a property of PROJECT_A...X, i want to grep only the files which contains the PROJECT_A settings and CONFIG_B is also present. I know it is bit challenging, but if i can narrow down ini files with both PROJECT_A and CONFIG_A is present, i can manually inspect them to minimum list. I have 1000 files like this :-(
Typical Config is like this
[F-Project:PROJECT_A]
stream-window-start=0
stream-window-end=0
network-feed=LIVE:
test-config=pdl tf_dms_hiab
Expected out:-
file1.ini
proj:PROJECT_A
cfg1:CONFIG_A
cfg1:CONFIG_B
cfg1:CONFIG_C
proj:PROJECT_B
cfg1:CONFIG_A
cfg1:CONFIG_C
file2.ini
proj:PROJECT_X
cfg1:CONFIG_A
cfg1:CONFIG_B
cfg1:CONFIG_C
proj:PROJECT_Y
cfg1:CONFIG_B
cfg1:CONFIG_C
file3.ini
proj:PROJECT_A
cfg1:CONFIG_B
cfg1:CONFIG_C
proj:PROJECT_B
cfg1:CONFIG_A
Results : file1.ini, file3.ini
find . -name *.ini -exec grep -w PROJECT_A {} \; -print | grep ini -exec grep CONFIG_A {} \;
[proj:PROJECT_A]
./PLATFORM/build/integration/suites/System_Maintenance_Suite/ini/Test_0621_1.ini
Since i get the output like above, im filtering only the lines containing .ini
find . -name *.ini -exec grep -w PROJECT_A {} \; -print | grep ini
./PLATFORM/build/integration/suites/System_Maintenance_Suite/ini/Test_0722_1.ini
./PLATFORM/build/integration/suites/System_Maintenance_Suite/ini/Test_0579_15.ini
./PLATFORM/build/integration/suites/System_Maintenance_Suite/ini/Test_0460_1.ini
how can i grep one line at a time for pattern CONFIG_A now
I understand i can write this to a file and read a line at a time, but i want a efficient way to do this.
Please help with your suggestions.
Saying:
find . -name *.ini -exec sh -c "grep -q PROJECT_A {} && grep -q CONFIG_A {} && echo {}" \;
would list files that contain both PROJECT_A and CONFIG_A.
Using the -q option for grep would evaluate to true only if the specified pattern existed in the file.
If you are looking for files where CONFIG_B occurs only in the stanza for proj:PROJECT_A, something like this?
find . -type f -name '*.ini' -exec awk '
/^proj:/ { prj=$1; next }
/CONFIG_B/ && prj="proj:PROJECT_A" {
print FILENAME; exit 0 }' {} \;
... or with the "real" values from the comment below,
find . -type f -name '*.ini' -exec awk '
/^F-Project:/ { prj=$1; next }
/LIVE:/ && prj="F-Project:PROJECT_A" {
print FILENAME; exit 0 }' {} \;
find . -name *.ini -exec sh -c "grep -q PROJECT_A {} && grep -q CONFIG_A {} && echo {}" \;
How it works ?
find . -name *.ini <= filters the .ini files
-exec <= executes the following command using the output from find one at a time
sh -c <= accepts string input for the shell, we need this for executing multiple commands which follws that
grep -q PROJECT_A {} <= quietly grep for PROJECT_A
grep -q PROJECT_A {} && grep -q CONFIG_A {} && echo {} <= prints the filename if both the strings are matches, simple logical and on three commands.
Hope it helps !!!
What about this neat awk solution:
awk -vPR="PROJECT_A" -vCF="CONFIG_A" 'BEGIN{R="(" CF "|" PR ")"}
{if($0 ~ R)d[FILENAME]+=1}
END{for(i in d)if(d[i]>=2)print i}' file*
file3.ini
file1.ini
I have some files:
/var/www/media/0001/0001_123456_12.jpg
/var/www/media/0002/0002_123456_12.jpg
/var/www/media/0003/0003_123456_12.jpg
and I want to rename them to:
/var/www/media/0001/0001_test.jpg
/var/www/media/0002/0002_test.jpg
/var/www/media/0003/0003_test.jpg
My idea was to find the first _, remove the rest of the file until the . then add test.
Any ideas?
find /var/www/media/ -name \*.jpg -exec sh -c '
a=$(echo {} | sed s/_123456_/_/);
[ "$a" != "{}" ] && mv "{}" "$a" '
You find all jpg files in the /var/www/media and run for each file the command:
a=$(echo {} | sed s/_123456_/_/)
[ "$a" != "{}" ] && mv "{}" "$a"
After this command, the a variable has rewritten name of the file inside:
a=$(echo {} | sed s/_123456_/_/)
The we compare the a variable and the realname ({}), and they are not equal
the file must be renamed.
Here's a solution in perl that allows you to use regular expressions.
If you can install the mmv package, then these operations become easy. With mmv, you can do what you want with:
cd /var/www/media
mmv '*/*_123456_*.jpg' '#1/#2_test.jpg'
Here is the mmv manpage: http://manpages.ubuntu.com/manpages/lucid/man1/mln.1.html
I want to iterate over a list of files. This list is the result of a find command, so I came up with:
getlist() {
for f in $(find . -iname "foo*")
do
echo "File found: $f"
# do something useful
done
}
It's fine except if a file has spaces in its name:
$ ls
foo_bar_baz.txt
foo bar baz.txt
$ getlist
File found: foo_bar_baz.txt
File found: foo
File found: bar
File found: baz.txt
What can I do to avoid the split on spaces?
You could replace the word-based iteration with a line-based one:
find . -iname "foo*" | while read f
do
# ... loop body
done
There are several workable ways to accomplish this.
If you wanted to stick closely to your original version it could be done this way:
getlist() {
IFS=$'\n'
for file in $(find . -iname 'foo*') ; do
printf 'File found: %s\n' "$file"
done
}
This will still fail if file names have literal newlines in them, but spaces will not break it.
However, messing with IFS isn't necessary. Here's my preferred way to do this:
getlist() {
while IFS= read -d $'\0' -r file ; do
printf 'File found: %s\n' "$file"
done < <(find . -iname 'foo*' -print0)
}
If you find the < <(command) syntax unfamiliar you should read about process substitution. The advantage of this over for file in $(find ...) is that files with spaces, newlines and other characters are correctly handled. This works because find with -print0 will use a null (aka \0) as the terminator for each file name and, unlike newline, null is not a legal character in a file name.
The advantage to this over the nearly-equivalent version
getlist() {
find . -iname 'foo*' -print0 | while read -d $'\0' -r file ; do
printf 'File found: %s\n' "$file"
done
}
Is that any variable assignment in the body of the while loop is preserved. That is, if you pipe to while as above then the body of the while is in a subshell which may not be what you want.
The advantage of the process substitution version over find ... -print0 | xargs -0 is minimal: The xargs version is fine if all you need is to print a line or perform a single operation on the file, but if you need to perform multiple steps the loop version is easier.
EDIT: Here's a nice test script so you can get an idea of the difference between different attempts at solving this problem
#!/usr/bin/env bash
dir=/tmp/getlist.test/
mkdir -p "$dir"
cd "$dir"
touch 'file not starting foo' foo foobar barfoo 'foo with spaces'\
'foo with'$'\n'newline 'foo with trailing whitespace '
# while with process substitution, null terminated, empty IFS
getlist0() {
while IFS= read -d $'\0' -r file ; do
printf 'File found: '"'%s'"'\n' "$file"
done < <(find . -iname 'foo*' -print0)
}
# while with process substitution, null terminated, default IFS
getlist1() {
while read -d $'\0' -r file ; do
printf 'File found: '"'%s'"'\n' "$file"
done < <(find . -iname 'foo*' -print0)
}
# pipe to while, newline terminated
getlist2() {
find . -iname 'foo*' | while read -r file ; do
printf 'File found: '"'%s'"'\n' "$file"
done
}
# pipe to while, null terminated
getlist3() {
find . -iname 'foo*' -print0 | while read -d $'\0' -r file ; do
printf 'File found: '"'%s'"'\n' "$file"
done
}
# for loop over subshell results, newline terminated, default IFS
getlist4() {
for file in "$(find . -iname 'foo*')" ; do
printf 'File found: '"'%s'"'\n' "$file"
done
}
# for loop over subshell results, newline terminated, newline IFS
getlist5() {
IFS=$'\n'
for file in $(find . -iname 'foo*') ; do
printf 'File found: '"'%s'"'\n' "$file"
done
}
# see how they run
for n in {0..5} ; do
printf '\n\ngetlist%d:\n' $n
eval getlist$n
done
rm -rf "$dir"
There is also a very simple solution: rely on bash globbing
$ mkdir test
$ cd test
$ touch "stupid file1"
$ touch "stupid file2"
$ touch "stupid file 3"
$ ls
stupid file 3 stupid file1 stupid file2
$ for file in *; do echo "file: '${file}'"; done
file: 'stupid file 3'
file: 'stupid file1'
file: 'stupid file2'
Note that I am not sure this behavior is the default one but I don't see any special setting in my shopt so I would go and say that it should be "safe" (tested on osx and ubuntu).
find . -iname "foo*" -print0 | xargs -L1 -0 echo "File found:"
find . -name "fo*" -print0 | xargs -0 ls -l
See man xargs.
Since you aren't doing any other type of filtering with find, you can use the following as of bash 4.0:
shopt -s globstar
getlist() {
for f in **/foo*
do
echo "File found: $f"
# do something useful
done
}
The **/ will match zero or more directories, so the full pattern will match foo* in the current directory or any subdirectories.
I really like for loops and array iteration, so I figure I will add this answer to the mix...
I also liked marchelbling's stupid file example. :)
$ mkdir test
$ cd test
$ touch "stupid file1"
$ touch "stupid file2"
$ touch "stupid file 3"
Inside the test directory:
readarray -t arr <<< "`ls -A1`"
This adds each file listing line into a bash array named arr with any trailing newline removed.
Let's say we want to give these files better names...
for i in ${!arr[#]}
do
newname=`echo "${arr[$i]}" | sed 's/stupid/smarter/; s/ */_/g'`;
mv "${arr[$i]}" "$newname"
done
${!arr[#]} expands to 0 1 2 so "${arr[$i]}" is the ith element of the array. The quotes around the variables are important to preserve the spaces.
The result is three renamed files:
$ ls -1
smarter_file1
smarter_file2
smarter_file_3
find has an -exec argument that loops over the find results and executes an arbitrary command. For example:
find . -iname "foo*" -exec echo "File found: {}" \;
Here {} represents the found files, and wrapping it in "" allows for the resultant shell command to deal with spaces in the file name.
In many cases you can replace that last \; (which starts a new command) with a \+, which will put multiple files in the one command (not necessarily all of them at once though, see man find for more details).
I recently had to deal with a similar case, and I built a FILES array to iterate over the filenames:
eval FILES=($(find . -iname "foo*" -printf '"%p" '))
The idea here is to surround each filename with double quotes, separate them with spaces and use the result to initialize the FILES array.
The use of eval is necessary to evaluate the double quotes in the find output correctly for the array initialization.
To iterate over the files, just do:
for f in "${FILES[#]}"; do
# Do something with $f
done
In some cases, here if you just need to copy or move a list of files, you could pipe that list to awk as well.
Important the \"" "\" around the field $0 (in short your files, one line-list = one file).
find . -iname "foo*" | awk '{print "mv \""$0"\" ./MyDir2" | "sh" }'
Ok - my first post on Stack Overflow!
Though my problems with this have always been in csh not bash the solution I present will, I'm sure, work in both. The issue is with the shell's interpretation of the "ls" returns. We can remove "ls" from the problem by simply using the shell expansion of the * wildcard - but this gives a "no match" error if there are no files in the current (or specified folder) - to get around this we simply extend the expansion to include dot-files thus: * .* - this will always yield results since the files . and .. will always be present. So in csh we can use this construct ...
foreach file (* .*)
echo $file
end
if you want to filter out the standard dot-files then that is easy enough ...
foreach file (* .*)
if ("$file" == .) continue
if ("file" == ..) continue
echo $file
end
The code in the first post on this thread would be written thus:-
getlist() {
for f in $(* .*)
do
echo "File found: $f"
# do something useful
done
}
Hope this helps!
Another solution for job...
Goal was :
select/filter filenames recursively in directories
handle each names (whatever space in path...)
#!/bin/bash -e
## #Trick in order handle File with space in their path...
OLD_IFS=${IFS}
IFS=$'\n'
files=($(find ${INPUT_DIR} -type f -name "*.md"))
for filename in ${files[*]}
do
# do your stuff
# ....
done
IFS=${OLD_IFS}