My directory contains mix of hundreds of files and directories similar to this:
508471/
ae_lstm__ts_ 508471_detected_anomalies.pdf
ae_lstm__508471_prediction_result.pdf
mlp_508471_prediction_result.pdf
mlp__ts_508471_detected_anomalies.pdf
vanilla_lstm_508471_prediction_result.pdf
vanilla_lstm_ts_508471_detected_anomalies.pdf
598690/
ae_lstm__ts_598690_detected_anomalies.pdf
ae_lstm__598690_prediction_result.pdf
mlp_598690_prediction_result.pdf
mlp__ts_598690_detected_anomalies.pdf
vanilla_lstm_598690_prediction_result.pdf
vanilla_lstm_ts_598690_detected_anomalies.pdf
There are folders with an ID number as their names, like 508471 and 598690.
In the same path as these folders, there are pdf files that have this ID number as part of their name. I need to move all the pdf files with the same ID in their name, to their related directories.
I tried the following shell script but it doesn't do anything. What am I doing wrong?
I'm trying to loop over all the directories, find the files that have id in their name, and move them to the same dir:
for f in ls -d */; do
id=${f%?} # f value is '598690/', I'm removing the last character, `\`, to get only the id part
find . -maxdepth 1 -type f -iname *.pdf -exec grep $id {} \; -exec mv -i {} $f \;
done
#!/bin/sh
find . -mindepth 1 -maxdepth 1 -type d -exec sh -c '
for d in "$#"; do
id=${d#./}
for file in *"$id"*.pdf; do
[ -f "$file" ] && mv -- "$file" "$d"
done
done
' findshell {} +
This finds every directory inside the current one (finding, for example, ./598690). Then, it removes ./ from the relative path and selects each file that contains the resulting id (598690), moving it to the corresponding directory.
If you are unsure of what this will do, put an echo between && and mv, it will list the mv actions the script would make.
And remember, do not parse ls.
The below code should do the required job.
for dir in */; do find . -mindepth 1 -maxdepth 1 -type f -name "*${dir%*/}*.pdf" -exec mv {} ${dir}/ \;; done
where */ will consider only the directories present in the given directory, find will search only files in the given directory which matches *${dir%*/}*.pdf i.e file name containing the directory name as its sub-string and finally mv will copy the matching files to the directory.
in Unix please use below command
find . -name '*508471*' -exec bash -c 'echo mv $0 ${0/508471/598690}' {} \;
You may use this for loop from the parent directory of these pdf files and directories:
for d in */; do
compgen -G "*${d%/}*.pdf" >/dev/null && mv *"${d%/}"*.pdf "$d"
done
compgen -G is used to check if there is a match for given glob or not.
Related
I am attempting to move some video files of mine into new subdirectories while also renaming them on my Unraid system. The files all follow a similar naming convention:
featurette name-featurette.mkv
I would like to move these files from their current directory to a subdirectory and rename them like this:
featurettes/featurette name.mkv
I am able to create the directories and relocate the files using find and execdir:
find . -type f -name *-featurette.mkv -maxdepth 2 -execdir mkdir ./featurettes/ \;
find . -type f -name *-featurette.mkv -maxdepth 2 -execdir mv {} ./featurettes/ \;
I am struggling with the renaming piece. I've tried the rename command but am unable to get it to work within the featurettes directory, let alone from two directories above, which is where I'd like to execute the command. I've tried the following command within the featurettes directory:
rename \-featurette.mkv .mkv *
However I get the error:
invalid option -- 'f'
I thought by escaping the dash I could avoid that issue, but it doesn't appear to work. Any advice on how to remove this pattern from all files within subdirectories matching it would be very much appreciated.
From man rename you see this command gets options and 3 positional parameters:
SYNOPSIS
rename [options] expression replacement file...
So in your case the first parameter is being interpreted as an option. You may use this syntax:
rename -- '-featurette' '' *-featurette.mkv
to rename the files. -- indicates that any options are over and what follows are only positional parameters.
Totally, to copy the files with one mv process and rename them:
mkdir -p target/dir
find . -maxdepth 2 -type f -name "*-featurette.mkv" -exec mv -t target/dir {} +
cd target/dir && rename -- '-featurette' '' *-featurette.mkv
If you want to rename many files located into different subdirectories, you can use this syntax:
find . -name "*-featurette.mkv" -print0 | xargs -r0 rename -- '-featurette' ''
find . \
-maxdepth 2 \
-type f \
-name '*-featurette.mkv' \
-execdir sh -c '
echo mkdir -p ./featurettes/
echo mv -- "$#" ./featurettes/
' _ {} \+
Issue with your implementations I fixed or improved:
-maxdepth 2 must precede -type f
-name '*-featurette.mkv' must have the pattern quoted to prevent the shell to expand globb it.
-execdir is best used with an inline shell, so it can also process multiple arguments from the same directory
Also keep in mind that while running a command with -execdir, find will cd to that directory. It means that mv -- "$#" ./featurettes/' will move files into the ./featurettes/' directory relative to were -execdir has just cd.
Version which also rename files while moving:
( has no echo dry-run protection, so use only if you are sure it does what you want )
#!/usr/bin/env sh
find . \
-maxdepth 2 \
-depth \
-name '*-featurette.mkv' \
-type f \
-execdir sh -c '
mkdir -p featurettes
for arg
do
basename=${arg##*/}
mv -- "$basename" "./featurettes/${basename%-featurette.mkv}.mkv"
done
' _ {} +
You can use Bash's shell parameter expansion feature to get the part of the file name, for example:
$> filename=name-featurette.mkv
$> echo ${filename%-*} #To print first part before '-'
name
$> echo ${filename##*.} #To get the extension
mkv
$> echo ${filename#*-} #To print the part after '-' with extension
featurette.mkv
With this and slightly modifying your find command, you should be able to move+rename your files:
find . -type f -name '*-featurette.mkv' -maxdepth 2 -execdir sh -c 'f="{}"; mv -- "$f" ./featurettes/"${f%-*}.mkv"' \;
In fact you should be able to combine both the find command into one to create_dir, move and rename file.
find . -type f -name '*-featurette.mkv' -maxdepth 2 -execdir sh -c 'f="{}"; mkdir ./featurettes; mv -- "$f" ./featurettes/"${f%-*}.mkv"' \;
I have a directory containing a large number of directories. Each directory contains some files and in some cases another directory.
parent_directory
sub_dir_1
sub_dir_1.txt
sub_dir_1_1.txt
sub_dir_2
sub_dir_2.txt
sub_dir_2_1.txt
sub_dir_3
sub_dir_3.txt
sub_dir_3_1.txt
sub_dir_4
sub_dir_4.txt
sub_dir_4_1.txt
sub_dir_5
sub_dir_5.txt
sub_dir_5_1.txt
I need to check that each sub_dir contains at least one file with the exact same name. I don' need to check any further down if there are sub directories within the sub_dirs.
I was thinking of using for d in ./*/ ; do (command here); done but I dont know how to get access to the sub_dir name inside the for loop
for d in ./*/ ;
do
(if directory does not contain 1 file that is the same name as the directory then echo directory name );
done
What is the best way to do this or is there a simpler way?
from the parent directory
find -maxdepth 1 -type d -printf "%f\n" |
xargs -I {} find {} -maxdepth 1 -type f -name {}.txt
will give you the name/name.txt pair. Compare with the all dir names to find the missing ones.
UPDATE
this might be simpler, instead of scanning you can check whether file exists or not
for f in $(find -maxdepth 1 -type d -printf "%f\n");
do if [ ! -e "$f/$f.txt" ];
then echo "$f not found";
fi; done
Maybe not understand fully, but
find . -print | grep -P '/(.*?)/\1\.txt'
this will print any file which is inside of the same-named directory, e.g:
./a/b/b.txt
./a/c/d/d.txt
etc...
Similarly
find . -print | sed -n '/\(.*\)\/\1\.txt/p'
this
find . -print | grep -P '/(.*?)/\1\.'
will list all files regardless of the extension in same-named dirs.
You can craft other regexes following the backreference logic.
I have a task, to find the pattern of the file, create a folder with the pattern name and copy the file to that folder. I am able to create the folders.
folders=`find /Location -type f -name "*.pdf" -printf "%f\n" | cut -f 1 -d '_' | sort -u`
for i in $folders
do
mkdir -p /LocationToCreateTheFolder/$i
done
Not able to go further on how to copy the files.
maybe try?
for i in $folders do mkdir -p /LocationToCreateTheFolder/$i && cp ./$i.pdf ./$i/
This will do the finding and the copying:
find Location -type f -name '*.pdf' -exec bash -c 'f=${1##*/}; d="LocationToCreateTheFolder/${f%%_*}"; mkdir -p "$d" && cp "$1" "$d"' None {} \;
This is safe for difficult file names even ones that contain spaces, tabs, or newlines in their names.
How it works
find Location -type f -name '*.pdf' -exec bash -c '...' None {} \;
This will find the pdf files under directory Location and, for each one found, the bash commands inside '...' will be executed with $1 set to the name of the file found. ($0 is set to None. We don't use $0.)
f=${1##*/}
This removes the directory names from the name of the file. This is an example of prefix removal: everything in $1 up to and including the last / is removed.
d="LocationToCreateTheFolder/${f%%_*}"
This creates the name of the directory to which we want to send the file.
${f%%_*}" is an example of suffix removal. Everything in $f from the first _ and after is removed.
mkdir -p "$d" && cp "$1" "$d"
This makes sure that the directory exists and then copies the file to it.
Can you please suggest efficient way to move files from one location to their sub directory in a loop.
Ex:
/MY_PATH/User1/1234/Daily/abc.txt to /MY_PATH/User1/1234/Daily/Archive/abc.txt
/MY_PATH/User2/3456/Daily/def.txt to /MY_PATH/User2/3456/Daily/Archive/def.txt
/MY_PATH/User1/1111/Daily/hij.txt to /MY_PATH/User1/1111/Daily/Archive/hij.txt
/MY_PATH/User2/2222/Daily/def.txt to /MY_PATH/User2/2222/Daily/Archive/def.txt
I started in this way, but need your suggestions and best way to write it:
#!/bin/bash
dir1="/MyPath/"
subs= `ls $dir1`
for i in $subs; do
mv $dir1/$i/*/Daily $dir1/$i/*/Daily/Archive
done
My one line bash
for dir in $(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
);do
mkdir -p $dir/Archives
find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} +
done
To quickly test:
mkdir -p MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily
touch MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily/{abc,bcd,def,feg,fds}.txt
for dir in $( find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily );do
mkdir -p $dir/Archives; find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t $dir/Archives {} + ; done
ls -lR MY_PATH
This seem match OP's request
For more robust solution
There is a solution wich work with spaces somewhere in path...
Edited to include #mklement0's well pointed suggestion.
while IFS= read dir;do
mkdir -p "$dir"/Archives
find "$dir" -maxdepth 1 -mindepth 1 ! -name Archives \
-exec mv -t "$dir/Archives" {} +
done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
)
Same demo;
mkdir -p MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily
touch MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily/{abc,"b c",def,hgz0}.txt
while read dir;do mkdir -p "$dir"/Archives;find "$dir" -maxdepth 1 -mindepth 1 \
! -name Archives -exec mv -t "$dir/Archives" {} +; done < <(
find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily )
ls -lR MY_PATH
Assuming the directory structure is as you have shown in your examples, i.e.
MY_PATH/
subdir-level-1/
subdir-level-2/
Daily/
files
Archive/
Here's what you can do:
shopt -s nullglob # defend against globbing failure -- inspired by mklement0's answer
root="/MyPath"
for dir in "${root}"/*/*/Daily/; do
mkdir -p "${dir}/Archive" # if Archive might not exist; to be pedantic you should look at David C. Rankin's answer for error handling, but usually we know what we're doing so that's not necessary
find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive"
done
The reason I use find and xargs is to save a few processes; you can as well move files in each ${dir} one by one.
Update: #mklement0 suggested that find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive" can be further improved to
find "${dir}" -maxdepth 1 -type f -exec mv -t "${dir}/Archive" +
which is a very good point.
Try the following:
dir1="/MyPath"
for d in "$dir1"/*/*/Daily/; do
[[ -d $d ]] || break # break, if no subdirectories match
for f in "$d"/*; do # loop over files in */*/Daily/
[[ -f "$f" ]] || continue # skip non-files or if nothing matches
mv "$f" "$d"/Archive/
done
done
"$dir1"*/*/Daily/ matches all grandchild subdirectories of $dir1; thanks to the terminating /, only directories match; note that, as a result, $d ends in /.
Note that $d therefore ends in /, and, strictly speaking, needs no / later on when synthesizing paths with it (e.g., "$d"/*), but doing so does no harm and helps readability, as #4ae1e1 points out in a comment.
[[ -d $d ]] || break ensures that the loop is exited if no grandchild directories match (by default, a glob (pattern) that has no matches is passed as is to the loop).
for f in "$d"* loops over all entries (files and/or subdirs.) in $d:
[[ -f "$f" ]] || continue ensures that only files are processed or, in the event that nothing matches, the loop is exited.
mv "$f" "$d"/Archive/ then moves each file to subdir. Archive.
You need to check for, and if not present, create the destination directory before moving the file to Archive. If you cannot create the directory (due to permissions or otherwise), you skip the move. The following does not assume any limitation on depth, but will omit any directory containing Archive as an intermediate subdirectory:
oldifs="$IFS"
IFS=$'\n'
for i in $(find /MY_PATH -type f); do
[[ "$i" =~ Archive ]] && continue
[ -d "${i%/*}/Archive" ] || mkdir -p "${i%/*}/Archive"
[ -d "${i%/*}/Archive" ] || {
printf "error: unable to create '%s'\n" "${i%/*}/Archive"
continue
}
mv -fv "$i" "${i/Daily/Daily\/Archive}"
done
IFS="$oldifs"
Output when run
$ bash archive_daily.sh
mv -fv /MY_PATH/User1/1111/Daily/hij.txt /MY_PATH/User1/1111/Daily/Archive/hij.txt
mv -fv /MY_PATH/User1/1234/Daily/abc.txt /MY_PATH/User1/1234/Daily/Archive/abc.txt
mv -fv /MY_PATH/User2/3456/Daily/def.txt /MY_PATH/User2/3456/Daily/Archive/def.txt
mv -fv /MY_PATH/User2/2222/Daily/def.txt /MY_PATH/User2/2222/Daily/Archive/def.txt
Note: you can limit/tighten the file selection by adjusting the call to find populating the for loop (e.g. -name or -iname). This simply checks/moves every file to its Archive folder. To limit to only files with the .txt extension, you can specify find /MY_PATH -type f -name "*.txt". To limit to only files in the /MY_PATH/User1 and /MY_PATH/User2directories with a .txt extension, use find /MY_PATH/User[12] -type f -name "*.txt".
Note2: when looping on filenames, the paths & filenames should not contain non-standard characters for the current locale. Certainly you should not have the '\n' as a character in your filename. Setting IFS is required to protect against word splitting on spaces in either the path or filename.
Since you said efficient, anything with a subshell will fail in funny ways with lots of entries. You're better off using xargs:
#!/bin/bash
dir1="/MyPath/"
find $dir1 -name Daily -type d -depth 3 | while read i
do
pushd .
cd $i
mkdir Archive
find . -type f -depth 1 | xargs -J {} mv {} Archive
popd
done
The outer find will look for you Daily directories. It's very specific in that they have to be at a certain depth and directories, not regular files. The results gets piped into read, where each directory is entered, Archive is created, and files batch-copied with xargs ... mv. Complete file lists and directory lists are never stored in memory, so it scales very well.
I've already used that command to rename the files in multiple directories and change JPG to jpg, so I have consistency.
find . -name '*.jpg' -exec sh -c 'mv "$0" "${0%.JPG}$.jpg"' {} \;
Do you have any idea how to change that to include the folder name in the name of the file
I am executing that in a folder that contains about 2000 folders (SKU's) or products ... and inside every SKU folder, there are 9 images. 1.jpg 2.jpg .... 9.jpg.
So the bottom-line is I have 2000 images with name 1.jpg, 2.jpg ... 9.jpg. I need those files to be unique, for example:
folder-name-1.jpg ... folder-name.2.jpg ... so on, in every folder.
Any help will be appreciated.
For example I can do as follows:
$ find . -iname '*.jpg' | while read fn; do name=$(basename "$fn") ; dir=$(dirname "$fn") ; mv "$fn" "$dir/$(basename "$dir")-$name" ;done
./lib/bukovina/version.jpg ./lib/bukovina/bukovina-version.jpg
./lib/bukovina.jpg ./lib/lib-bukovina.jpg
You can use fine one-liner:
find . -name '*.jpg' -execdir \
bash -c 'd="${PWD##*/}"; [[ "$1" != "$d-"* ]] && mv "$1" "./$d-$1"' - '{}' \;
This command uses safe approach to check whether image name is already not prefixed by the current directory name. You can run it multiple times also and image name won't be renamed after first run.
To get the folder name of a file you can do $(basename $(dirname ${FILE})), where ${FILE} is a path that may be relative but must contain at least one folder before the file name in it. This should not be a problem with find. If it is, just run it from one directory up.
find . -name '*.jpg' -exec sh -c 'mv "$0" "$(basename $(dirname $0))-${0%.JPG}$.jpg"' {} \;
Or, if you have JPEGs in your current directory:
find ../<dirname> -name '*.jpg' -exec sh -c 'mv "$0" "$(basename $(dirname $0))-${0%.JPG}$.jpg"' {} \;