I have a directory structure like this
a/1/01.jpg
b/2/01.jpg
c/3/01.jpg
I want to make it into a structure like this
a1/01.jpg
b2/01.jpg
c3/01.jpg
So far I have a bash command that looks like this
find . -mindepth 2 -type d -execdir bash -c 'mv -i \"$1\" ./\"${1//\/\[/_[}\"' bash {} \;
However the command failed with these statements
mv: cannot stat '"./3"': No such file or directory
mv: cannot stat '"./2"': No such file or directory
mv: cannot stat '"./1"': No such file or directory
What am I doing wrong here? is there a better way to do this?
Doing that by using plain bash would be easier:
#!/bin/bash
for src in */*/; do
dst=${src/\/}
echo mkdir "$dst"
echo mv "$src"* "$dst"
done
Remove both echos if the output looks fine.
Or, a more efficient version:
for src in */*/; do
mv "$src" "${src/\/}"
done
but this version won't work properly when the destination directory (a1, b2, c3, etc.) already exists.
All operations need to be performed on the leaf directories. These are identified by having two links which you can find with:
$ find -type d -links 2
Once you have these directories, you only have to rename the directory
$ find -type d -links 2 -exec sh -c 'd1={};d2="${d1#./}"; [ "$d2" = "${d2//\//}" ] || mv -v "$d1" "${d2//\//}"' \;
Find will complain a bit as you moved a directory away that it was traversing, but this should do it.
If you have a pure structure, i.e. files only appear in leaf-directories, this should do it. All you need to do now is delete the empty directories:
$ find -type d -empty -delete
Be aware, however, that if a leaf directory was already empty, the latter will remove it.
Related
My directory contains mix of hundreds of files and directories similar to this:
508471/
ae_lstm__ts_ 508471_detected_anomalies.pdf
ae_lstm__508471_prediction_result.pdf
mlp_508471_prediction_result.pdf
mlp__ts_508471_detected_anomalies.pdf
vanilla_lstm_508471_prediction_result.pdf
vanilla_lstm_ts_508471_detected_anomalies.pdf
598690/
ae_lstm__ts_598690_detected_anomalies.pdf
ae_lstm__598690_prediction_result.pdf
mlp_598690_prediction_result.pdf
mlp__ts_598690_detected_anomalies.pdf
vanilla_lstm_598690_prediction_result.pdf
vanilla_lstm_ts_598690_detected_anomalies.pdf
There are folders with an ID number as their names, like 508471 and 598690.
In the same path as these folders, there are pdf files that have this ID number as part of their name. I need to move all the pdf files with the same ID in their name, to their related directories.
I tried the following shell script but it doesn't do anything. What am I doing wrong?
I'm trying to loop over all the directories, find the files that have id in their name, and move them to the same dir:
for f in ls -d */; do
id=${f%?} # f value is '598690/', I'm removing the last character, `\`, to get only the id part
find . -maxdepth 1 -type f -iname *.pdf -exec grep $id {} \; -exec mv -i {} $f \;
done
#!/bin/sh
find . -mindepth 1 -maxdepth 1 -type d -exec sh -c '
for d in "$#"; do
id=${d#./}
for file in *"$id"*.pdf; do
[ -f "$file" ] && mv -- "$file" "$d"
done
done
' findshell {} +
This finds every directory inside the current one (finding, for example, ./598690). Then, it removes ./ from the relative path and selects each file that contains the resulting id (598690), moving it to the corresponding directory.
If you are unsure of what this will do, put an echo between && and mv, it will list the mv actions the script would make.
And remember, do not parse ls.
The below code should do the required job.
for dir in */; do find . -mindepth 1 -maxdepth 1 -type f -name "*${dir%*/}*.pdf" -exec mv {} ${dir}/ \;; done
where */ will consider only the directories present in the given directory, find will search only files in the given directory which matches *${dir%*/}*.pdf i.e file name containing the directory name as its sub-string and finally mv will copy the matching files to the directory.
in Unix please use below command
find . -name '*508471*' -exec bash -c 'echo mv $0 ${0/508471/598690}' {} \;
You may use this for loop from the parent directory of these pdf files and directories:
for d in */; do
compgen -G "*${d%/}*.pdf" >/dev/null && mv *"${d%/}"*.pdf "$d"
done
compgen -G is used to check if there is a match for given glob or not.
I want to copy recursively files to its parent folder for a specific file extension. For example:
./folderA/folder1/*.txt to ./folderA/*.txt
./folderB/folder2/*.txt to ./folderB/*.txt
etc.
I checked cp and find commands but couldn't get it working.
I suspect that while you say copy, you actually mean to move the files up to their respective parent directories. It can be done easily using find:
$ find . -name '*.txt' -type f -execdir mv -n '{}' ../ \;
The above command recurses into the current directory . and then applies the following cascade of conditionals to each item found:
-name '*.txt' will filter out only files that have the .txt extension
-type f will filter out only regular files (eg, not directories that – for whatever reason – happen to have a name ending in .txt)
-execdir mv -n '{}' ../ \; executes the command mv -n '{}' ../ in the containing directory where the {} is a placeholder for the matched file's name and the single quotes are needed to stop the shell from interpreting the curly braces. The ; terminates the command and again has to be escaped from the shell interpreting it.
I have passed the -n flag to the mv program to avoid accidentally overwriting an existing file.
The above command will transform the following file system tree
dir1/
dir11/
file3.txt
file4.txt
dir12/
file2.txt
dir2/
dir21/
file6.dat
dir22/
dir221/
dir221/file8.txt
file7.txt
file5.txt
dir3/
file9.dat
file1.txt
into this one:
dir1/
dir11/
dir12/
file3.txt
file4.txt
dir2/
dir21/
file6.dat
dir22/
dir221/
file8.txt
file7.txt
dir3/
file9.dat
file2.txt
file5.txt
To get rid of the empty directories, run
$ find . -type d -empty -delete
Again, this command will traverse the current directory . and then apply the following:
-type d this time filters out only directories
-empty filters out only those that are empty
-delete deletes them.
Fine print: -execdir is not specified by POSIX, though major implementations (at least the GNU and BSD one) support it. If you need strict POSIX compliance, you'll have to make do with the less safe -exec which would need additional thought to be applied correctly in this case.
Finally, please try your commands in a test directory with dummy files, not your actual data. Especially with the -delete option of find, you can loose all your data quicker than you might imaging. Read the man page and, if that is not enough, the reference manual of find. Never blindly copy shell commands from random strangers posted on the internet if you don't understand them.
$cp ./folderA/folder1/*.txt ./folderA
Try this commnad
Run something like this from the root(ish) directory:
#! /bin/bash
BASE_DIR=./
new_dir() {
LOC_DIR=`pwd`
for i in "${LOC_DIR}"/*; do
[[ -f "${i}" ]] && cp "${i}" ../
[[ -d "${i}" ]] && cd "${i}" && new_dir
cd ..
done
return 0
}
new_dir
This will search each directory. When a file is encountered, it copies the file up a directory. When a directory is found, it will move down into the directory and start the process over again. I think it'll work for you.
Good luck.
I'm making a bash script that should backup all files and dir structure to another dir.
I made the following code to do that:
find . -type f -exec cp {} $HOME/$bdir \; -o -type d -exec mkdir -p {} $HOME/$bdir \; ;
The problem is, is that this only copies the files and not the dir structure.
NOTE: I may not use cp -r, cp -R or something like it because this code is part of an assignment.
I hope somebody can put me in the right direction. ;)
Joeri
EDIT:
I changed it to:
find . -type d -exec mkdir -p $HOME/$bdir/{} \; ;
find . -type f -exec cp {} $HOME/$bdir/{} \; ;
And it works! Ty guys ;)
This sounds like a job for rsync.
You mention that this is an assignment. What are your restrictions? Are you limited to only using find? Does it have to be a single command?
One way to do this is to do it in two find calls. The first call only looks for directories. When a directory is found, mkdir the corresponding directory in the destination hierarchy. The second find call would look for files, and would use a cp command like you currently have.
You can also take each filename, transform the path manually, and use that with the cp command. Here's an example of how to generate the destination filename:
> find . -type f | sed -e "s|^\./|/new/dir/|"
/new/dir/file1.txt
/new/dir/file2.txt
/new/dir/dir1/file1_1.txt
/new/dir/dir1/file1_2.txt
For your purposes, you could write a short bash script that take the source file as input, uses sed to generate the destination filename, and then passes those two paths to cp. The dirname command will return the directory portion of a filename, so mkdir -p $(dirname $destination_path) will ensure that the destination directory exists before you call cp. Armed with a script like that, you can simply have find execute the script for every file it finds.
cd olddir; tar c . | (cd newdir; tar xp)
Can you do your find with "-type d" and exec a "mkdir -p" first, followed by your find that copies the files rather than having it all in one command? It should probably also be mkdir -p $HOME/$bdir/{}.
I want to create a clone of the structure of our multi-terabyte file server. I know that cp --parents can move a file and it's parent structure, but is there any way to copy the directory structure intact?
I want to copy to a linux system and our file server is CIFS mounted there.
You could do something like:
find . -type d > dirs.txt
to create the list of directories, then
xargs mkdir -p < dirs.txt
to create the directories on the destination.
cd /path/to/directories &&
find . -type d -exec mkdir -p -- /path/to/backup/{} \;
Here is a simple solution using rsync:
rsync -av -f"+ */" -f"- *" "$source" "$target"
one line
no problems with spaces
preserve permissions
I found this solution there
1 line solution:
find . -type d -exec mkdir -p /path/to/copy/directory/tree/{} \;
I dunno if you are looking for a solution on Linux. If so, you can try this:
$ mkdir destdir
$ cd sourcedir
$ find . -type d | cpio -pdvm destdir
This copy the directories and files attributes, but not the files data:
cp -R --attributes-only SOURCE DEST
Then you can delete the files attributes if you are not interested in them:
find DEST -type f -exec rm {} \;
This works:
find ./<SOURCE_DIR>/ -type d | sed 's/\.\/<SOURCE_DIR>//g' | xargs -I {} mkdir -p <DEST_DIR>"/{}"
Just replace SOURCE_DIR and DEST_DIR.
The following solution worked well for me in various environments:
sourceDir="some/directory"
targetDir="any/other/directory"
find "$sourceDir" -type d | sed -e "s?$sourceDir?$targetDir?" | xargs mkdir -p
This solves even the problem with whitespaces:
In the original/source dir:
find . -type d -exec echo "'{}'" \; > dirs2.txt
then recreate it in the newly created dir:
mkdir -p <../<SOURCEDIR>/dirs2.txt
Substitute target_dir and source_dir with the appropriate values:
cd target_dir && (cd source_dir; find . -type d ! -name .) | xargs -i mkdir -p "{}"
Tested on OSX+Ubuntu.
If you can get access from a Windows machine, you can use xcopy with /T and /E to copy just the folder structure (the /E includes empty folders)
http://ss64.com/nt/xcopy.html
[EDIT!]
This one uses rsync to recreate the directory structure but without the files.
http://psung.blogspot.com/2008/05/copying-directory-trees-with-rsync.html
Might actually be better :)
A python script from Sergiy Kolodyazhnyy
posted on Copy only folders not files?:
#!/usr/bin/env python
import os,sys
dirs=[ r for r,s,f in os.walk(".") if r != "."]
for i in dirs:
os.makedirs(os.path.join(sys.argv[1],i))
or from the shell:
python -c 'import os,sys;dirs=[ r for r,s,f in os.walk(".") if r != "."];[os.makedirs(os.path.join(sys.argv[1],i)) for i in dirs]' ~/new_destination
FYI:
Copy top level folder structure without copying files in linux
How do I copy a directory tree but not the files in Linux?
Another approach is use the tree which is pretty handy and navigating directory trees based on its strong options. There are options for directory only, exclude empty directories, exclude names with pattern, include only names with pattern, etc. Check out man tree
Advantage: you can edit or review the list, or if you do a lot of scripting and create a batch of empty directories frequently
Approach: create a list of directories using tree, use that list as an arguments input to mkdir
tree -dfi --noreport > some_dir_file.txt
-dfi lists only directories, prints full path for each name, makes tree not print the indentation lines,
--noreport Omits printing of the file and directory report at the end of the tree listing, just to make the output file not contain any fluff
Then go to the destination where you want the empty directories and execute
xargs mkdir < some_dir_file.txt
find source/ -type f | rsync -a --exclude-from - source/ target/
Copy dir only with associated permission and ownership
Simple way:
for i in `find . -type d`; do mkdir /home/exemplo/$i; done
cd oldlocation
find . -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}
You can also create top directories only:
cd oldlocation
find . -maxdepth 1 -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}
Here is a solution in php that:
copies the directories (not recursively, only one level)
preserves permissions
unlike the rsync solution, is fast even with directories containing thousands of files as it does not even go into the folders
has no problems with spaces
should be easy to read and adjust
Create a file like syncDirs.php with this content:
<?php
foreach (new DirectoryIterator($argv[1]) as $f) {
if($f->isDot() || !$f->isDir()) continue;
mkdir($argv[2].'/'.$f->getFilename(), $f->getPerms());
chown($argv[2].'/'.$f->getFilename(), $f->getOwner());
chgrp($argv[2].'/'.$f->getFilename(), $f->getGroup());
}
Run it as user that has enough rights:
sudo php syncDirs.php /var/source /var/destination
In a shell script how would I find a file by a particular name and then navigate to that directory to do further operations on it?
From here I am going to copy the file across to another directory (but I can do that already just adding it in for context.)
You can use something like:
cd -- "$(dirname "$(find / -type f -name ls | head -1)")"
This will locate the first ls regular file then change to that directory.
In terms of what each bit does:
The find will start at / and search down, listing out all regular files (-type f) called ls (-name ls). There are other things you can add to find to further restrict the files you get.
The | head -1 will filter out all but the first line.
$() is a way to take the output of a command and put it on the command line for another command.
dirname can take a full file specification and give you the path bit.
cd just changes to that directory, the -- is used to prevent treating a directory name beginning with a hyphen from being treated as an option to cd.
If you execute each bit in sequence, you can see what happens:
pax[/home/pax]> find / -type f -name ls
/usr/bin/ls
pax[/home/pax]> find / -type f -name ls | head -1
/usr/bin/ls
pax[/home/pax]> dirname "$(find / -type f -name ls | head -1)"
/usr/bin
pax[/home/pax]> cd -- "$(dirname "$(find / -type f -name ls | head -1)")"
pax[/usr/bin]> _
The following should be more safe:
cd -- "$(find / -name ls -type f -printf '%h' -quit)"
Advantages:
The double dash prevents the interpretation of a directory name starting with a hyphen as an option (find doesn't produce such file names, but it's not harmful and might be required for similar constructs)
-name check before -type check because the latter sometimes requires a stat
No dirname required because the %h specifier already prints the directory name
-quit to stop the search after the first file found, thus no head required which would cause the script to fail on directory names containing newlines
no one suggesting locate (which is much quicker for huge trees) ?
zsh:
cd $(locate zoo.txt|head -1)(:h)
cd ${$(locate zoo.txt)[1]:h}
cd ${$(locate -r "/zoo.txt$")[1]:h}
or could be slow
cd **/zoo.txt(:h)
bash:
cd $(dirname $(locate -l1 -r "/zoo.txt$"))
Based on this answer to a similar question, other useful choice could be having 2 commands, 1st to find the file and 2nd to navigate to its directory:
find ./ -name "champions.txt"
cd "$(dirname "$(!!)")"
Where !! is history expansion meaning 'the previous command'.
Expanding on answers already given, if you'd like to navigate iteratively to every file that find locates and perform operations in each directory:
for i in $(find /path/to/search/root -name filename -type f)
do (
cd $(dirname $(realpath $i));
your_commands;
)
done
if you are just finding the file and then moving it elsewhere, just use find and -exec
find /path -type f -iname "mytext.txt" -exec mv "{}" /destination +;
function fReturnFilepathOfContainingDirectory {
#fReturnFilepathOfContainingDirectory_2012.0709.18:19
#$1=File
local vlFl
local vlGwkdvlFl
local vlItrtn
local vlPrdct
vlFl=$1
vlGwkdvlFl=`echo $vlFl | gawk -F/ '{ $NF="" ; print $0 }'`
for vlItrtn in `echo $vlGwkdvlFl` ;do
vlPrdct=`echo $vlPrdct'/'$vlItrtn`
done
echo $vlPrdct
}
Simply this way, isn't this elegant?
cdf yourfile.py
Of course you need to set it up first, but you need to do this only once:
Add following line into your .bashrc or .zshrc, whatever you use as your shell initialization script.
source ~/bin/cdf.sh
And add this code into ~/bin/cdf.sh file that you need to create from scratch.
#!/bin/bash
function cdf() {
THEFILE=$1
echo "cd into directory of ${THEFILE}"
# For Mac, replace find with mdfind to get it a lot faster. And it does not need args ". -name" part.
THEDIR=$(find . -name ${THEFILE} |head -1 |grep -Eo "/[ /._A-Za-z0-9\-]+/")
cd ${THEDIR}
}
If it's a program in your PATH, you can do:
cd "$(dirname "$(which ls)")"
or in Bash:
cd "$(dirname "$(type -P ls)")"
which uses one less external executable.
This uses no externals:
dest=$(type -P ls); cd "${dest%/*}"
If your file is only in one location you could try the following:
cd "$(find ~/ -name [filename] -exec dirname {} \;)" && ...
You can use -exec to invoke dirname with the path that find returns (which goes where the {} placeholder is). That will change directories. You can also add double ampersands ( && ) to execute the next command after the shell has changed directory.
For example:
cd "$(find ~/ -name need_to_find_this.rb -exec dirname {} \;)" && ruby need_to_find_this.rb
It will look for that ruby file, change to the directory, then run it from within that folder. This example assumes the filename is unique and that for some reason the ruby script has to run from within its directory. If the filename is not unique you'll get many locations passed to cd, it will return an error then it won't change directories.
try this. i created this for my own use.
cd ~
touch mycd
sudo chmod +x mycd
nano mycd
cd $( ./mycd search_directory target_directory )"
if [ $1 == '--help' ]
then
echo -e "usage: cd \$( ./mycd \$1 \$2 )"
echo -e "usage: cd \$( ./mycd search_directory target_directory )"
else
find "$1"/ -name "$2" -type d -exec echo {} \; -quit
fi
cd -- "$(sudo find / -type d -iname "dir name goes here" 2>/dev/null)"
keep all quotes (all this does is just send you to the directory you want, after that you can just put commands after that)