find list of files, make a directory and copy those files in linux - linux

For example, I want to copy the "file-to-be-copied.txt" from different directories
/home/user1/file-to-be-copied.txt
/home/user2/file-to-be-copied.txt
/home/user3/file-to-be-copied.txt
Then create a new directory based on the user account
/home/user4/user1/
/home/user4/user2/
/home/user4/user3/
Then copy the "file-to-be-copied.txt" to the new created directories
/home/user4/user1/file-to-be-copied.txt
/home/user4/user2/file-to-be-copied.txt
/home/user4/user3/file-to-be-copied.txt
All I know is that it should be done using bash scripting but I don't know how. This is as far as I go
find /home . "file-to-be-copied.txt" | xargs -i mkdir ... cp {} ...

No find necessary, and more so: no xargs (which is almost always superfluous with find, since find has -exec).
cd /home
cp --parents user?/file-to-be-copied.txt user4

for f in $(/usr/bin/find '/home' -name 'file-to-be-copied.txt'); do
tmpname=${f%/*}
dirname=${tmpname##*/}
/bin/mkdir -p $dirname && /bin/cp -p $f $dirname
done

This is the code I used.
for f in $(find '/home' -name 'file-to-be-copied.txt')
do
tmpname=${f%/*}
dirname=${tmpname##*/}
mkdir -p /home/user4/$dirname && /bin/cp -p $f /home/user4/$dirname
echo $f copied to /home/user4/$dirname
done

Related

How to copy list of folders keeping the partial folder structure that of parent directory

Say I want to copy some folders from find /40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD -name "plink_data*" which has these folders:
/40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD/ADGC_NHW/ADNI/TOPMEDr2/vcffile/plink_data
/40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD/ADGC_ASIAN/BIOCARD/TOPMEDr2/vcffile/plink_data
into /40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD/NEW_DIR/, but I want the new directory to have :
/40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD/NEW_DIR/ADGC_NHW/ADNI/TOPMEDr2/vcffile/plink_data
/40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD/NEW_DIR/ADGC_ASIAN/BIOCARD/TOPMEDr2/vcffile/plink_data
I tried this but it copies the whole path: find /40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD -name "plink_data*" -exec cp --parents {} /target \;
How do I go about doing it? Thanks!
UPDATE: I was able to perform my task with cp using answer from #Cyrus, but not with mv. I thought applying cp and mv in this command would not be any different, but I was wrong. In fact, I needed to use both mv and cp for different tasks, so I resorted to using a loop
for line in $(find . -name "*plink_data*"); do
new_FOLD="$(echo $line| cut -d"." -f2-)"
mkdir -p "NEW_DIR/${new_FOLD}"
cp/mv $line "NEW_DIR/${new_FOLD}"
done
I suggest:
cd /40/AD/GWAS_data/Source_Plink/2021_ADGC_EOAD
mkdir -p NEW_DIR
find . -name "plink_data" -not -path "./NEW_DIR/*" -exec cp --parent {} NEW_DIR \;

Find and move files to appropriate directories using shell script

I have below setup and I want to find and move files.
I have files /home/backup/abc/123.wav and /home/backup/xyz/456.wav.
Same directories exist at /usr/src/abc and /usr/src/xyz which does not have any files.
I want to find .wav files from home_dir and move them to particular dest_dir.
So 123.wav should move to /usr/src/abc and 456.wav should move to /usr/src/xyz.
I am using below command for that.
home_dir=/home/backup/
dest_dir=/usr/src/
cd $home_dir && find . -iname "*.wav" -exec mv {} $dest_dir \;
But all the .wav files(123.wav and 456.wav) moved to /usr/src and not to its respective directories(/usr/src/abc and /usr/src/xyz).
Is it possible to achieve what I want ?
Please suggest.
Use cp --parents option with find to create parent directories of each file being copied:
home_dir=/home/backup/
dest_dir=/usr/src/
cd "$home_dir"
find . -iname "*.wav" -exec cp --parents {} "$dest_dir" \; -delete
This would be a lot easier if mv had a --parents option, but unfortunately it doesn't. It's best to use mv instead of cp because cp will copy all the data unnecessarily if the source and destination directories are on the same filesystem. if you've got Bash 4 (which supports globstar) you could do it like this:
home_dir=/home/backup/
dest_dir=/usr/src/
shopt -s globstar nullglob dotglob
for src_wav in "$home_dir"/**/*.wav ; do
rel_wav=${src_wav#$home_dir/}
dst_wav=$dest_dir/$rel_wav
dst_parent=${dst_wav%/*}
[[ -d $dst_parent ]] || mkdir -p -- "$dst_parent"
mv -- "$src_wav" "$dst_wav"
done

copy entire directory excluding a file

As we know, cp -r source_dir intended_new_directory creates a copy of source directory with a new name. Now I want to do the same but want to exclude a particular file. I have found some related answers here, using tar and rsync, but in those solutions I need to create the destination directory first (using mkdir).
I honestly searched a lot, but didn't find exactly what I want.
So far the best I got is this:
tar -c --exclude=\*.dll --exclude=\*.exe sourceDir | tar -x -C destDir
(from http://www.linuxquestions.org/questions/programming-9/how-to-copy-an-entire-directory-structure-except-certain-files-385321/)
If you have binutils, you could use find to filter next cpio to copy (and create directories) :
find <sourceDir> \( ! -name *.dll \) -a \( ! -name *.exe \) | cpio -dumpv <destDir>
Try this by excluding the file using 'grep -v' ->
cp `ls | grep -v <exclude-file>` <dest-dir>
If the directory is not very large I used to write something like this:
src=path/to/source/directory
dst=path/to/destination/directory
find $src -type f | while read f ; do mkdir -p "$dst/`dirname $f`"; cp "$f" "$dst/$f" ; done
Here we list all regular files in $src, iterate over this list and for each file make a directory in $dst if it does not exist yet (-p option of mkdir), then copy the file to that directory.
The above command will copy all the files. Finally, just use
find $src -type f | grep -v whatever | while ...... # same as above
to filter out the files you don't need (e.g. \.bak$, \.orig$, or whatever files you don't want to copy).
Move all exclude file into home or other directory,copy the directory containing all remaining files to the destination folder then restore all exclude files.
#cd mydirectory
#mv exclude1 exclude2 /home/
#cp mydirectory destination_folder/
#cd /home/
#mv eclude1 exclude2 mydirectory/

moving files to different directories

I'm trying to move media and other files which are in a specified directory to another directory and create another one if it does not exits (where the files will go), and create a directory the remaining files with different extensions will go. My first problem is that my script is not making a new directory and it is not moving the files to other directories and what code can I use to move files with different extensions to one directory?
This is what i have had so far, correct me where I'm wrong and help modify my script:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo #directory where the media files will go
WA=/home/katy/do # directory where the other files will go
if [ ! -d "$To" ]; then
mkdir -p "$To"
fi
cd $From
find path -type f -name"*.mp4" -exec mv {} $To \;
I'd solve it somewhat like this:
#!/bin/bash
From=/home/katy/doc
To=/home/katy/mo # directory where the media files will go
WA=/home/katy/do # directory where the other files will go
cd "$From"
find . -type f \
| while read file; do
dir="$(dirname "$file")"
base="$(basename "$file")"
if [[ "$file" =~ \.mp4$ ]]; then
target="$To"
else
target="$WA"
fi
mkdir -p "$target/$dir"
mv -i "$file" "$target/$dir/$base"
done
Notes:
mkdir -p will not complain if the directory already exists, so there's no need to check for that.
Put double quotes around all filenames in case they contain spaces.
By piping the output of find into a while loop, you also avoid getting bitten by spaces, because read will read until a newline.
You can modify the regex according to taste, e.g. \.(mp3|mp4|wma|ogg)$.
In case you didn't know, $(...) will run the given command and stick its output back in the place of the $(...) (called command substitution). It is almost the same as `...` but slightly better (details).
In order to test it, put echo in front of mv. (Note that quotes will disappear in the output.)
cd $From
find . -type f -name "*.mp4" -exec mv {} $To \;
^^^
or
find $From -type f -name "*.mp4" -exec mv {} $To \;
^^^^^
cd $From
mv *.mp4 $To;
mv * $WA;

Copy folder structure (without files) from one location to another

I want to create a clone of the structure of our multi-terabyte file server. I know that cp --parents can move a file and it's parent structure, but is there any way to copy the directory structure intact?
I want to copy to a linux system and our file server is CIFS mounted there.
You could do something like:
find . -type d > dirs.txt
to create the list of directories, then
xargs mkdir -p < dirs.txt
to create the directories on the destination.
cd /path/to/directories &&
find . -type d -exec mkdir -p -- /path/to/backup/{} \;
Here is a simple solution using rsync:
rsync -av -f"+ */" -f"- *" "$source" "$target"
one line
no problems with spaces
preserve permissions
I found this solution there
1 line solution:
find . -type d -exec mkdir -p /path/to/copy/directory/tree/{} \;
I dunno if you are looking for a solution on Linux. If so, you can try this:
$ mkdir destdir
$ cd sourcedir
$ find . -type d | cpio -pdvm destdir
This copy the directories and files attributes, but not the files data:
cp -R --attributes-only SOURCE DEST
Then you can delete the files attributes if you are not interested in them:
find DEST -type f -exec rm {} \;
This works:
find ./<SOURCE_DIR>/ -type d | sed 's/\.\/<SOURCE_DIR>//g' | xargs -I {} mkdir -p <DEST_DIR>"/{}"
Just replace SOURCE_DIR and DEST_DIR.
The following solution worked well for me in various environments:
sourceDir="some/directory"
targetDir="any/other/directory"
find "$sourceDir" -type d | sed -e "s?$sourceDir?$targetDir?" | xargs mkdir -p
This solves even the problem with whitespaces:
In the original/source dir:
find . -type d -exec echo "'{}'" \; > dirs2.txt
then recreate it in the newly created dir:
mkdir -p <../<SOURCEDIR>/dirs2.txt
Substitute target_dir and source_dir with the appropriate values:
cd target_dir && (cd source_dir; find . -type d ! -name .) | xargs -i mkdir -p "{}"
Tested on OSX+Ubuntu.
If you can get access from a Windows machine, you can use xcopy with /T and /E to copy just the folder structure (the /E includes empty folders)
http://ss64.com/nt/xcopy.html
[EDIT!]
This one uses rsync to recreate the directory structure but without the files.
http://psung.blogspot.com/2008/05/copying-directory-trees-with-rsync.html
Might actually be better :)
A python script from Sergiy Kolodyazhnyy
posted on Copy only folders not files?:
#!/usr/bin/env python
import os,sys
dirs=[ r for r,s,f in os.walk(".") if r != "."]
for i in dirs:
os.makedirs(os.path.join(sys.argv[1],i))
or from the shell:
python -c 'import os,sys;dirs=[ r for r,s,f in os.walk(".") if r != "."];[os.makedirs(os.path.join(sys.argv[1],i)) for i in dirs]' ~/new_destination
FYI:
Copy top level folder structure without copying files in linux
How do I copy a directory tree but not the files in Linux?
Another approach is use the tree which is pretty handy and navigating directory trees based on its strong options. There are options for directory only, exclude empty directories, exclude names with pattern, include only names with pattern, etc. Check out man tree
Advantage: you can edit or review the list, or if you do a lot of scripting and create a batch of empty directories frequently
Approach: create a list of directories using tree, use that list as an arguments input to mkdir
tree -dfi --noreport > some_dir_file.txt
-dfi lists only directories, prints full path for each name, makes tree not print the indentation lines,
--noreport Omits printing of the file and directory report at the end of the tree listing, just to make the output file not contain any fluff
Then go to the destination where you want the empty directories and execute
xargs mkdir < some_dir_file.txt
find source/ -type f | rsync -a --exclude-from - source/ target/
Copy dir only with associated permission and ownership
Simple way:
for i in `find . -type d`; do mkdir /home/exemplo/$i; done
cd oldlocation
find . -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}
You can also create top directories only:
cd oldlocation
find . -maxdepth 1 -type d -print0 | xargs -0 -I{} mkdir -p newlocation/{}
Here is a solution in php that:
copies the directories (not recursively, only one level)
preserves permissions
unlike the rsync solution, is fast even with directories containing thousands of files as it does not even go into the folders
has no problems with spaces
should be easy to read and adjust
Create a file like syncDirs.php with this content:
<?php
foreach (new DirectoryIterator($argv[1]) as $f) {
if($f->isDot() || !$f->isDir()) continue;
mkdir($argv[2].'/'.$f->getFilename(), $f->getPerms());
chown($argv[2].'/'.$f->getFilename(), $f->getOwner());
chgrp($argv[2].'/'.$f->getFilename(), $f->getGroup());
}
Run it as user that has enough rights:
sudo php syncDirs.php /var/source /var/destination

Resources