I want to transfer selective sub folders from a range of parent folders:
/home/user/sample_rsync/
FolderA/sub1
FolderA/sub2
FolderA/sub3
FolderB/sub1
FolderB/sub2
FolderB/sub3
FolderC/sub1
FolderC/sub2
FolderC/sub3
Say from the above example I want to copy just sub1 from each directory. i.e. in my destination I want the following folders to be created (along with the files they contain)
/destination/
sample_rsync/FolderA/sub1
sample_rsync/FolderB/sub1
sample_rsync/FolderC/sub1
How do I go about doing this?
I tried out
rsync -avh -f"- *" -f"+ *sub1/*" /home/user/sample_rsync /destination/
In an attempt to exclude everything and then just include sub1's - didnt work.
Any way I can get this working?
Assuming your source folders are in a file called "sources" as typed in your first code sement (without trailing / characters)
for s in $(cat sources)
do
rsync -av ${s} /destination/sample_rsync/$(echo ${s}| awk -F "/" '{print $1}')
done
of course this is only valid if you have a certain level deep directories in your sources file. If the depth level of the directories to be copied changes, this script will need to be heavily modified. But at least it is a starting point I hope.
upon your question below, you might want to use something like this: (ignore the code segment above. I just left it there for history purposes)
cd /home/user/sample_rsync
for dir in $(find ./ -type d -name sub1)
do
dest=$(echo ${dir} | sed -e "1,1s+/sub1++")
mkdir /destination/sample_rsync/${dest}
rsync -av ${dir} /destination/sample_rsync/${dest}
done
please do not take it as the word of gospel. I have not tested the code whatsoever. So. it might yield some unexpected results. Please test it on a system that you wouldn't mind having problems if it gets haywire.
Related
I have a folder/subfolders that contain some files with filenames that end with a random numeric extension:
DWH..AUFTRAG.20211123115143.A901.3801176
DWH..AUFTRAGSPOSITION.20211122002147.A901.3798013
I would like to remove everything after A901 from the above filenames.
For example:
DWH..AUFTRAG.20211123115143.A901 (remove this .3801176)
DWH..AUFTRAGSPOSITION.20211122002147.A901 (remove this .3798013) from the filename
How do I use rename or any other command in linux to remove only after A901 everything from finale rest file name keep as it is?
I can see there is 5 '.' (dots) before the number so I did some desi jugad.
I made some files in folder and also made a folder and created some files inside that folder accourding to the name pattern that you gave.
I created a command and it somewhat looks like this.
find "$PWD"|grep A901|while read F; do mv "${F}" `echo ${F}|cut -d . -f 1-5`;done
When executed it worked for me.
terminal output below.
rexter#rexter:~/Desktop/test$ find $PWD
/home/rexter/Desktop/test
/home/rexter/Desktop/test/test1
/home/rexter/Desktop/test/test1/DWH..AUFTRAG.20211123115143.A901.43214
/home/rexter/Desktop/test/test1/DWH..AUFTRAGSPOSITION.2021112200fsd2147.A901.31244324
/home/rexter/Desktop/test/DWH..AUFTRAG.20211123115143.A901.321423
/home/rexter/Desktop/test/DWH..AUFTRAGSPOSITION.20211122002147.A901.3124325
rexter#rexter:~/Desktop/test$ find "$PWD"|grep A901|while read F; do mv "${F}" `echo ${F}|cut -d . -f 1-5`;done
rexter#rexter:~/Desktop/test$ find $PWD
/home/rexter/Desktop/test
/home/rexter/Desktop/test/test1
/home/rexter/Desktop/test/test1/DWH..AUFTRAG.20211123115143.A901
/home/rexter/Desktop/test/test1/DWH..AUFTRAGSPOSITION.2021112200fsd2147.A901
/home/rexter/Desktop/test/DWH..AUFTRAG.20211123115143.A901
/home/rexter/Desktop/test/DWH..AUFTRAGSPOSITION.20211122002147.A901
rexter#rexter:~/Desktop/test$
I dont know if this is a proper way to do it but it just make things work.
Let me know if it is useful to you.
This question already has answers here:
Shell script: How to delete all files in a directory except ones listed in a file?
(2 answers)
Closed 2 years ago.
I have a directory called a00 containing 3000 files with extension .SAC. I have a text file called gd.list containing names of 88 of those 3000 files. I am trying to write a code that will delete all .SAC files except those mentioned in gd.list
How to do that using shell/bash?
The rm command is commented out so that you can check and verify that it's working as needed. Then just un-comment that line.
The check directory section will ensure you don't accidentally run the script from the wrong directory and clobber the wrong files.
You can remove the echo deleting line to run silently.
#!/bin/bash
cd /home/me/myfolder2tocleanup/
# Exit if the directory isn't found.
if (($?>0)); then
echo "Can't find work dir... exiting"
exit
fi
for i in *; do
if ! grep -qxFe "$i" filelist.txt; then
echo "Deleting: $i"
# the next line is commented out. Test it. Then uncomment to removed the files
# rm "$i"
fi
done
You can find the answer here https://askubuntu.com/questions/830776/remove-file-but-exclude-all-files-in-a-list by L. D. James
there are a few alternatives.
I'd prefer to see find -Z as it more clearly demarcates the file names:
find . -maxdepth 1 -name '*.sac' -print0 | grep -x -z -Z -f gd.list | xargs -0 echo rm
Again, test this first. Perhaps sort the output and make sure it is unique versus the original file.
For a smaller list of filenames I would recommend just using find with -and -not -name and -delete, but with a larger list that can be tricky.
You could tag the files you want to keep as read-only, then delete the wildcard with the appropriate setting in rm or find to skip read-only files. That assumes you own the read-only flag. You could tag the files as executable, and use find, if the read-only flag is not for you.
Another option would be to move the matching files to a temp folder, delete the wildcard, then move the files you want to keep back. That is assuming you can afford for the files to disappear temporarily.
To make them disappear for a shorter time, move the kept files out to a temp directory, move the original directory out, move the temp directory in, then delete the movced out directory.
If you are feeling brave, try something like
ls *.sac | fgrep -v -f gd.list | xargs echo rm
Note that I've put an echo in that xargs, just to make sure no one has a cut and paste accident.
Note also the limitations of this approach mentioned in the comments. As I said, if you are feeling brave...
I have this question after quite a day of searching the net, perhaps I'm doing something wrong , here is my script:
#!/bin/bash
shopt -s extglob
FILE_EXTENSIONS=properties\|xml\|sh\|sql\|ksh
SOURCE_FOLDER=$1
if [ -z "$SOURCE_FOLDER" ]; then
SOURCE_FOLDER=$(pwd)
fi # Set directory to current working folder if no input parameter.
for file in $SOURCE_FOLDER/**/*.*($FILE_EXTENSIONS)
do
echo Working with file: $file
done
Basically, I want to recursively get all the files filtered by a list of extensions within folders from a directory that is passed as an argument including the directory itself.
I would like to know if there is a way of doing this and how without the use of the find command.
Imagine I have this file tree:
bin/props.properties
bin/xmls.xml
bin/source/sources.sh
bin/config/props.properties
bin/config/folders/moreProps.xml
My script, as it is right now and running from /bin, would echo:
bin/source/sources.sh
bin/config/props.properties
bin/config/folders/moreProps.xml
Leaving the ones in the working path aside.
P.S. I know this can be done with find but I really want to know if there's another way for the sake of learning.
Thanks!
You can use find with grep, just like this:
#!/bin/bash
SOURCE_FOLDER=$1
EXTENSIONS="properties|xml|sh|sql|ksh"
find $SOURCE_FOLDER | grep -E ".(${EXTENSIONS})"
#or even better
find $SOURCE_FOLDER -regextype posix-egrep -regex ".*(${EXTENSIONS})"
I want to write a script that do specific thing:
I have a txt file e.g.
from1/from2/from3/apple.file;/to1/to2/to3;some not important stuff
from1/from2/banana.file;/to1/to5;some not important stuff
from1/from10/plum.file;/to1//to5/to100;some not important stuff
Now i want to copy file from each line (e.g. apple.file), from original directory tree to new, non existing directories, after first semicolon (;).
I try few code examples from similar questions, but nothing works fine and I'm too weak in bash scripting, to find errors.
Please help :)
need to add some conditions:
file not only need to be copy, but also rename. Example line in file.txt:
from1/from2/from3/apple.file;to1/to2/to3/juice.file;some1
from1/from2/banana.file;to1/to5/fresh.file;something different from above
so apple.file need to be copy and rename to juice.file and put in to1/to2/to3/juice.file
I think thaht cp will also rename file but
mkdir -p "$to"
from answer below will create full folder path with juice.file as folder
In addidtion after second semicolon in each line will be something different, so how to cut it off?
Thanks for all help
EDIT: There will be no spaces in input txt file.
Try this code..
cat file | while IFS=';' read from to some_not_important_stuff
do
to=${to:1} # strip off leading space
mkdir -p "$to" # create parent for 'to' if not existing yet
cp -i "$from" "$to" # option -i to get a warning when it would overwrite something
done
Using awk
(run the awk command first and confirm the output is fine, then add |sh to do the copy)
awk -F";" '{printf "cp %s %s\n",$1,$2}' file |sh
Using shell (get updated that need manually create folder, base on alfe's
while IFS=';' read from to X
do
mkdir -p $to
cp $from $to
done < file
I had this same problem and used tar to solve it! Posted here:
tmpfile=/tmp/myfile.tar
files="/some/folder/file1.txt /some/other/folder/file2.txt"
targetfolder=/home/you/somefolder
tar --file="$tmpfile" "$files"
tar --extract --file="$tmpfile" --directory="$targetfolder"
In this case, tar will automatically create all (sub)folders for you! Best,
Nabi
I use the following bash script to copy only files of certain extension(in this case *.sh), however it still copies over all the files. what's wrong?
from=$1
to=$2
rsync -zarv --include="*.sh" $from $to
I think --include is used to include a subset of files that are otherwise excluded by --exclude, rather than including only those files.
In other words: you have to think about include meaning don't exclude.
Try instead:
rsync -zarv --include "*/" --exclude="*" --include="*.sh" "$from" "$to"
For rsync version 3.0.6 or higher, the order needs to be modified as follows (see comments):
rsync -zarv --include="*/" --include="*.sh" --exclude="*" "$from" "$to"
Adding the -m flag will avoid creating empty directory structures in the destination. Tested in version 3.1.2.
So if we only want *.sh files we have to exclude all files --exclude="*", include all directories --include="*/" and include all *.sh files --include="*.sh".
You can find some good examples in the section Include/Exclude Pattern Rules of the man page
The answer by #chepner will copy all the sub-directories whether it contains files or not. If you need to exclude the sub-directories that don't contain the file and still retain the directory structure, use
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
Here's the important part from the man page:
As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.
To summarize:
Not matching any pattern means a file will be copied!
The algorithm quits once any pattern matches
Also, something ending with a slash is matching directories (like find -type d would).
Let's pull apart this answer from above.
rsync -zarv --prune-empty-dirs --include "*/" --include="*.sh" --exclude="*" "$from" "$to"
Don't skip any directories
Don't skip any .sh files
Skip everything
(Implicitly, don't skip anything, but the rule above prevents the default rule from ever happening.)
Finally, the --prune-empty-directories keeps the first rule from making empty directories all over the place.
One more addition: if you need to sync files by its extensions in one dir only (without of recursion) you should use a construction like this:
rsync -auzv --include './' --include '*.ext' --exclude '*' /source/dir/ /destination/dir/
Pay your attention to the dot in the first --include. --no-r does not work in this construction.
EDIT:
Thanks to gbyte.co for the valuable comment!
EDIT:
The -uzv flags are not related to this question directly, but I included them because I use them usually.
Wrote this handy function and put in my bash scripts or ~/.bash_aliases. Tested sync'ing locally on Linux with bash and awk installed. It works
selrsync(){
# selective rsync to sync only certain filetypes;
# based on: https://stackoverflow.com/a/11111793/588867
# Example: selrsync 'tsv,csv' ./source ./target --dry-run
types="$1"; shift; #accepts comma separated list of types. Must be the first argument.
includes=$(echo $types| awk -F',' \
'BEGIN{OFS=" ";}
{
for (i = 1; i <= NF; i++ ) { if (length($i) > 0) $i="--include=*."$i; } print
}')
restargs="$#"
echo Command: rsync -avz --prune-empty-dirs --include="*/" $includes --exclude="*" "$restargs"
eval rsync -avz --prune-empty-dirs --include="*/" "$includes" --exclude="*" $restargs
}
Advantages:
short handy and extensible when one wants to add more arguments (i.e. --dry-run).
Example:
selrsync 'tsv,csv' ./source ./target --dry-run
If someone looks for this…
I wanted to rsync only specific files and folders and managed to do it with this command: rsync --include-from=rsync-files
With rsync-files:
my-dir/
my-file.txt
- /*