I have a directory that is filled with subdirectories exceeding 450 GBs. Inside of these subdirectories is an instruction file in each subdirectory. I have a script that copies the instruction file in the directory I am currently in and puts it inside every subdirectory via:
#!/bin/bash
for d in */; do cp "INSTALLATION INSTRUCTIONS.rtf" "$d"; done
I need to remove all of these files in the subdirectories and replace them with new instructions. Can I simple write another script that does this:
#!/bin/bash
for d in */; do rm "INSTALLATION INSTRUCTIONS.rtf" "$d"; done
I am very hesitant and wanted to make sue as these files are vitally important and I don't want to accidentally remove anything and making a backup of 450+ GBs is very taxing.
find . -mindepth 2 -name "INSTALLATION INSTRUCTIONS.rtf" -exec rm -f '{}' +
Since this is "vitally important" data, I would first list all files that match the file name you want to delete/overwrite, without taking any action on it (other than listing):
find /folder/ -type f -name "INSTALLATION INSTRUCTIONS.rtf" -print > /tmp/holder
That would create a list of matches on /tmp/holder. Then you could analyze this list before taking any action (either visually or programatically) to make sure that the list does not include anything you don't want to delete (when dealing with big amounts of data, strange things can happen, so be proactive on protecting the data).
If you are happy with what the list shows, then you could delete the old instructions, or if possible, overwrite them with the new file. Here's an example to overwrite the old file with the new one:
while read -r line; do cp --no-preserve=all /folder/newfile "$line"; done < /tmp/holder
The cp --no-preserve=all command (available on GNU bash) would ensure that the new file has permissions that are "adequate" to the folder where they are located. You may change that to a simple cp if you don't want that to happen.
Related
Maybe i'm just going about this wrong and making it harder than it has to be.
This is my problem. I have 2 different scripts that download various picture files. the first downloads from email and the downloaded files go into the /attachments/ directory. The second script copies the contents of google drive, all files and folders get copied into ~/gdrive/ directory. i want to be able to move all picture files from both these folders as well as any subfolders to ~/Pictures/$today and prevent any overwriting in the case of duplicate file names. I don't mind having 2 separate scripts to handle the pictures in the 2 different directories, but I do need it to be able to get all files in subdirectories of the starting point. it also needs to be able to handle a variety of file extensions. my current solution adds a numbered extension such as .~1~ after the files normal extension .jpg, .png, .tiff, etc. I dont lose any files this way but any that wind up with a backup number after the extension are rendered useless to my project. This is what I am currently using
TODAY=$(date +"%m-%d-%Y")
mkdir -p ~/Pictures/$TODAY &&
sudo find /attachments -type f -exec mv --backup=numbered -t ~/Pictures/$TODAY {} +
My result if there are duplicate file names looks like this:
DSC07286.JPG
DSC07286.JPG.~1~
Is there a better approach than what i am doing? Is there a way to dissect the filename parts and reorganize them and do it recursively for all files in the directory? Thanks
Something like this should do it (untested; uses standard lowercase variable names and puts the index just before the extension to not mess with sorting):
for path in ~/Pictures/"$today"/*.JPG
do
index=0
for duplicate_path in "$path".~[0-9]*
do
new_path="${duplicate_path%%.*}${index}.JPG"
echo "$duplicate_path" "$new_path"
((++index))
done
done
When you're confident it's doing the right thing, simply replace echo with mv to actually move the files.
Here is my solution.
#!/bin/bash
TODAY=$(date +"%m-%d-%Y")
NOW=$(date +"%D %T")
sudo mkdir -p /home/pi/Pictures/emailpics/$TODAY &&
sudo find /attachments -type f -exec mv --backup=numbered -t /home/pi/Pictures/emailpics/$TODAY/ {} + &&
for f in /home/pi/Pictures/emailpics/$TODAY/*.~?~
do
fullfilename=$f
filepath=$(dirname "$fullfilename")
filename=$(basename "$fullfilename")
fname="${filename%.*}"
bkpnum="${filename##*.}"
file="${fname%.*}"
ext="${fname##*.}"
sudo mv $f $filepath/$file$bkpnum.$ext
done
Can't say i fully understand all the syntax for the parsing bits, but it works. maybe someone else can explain what is going on.
I need to create empty clone directories with the same directory names as subdirectories under another directory in another location, I need to accomplish this using bash scripting.
To be more specific, I have a number of directories that are being generated by a data logging system with each directory being named according to the day, month, and, year of creation/recording, hence I have a starting directory say 24012016 and so on with increments in the day number, month, and, year; also there are gaps in records for certain days due to technical reasons.
Each such directory contains files with two different extensions; I need a script that will create a directory with the same name, that is 24012016, as a subdirectory under another directory in another location, but without the files within it, and, also copy files, one of the two with different extensions into the new clone directory, have this process repeated for all the directories, have this process repeated for all the directories.
given old_root the base directory to start coping and new_root the base destination:
find <old_root> -type d -exec mkdir -p "<new_root>/{}" \;
the second part is unclear
For example, something like that:
cd ${where_to_search}
find . -type d | while read dir; do
mkdir -p ${new_parentdir}/${dir}
done
find . -type f -name "*.${extension_to_copy}" | while read file; do
cp ${file} ${new_parent}/${file};
done
I'm trying write a script in linux. Where I have some csv files in Two different folders(A and B) and then after some processing copy of rejected files are moving to Bad Folder.
SO I want bad files to be deleted from Table A and B which have copied to Bad Folder.
Can you help me to write this script for linux?
Best
lets say name of Bad Folder is 'badFolder' and considering 'A', 'B' and 'badFolder' are in same directory
Steps to delete files from folder A and B:
step 1: change current directory to your 'badFolder'
cd badFolder
step 2: delete identical files
find . -type f -exec rm -f ../A/{} \;
find . -type f -exec rm -f ../B/{} \;
The argument -type f tells to look for files, not directories.
The -exec ... \; argument tells that, once it finds a file in 'badFolder', it should run the command rm -f on its counterpart in the A subdirectory.
Because rm is given with the -f option, it will silently ignore files that don't exist.
Also, it will not prompt before deleting files. This is very handy when deleting a large number of files. However, be sure that you really want to delete the files before running this script.
#!/bin/bash
#Set the working folder in which you want to delete the file
Working_folder=/<Folder>/<path>
cd $Working_folder
#command to delete all files present in folders
rm <filenames seperated by space>
echo "files are deleted"
#if you want to delete all files you can use wild card character
# e.g. command rm *.*
# if you want to delete a particular file say for deleting .csv file you can use command rm *.csv command
Set variables containing the paths of your A, B and BAD directories.
Then you can do something along the lines of
for file in ls ${PATH_TO_BAD}
do
rm ${PATH_TO_A}/$file
rm ${PATH_TO_B}/$file
done
This is iterating over the BAD directory and any file it finds, it deletes from the A and B directories.
Let's assume I have following source directory
source/
subdir1/file1
subdir1/file2
subdir2/file3
subdir3/file4
and target directory
target
subdir1/file5
subdir2/file6
subdir4/file7
I would like to move content of source subdirectories to right target subdirectories so result look like this
target
subdir1/file1
subdir1/file2
subdir1/file5
subdir2/file6
subdir2/file3
subdir3/file4
subdir4/file7
Is there some Linux command to do this or must I write a script myself?
To suimmarize, it is important to move, not copy. That rules out cp and rsync but allows mv. mv, however, has the issue that it is not good at merging the old directory into the new.
In the examples that you gave, the target directory had the complete directory tree but lacked files. If that is the case, try:
cd /source ; find . -type f -exec sh -c 'mv "$1" "/target/$1"' _ {} \;
The above starts by selecting the source as the current directory with cd /source. Next, we use find which is the usual *nix utility for finding files/directories. In this case, give find the -type f option to tell it to look only for files. With the -exec option, we tell it to move any such files found to the target directory.
You have choices for how to deal with conflicts between the two directory trees. You can give mv the -f option and it will overwrite files in the target without asking, or you can give it the -n option and it will never overwrite a target file, or your can give it the -i option and it will ask you each time.
In case the target directory tree is incomplete
If the target directory tree is missing some directories that are in the source, the we have to create them on the fly. This adds just minor complication:
cd /source ; find . -type f -exec sh -c 'mkdir -p "/target/${1%/*}"; mv "$1" "/target/$1"' _ {} \;
The mkdir -p command assures that the directory we want exists before we try to move the file there.
Additional notes
The form ${1%/*} is an example of one of the shells powerful features called "parameter expansion". This particular feature is suffix removal. In general, it looks like ${parameter%word} which tells bash to expand word and remove it from the end of parameter. In our case, the name of the parameter is 1, meaning the first argument to the script. We want to remove the file name and just leave behind the directory that the file is in. So, the word /* tells the shell to remove the last slash and any characters which follow.
The commands above use both single and double quotes. They have to be copied exactly for the command to work properly.
To sync dorectory maybe used rsync
Example:
rsync -avzh source/ target/
More info man rsync
Move (no copy)
rsync --remove-source-files -avzh source/ target/
i'm trying to elaborate a command that will find files that haven't been modified in over 6 months and zip them with one command. Afterwards i want to delete all those files and i just archived.
My current command to find the directories with the files is
find /var/www -type d -mtime -400 ! -mtime -180 | xargs ls -l > testd.txt
This gave me all the directories including the files that are older than 6 months
Now i was wondering if there was a way of zipping all the results and deleting them afterwards. Something amongst the line of
find /var/www -type f -mtime -400 ! -mtime -180 | gzip -c archive.gz
If anyone knows the proper syntax to achieve this i'd love to know. Thakns!
Edit, after a few tests this command results in a corrupted file
find /var/www -mtime -900 ! -mtime -180 | xargs tar -cf test4.tar
Any ideas?
Break this into several distinct steps that you can implement and thoroughly test separately:
Build a list of files to be archived and then deleted, saved to a temp file
Use the list from step 1 to add the files to .tar.gz archives. Give the archive file a name following a specific pattern that won't appear in the files to be archived, and put it in a directory outside the hierarchy of files being archived.
Read back the files from the .tar.gz and compare them (or their hashes) to the original files to ENSURE that you got them all without corruption
Use the list from step 1 to delete the files. Do not use a wildcard for deletion. Put in some guard code to prevent deletion of any file matching the name pattern of the archive .tar.gz file(s) created in step 2.
When testing a script that can do irreversible damage, always code the dangerous command with a leading echo and leave it that way until you are sure everything works. Only then remove the echo.
Consider zip, it should meet your requirements.
find ... | zip -m# archive.zip
-m (move) deletes the input directories/files after making the specified zip archive.
-# takes the list of input files from standard input.
You may find more options which are useful to you in the zip manual, e. g.
-r (recurse) travels the directory structure recursively.
-sf (show-files) shows the files that would be operated on, then exits.
-t or --from-date operates on files not modified prior to the specified date.
-tt or --before-date operates on files not modified after or at the specified date.
This could possibly make findexpendable.
zip -mr --from-date 2012-09-05 --before-date 2013-04-13 archive /var/www