I'm trying write a script in linux. Where I have some csv files in Two different folders(A and B) and then after some processing copy of rejected files are moving to Bad Folder.
SO I want bad files to be deleted from Table A and B which have copied to Bad Folder.
Can you help me to write this script for linux?
Best
lets say name of Bad Folder is 'badFolder' and considering 'A', 'B' and 'badFolder' are in same directory
Steps to delete files from folder A and B:
step 1: change current directory to your 'badFolder'
cd badFolder
step 2: delete identical files
find . -type f -exec rm -f ../A/{} \;
find . -type f -exec rm -f ../B/{} \;
The argument -type f tells to look for files, not directories.
The -exec ... \; argument tells that, once it finds a file in 'badFolder', it should run the command rm -f on its counterpart in the A subdirectory.
Because rm is given with the -f option, it will silently ignore files that don't exist.
Also, it will not prompt before deleting files. This is very handy when deleting a large number of files. However, be sure that you really want to delete the files before running this script.
#!/bin/bash
#Set the working folder in which you want to delete the file
Working_folder=/<Folder>/<path>
cd $Working_folder
#command to delete all files present in folders
rm <filenames seperated by space>
echo "files are deleted"
#if you want to delete all files you can use wild card character
# e.g. command rm *.*
# if you want to delete a particular file say for deleting .csv file you can use command rm *.csv command
Set variables containing the paths of your A, B and BAD directories.
Then you can do something along the lines of
for file in ls ${PATH_TO_BAD}
do
rm ${PATH_TO_A}/$file
rm ${PATH_TO_B}/$file
done
This is iterating over the BAD directory and any file it finds, it deletes from the A and B directories.
Related
Lets say i have a folder named Pictures and I want to move or copy all files out of this folder.
However I also want to move and harvest all of the files who are in sub folders so:
Pictures/1.png
Pictures/yolo/2.png
Pictures/yolo/swag/sand/3.png
Pictures/extra/fire/4.png
I want to move or copy all these files to another folder like results so I get:
results/1.png
results/2.png
results/3.png
results/4.png
Only I have no idea in advance what sub folders will be in the Pictures folder.
How can I accomplish this in bash/shell scripts ?
I also appreciate making it file type neutral so any files are harvested from their directories (not only .png like in my example) and I have no idea what the file name will be (I only used 1...4 because i did not have any idea how to name them).
You can do it like this:
find /absolute/path/to/Pictures -type f -name '*.png' -exec mv -i {} /absolute/path/to/results \;
Another option is to use xargs
find /absolute/path/to/Pictures -name '*.png' | xargs -I files mv files /absolute/path/to/results
You can simply copy all files and subdirectories along with their contents using cp's recursive option:
cp -pr <source_path>/* <destination_path>/
But, moving them recursively is a bit tricky, you will need to create tar files of the subdirectories and move them and then untar the tar files in destination path. As this is a complex process, as a workaround, you can copy the files/directories recursively and then delete the files from original path.
cp -pr <source_path>/* <destination_path>/ && rm -rf <source_path>/*
I have a directory structure as :-
/usr/testing/member/
---> public--->folder1--->file1
\----> file2
---> folder3:- contains files and folders
---> folder4:- contains files and folders
---> several files
I want to keep the public folder and all its contents (further folders and files within it) but want to delete everything else under the directory /usr/testing/member/. But that also means member folder is not deleted.
Is there any shell script or command that can be used to achieve this exactly as i stated.
Here's one way to do it:
(cd /usr/testing/member; find . -maxdepth 1 \( ! -name . -a ! -name public \) -exec echo rm -fr {} +)
That is: cd into /usr/testing/member, find all files and directories there, without going further below, and exclude the current directory (".") and any file or directory named "public", and execute a command for the found files.
This will print what would be deleted.
Verify it looks good, and then drop the echo.
I think below will do the work,
$ cd /usr/testing/member/
$ rm -rf $(ls | grep -v "public")
explanation:
we are passing everything inside /usr/testing/member/ but public to rm by making use of -v(exclude) option of grep
I have a directory that is filled with subdirectories exceeding 450 GBs. Inside of these subdirectories is an instruction file in each subdirectory. I have a script that copies the instruction file in the directory I am currently in and puts it inside every subdirectory via:
#!/bin/bash
for d in */; do cp "INSTALLATION INSTRUCTIONS.rtf" "$d"; done
I need to remove all of these files in the subdirectories and replace them with new instructions. Can I simple write another script that does this:
#!/bin/bash
for d in */; do rm "INSTALLATION INSTRUCTIONS.rtf" "$d"; done
I am very hesitant and wanted to make sue as these files are vitally important and I don't want to accidentally remove anything and making a backup of 450+ GBs is very taxing.
find . -mindepth 2 -name "INSTALLATION INSTRUCTIONS.rtf" -exec rm -f '{}' +
Since this is "vitally important" data, I would first list all files that match the file name you want to delete/overwrite, without taking any action on it (other than listing):
find /folder/ -type f -name "INSTALLATION INSTRUCTIONS.rtf" -print > /tmp/holder
That would create a list of matches on /tmp/holder. Then you could analyze this list before taking any action (either visually or programatically) to make sure that the list does not include anything you don't want to delete (when dealing with big amounts of data, strange things can happen, so be proactive on protecting the data).
If you are happy with what the list shows, then you could delete the old instructions, or if possible, overwrite them with the new file. Here's an example to overwrite the old file with the new one:
while read -r line; do cp --no-preserve=all /folder/newfile "$line"; done < /tmp/holder
The cp --no-preserve=all command (available on GNU bash) would ensure that the new file has permissions that are "adequate" to the folder where they are located. You may change that to a simple cp if you don't want that to happen.
I want to know how I can remove all the files in a directory say directory1 contains some 100 files. I just want to remove the files and not the directory.
I know that rmdir directory1 will remove directory1 completely. But I want to only remove all the files inside.
Try this:
rm /path/to/directory1/*
by adding the -r option you can additionally remove contained directories and their content recursively.
find /path/to/directory1 -type f | xargs rm -f
This recursively deletes all normal files in the directory.
i'm trying to elaborate a command that will find files that haven't been modified in over 6 months and zip them with one command. Afterwards i want to delete all those files and i just archived.
My current command to find the directories with the files is
find /var/www -type d -mtime -400 ! -mtime -180 | xargs ls -l > testd.txt
This gave me all the directories including the files that are older than 6 months
Now i was wondering if there was a way of zipping all the results and deleting them afterwards. Something amongst the line of
find /var/www -type f -mtime -400 ! -mtime -180 | gzip -c archive.gz
If anyone knows the proper syntax to achieve this i'd love to know. Thakns!
Edit, after a few tests this command results in a corrupted file
find /var/www -mtime -900 ! -mtime -180 | xargs tar -cf test4.tar
Any ideas?
Break this into several distinct steps that you can implement and thoroughly test separately:
Build a list of files to be archived and then deleted, saved to a temp file
Use the list from step 1 to add the files to .tar.gz archives. Give the archive file a name following a specific pattern that won't appear in the files to be archived, and put it in a directory outside the hierarchy of files being archived.
Read back the files from the .tar.gz and compare them (or their hashes) to the original files to ENSURE that you got them all without corruption
Use the list from step 1 to delete the files. Do not use a wildcard for deletion. Put in some guard code to prevent deletion of any file matching the name pattern of the archive .tar.gz file(s) created in step 2.
When testing a script that can do irreversible damage, always code the dangerous command with a leading echo and leave it that way until you are sure everything works. Only then remove the echo.
Consider zip, it should meet your requirements.
find ... | zip -m# archive.zip
-m (move) deletes the input directories/files after making the specified zip archive.
-# takes the list of input files from standard input.
You may find more options which are useful to you in the zip manual, e. g.
-r (recurse) travels the directory structure recursively.
-sf (show-files) shows the files that would be operated on, then exits.
-t or --from-date operates on files not modified prior to the specified date.
-tt or --before-date operates on files not modified after or at the specified date.
This could possibly make findexpendable.
zip -mr --from-date 2012-09-05 --before-date 2013-04-13 archive /var/www