Linux bash script: Recurisely delete files if empty based on file format from user input [duplicate] - linux

This question already has answers here:
How do I pass a wildcard parameter to a bash file
(2 answers)
Closed 5 years ago.
So as the title describes, I want to recursively delete all files which match a naming pattern given by the user, but only if the file is empty. Here is my attempt:
#!/bin/bash
_files="$1"
[ $# -eq 0 ] && { echo "Usage: $0 filename"; exit 1; }
[ ! -f "$_files" ] && { echo "Error: $0 no files were found which match given naming structure."; exit 2; }
for f in $(find -name $_files)
do
if [ -s "$f" ]
then
echo "$f has some data."
# do something as file has data
else
echo "$f is empty. Deleting file."
rm $f
fi
done
Example output:
./remove_blank.sh *.o*
./Disp_variations_higher_0.o1906168 has some data.
./remove_blank.sh *.e*
./Disp_variations_higher_15.e1906183 is empty. Deleting file.
As you can see, the code works, but only for one file at a time. Should be a relatively simple fix to get it to work, but I extremely new to bash scripting and can't seem to figure it out. Sorry for the noobish question. I did some research to find an answer but didn't find exactly what I needed. Thanks in advance for any help.
Edit
I have found two different solutions to the problem. As #David Z's suggestion, one can fix this by 1st deleting the Error checking part of the script as well as putting quotes around the $_files variable in the find function. Then the code looks like this:
#!/bin/bash
_files=$1
[ $# -eq 0 ] && { echo "Usage: $0 filename"; exit 1; }
for f in $(find -name "$_files")
do
if [ -s $f ]
then
echo "$f has some data."
# do something as file has data
else
echo "$f is empty. Deleting file."
rm $f
fi
done
Or, one can also simply change the for loop to for f in "$#", which allows the error check to be kept in the script. I am not sure which method is better but will update again if I find out.

It looks like the way you're invoking the script, the shell expands the pattern before running your script. For example, in
./remove_blank.sh *.o*
the shell converts *.o* to the list of filenames that match that pattern, and then what it actually runs is something like
./remove_blank.sh Disp_variations_higher_0.o1906168 other_file.o12345 ...
But you only check the first argument in your script, so that's the one that winds up getting deleted.
Solution: quote the pattern when you run the script.
./remove_blank.sh '*.o*'
You will also need to remove the test [ ! -f "$_files" ] ... because $_files is being set to the pattern (such as *.o*), not a filename. In fact, you might want to rename the variable to make that clear. And finally, you need to quote the variable in your find command,
... $(find -name "$_files") ...
so that the pattern makes it all the way through to find without being converted into filenames by the shell.
There are some other issues with the script but you might want to ask about those on Code Review. What I've identified here is just the minimum needed to get it working.
Incidentally, you can accomplish this whole task using find itself, which will at least give cleaner code:
find -name "$_files" -type f -empty -delete

Related

Shell Script working with multiple files [duplicate]

This question already has answers here:
How to iterate over arguments in a Bash script
(9 answers)
Closed 5 years ago.
I have this code below:
#!/bin/bash
filename=$1
file_extension=$( echo $1 | cut -d. -f2 )
directory=${filename%.*}
if [[ -z $filename ]]; then
echo "You forgot to include the file name, like this:"
echo "./convert-pdf.sh my_document.pdf"
else
if [[ $file_extension = 'pdf' ]]; then
[[ ! -d $directory ]] && mkdir $directory
convert $filename -density 300 $directory/page_%04d.jpg
else
echo "ERROR! You must use ONLY PDF files!"
fi
fi
And it is working perfectly well!
I would like to create a script which I can do something like this: ./script.sh *.pdf
How can I do it? Using asterisk.
Thank you for your time!
Firstly realize that the shell will expand *.pdf to a list of arguments. This means that your shell script will never ever see the *. Instead it will get a list of arguments.
You can use a construction like the following:
#!/bin/bash
function convert() {
local filename=$1
# do your thing here
}
if (( $# < 1 )); then
# give your error message about missing arguments
fi
while (( $# > 0 )); do
convert "$1"
shift
done
What this does is first wrap your functionality in a function called convert. Then for the main code it first checks the number of arguments passed to the script, if this is less than 1 (i.e. none) you give the error that a filename should be passed. Then you go into a while loop which is executed as long as there are arguments remaining. The first argument you pass to the convert function which does what your script already does. Then the shift operation is performed, what this does is it throws away the first argument and then shifts all the remaining arguments "left" by one place, that is what was $2 now is $1, what was $3 now is $2, etc. By doing this in the while loop until the argument list is empty you go through all the arguments.
By the way, your initial assignments have a few issues:
you can't assume that the filename has an extension, your code could match a dot in some directory path instead.
your directory assignment seems to be splitting on . instead of /
your directory assignment will contain the filename if no absolute or relative path was given, i.e. only a bare filename
...
I think you should spend a bit more time on robustness
Wrap your code in a loop. That is, instead of:
filename=$1
: code goes here
use:
for filename in "$#"; do
: put your code here
done

Iterating with ls and checking with -f doesn't work

I have a piece of code that should work, but it doesn't.
I want to iterate through the files and subdirectories of directories given in the command line and see which one of them is a file. The program never entries in the if statement.
for i in $#;do
for j in `ls $i`;do
if [ -f $j ];then
echo $j is a file!
fi
done
done
Things can go wrong with your approach. Do it this way.
for i in "$#" ; do
for j in "$i"/* ; do
if [ -f "$j" ]; then
echo "$j is a regular file!"
fi
done
done
Changes :
Quoted the "$#" to avoid problems with file paths containing spaces, newlines.
Used shell globbing in the inner loop, as parsing ls output is not a good idea (see http://mywiki.wooledge.org/ParsingLs)
Double-quoted variable expansion inside the test, once again to allow for files with spaces, newlines.
Added "regular" in the output, because this is what this specific test operator tests for (e.g. will exclude files that correspond to devices, FIFOs, not just directories).
You could simplify a bit if you are so inclined :
for i in "$#" ; do
for j in "$i"/* ; do
! [ -f "$j" ] || echo "$j is a regular file!"
done
done
If you want to use find, you need to make sure you only list files at a depth of one level (or else the results could be different from your code). You can do it this way :
find "$#" -mindepth 1 -maxdepth 1 -type f -exec echo "{} is a file" \;
Please note that this will still be a bit different, as globbing (by default) excludes files that start with a period. Adding shopt -s dotglob to the loop-based solution would allow globbing to consider all files, which should then make both solutions operate on the same files.
I think you'll be better off using find:
find $# -type f

Show where directory exists

I have a bunch of code thats relatively new i.e. lots of bugs hiding and I have code as such:
if [ -d $DATA_ROOT/$name ], I've done research and understand that this means if directory exists but now I'm trying to print out those directories that exist to fix a problem.
Tried using
echo `First: $DATA_ROOT`
echo `Second: $name`
echo `Last: $DATA_ROOT/$name`
exit 1;
Got command not found for all, the code is meant to fix the bug I'm trying to by extracting all files but does not end up extracting all ending up with the data extraction failed error below, code:
num_files=`find $DATA_ROOT/$name -name '*' | wc -l`
if [ ! $num_files -eq $extract_file ] ; then
echo "Data extraction failed! Extracted $num_files instead of $extract_file"
exit 1;
I just want to extract all files correctly, how to do this please?
The back-ping you are using means "execute this as an command"
echo `First: $DATA_ROOT`
echo `Second: $name`
echo `Last: $DATA_ROOT/$name`
would try to execute a command called "First:" which does not exists.
Instead use double quotes as they allow for variable substitution, like this and does not try to execute it as a command
echo "First: $DATA_ROOT"
echo "Second: $name"
echo "Last: $DATA_ROOT/$name"
Also
find $DATA_ROOT/$name -name '*'
is probably not what you want, the -name '*' is the default so you don't need it. As others points out, find will return everything, including directories and special files if you have any of those. find "$DATA_ROOT/$name" -type f is what you want if you only want to list the files or find "$DATA_ROOT/$name" -type d if you only want to list directories. Also always use double quotes around your "$DATA_ROOT/$name" as it allows you to handle file names with spaces -- if you have a $name that contains a space you will fail otherwise.
find reports not only ordinary files, but also directories (including .).
Use find "$DATA_ROOT/$name" -type f.
You are using backticks and hence anything under backticks is treated as a command to execute and as a result you are getting command not found exception. You could use double quotes to avoid the error like below:
echo "First: $DATA_ROOT"
echo "Second: $name"
echo "Last: $DATA_ROOT/$name"
You could use find command to list down all directories like:
find $DATA_ROOT/$name -type d
Above command would list all the directories (with -type d option and use -type f to list all the files) within $DATA_ROOT/$name and then you can perform operations on those directories.

Recursion in a linux file search - Bash script

I need to make a linux file search which involves recursion for a project. I got a bit of help making this so I don't understand this code fully only parts of it. Could someone explain what it means and also give a bit of help as to how I would go about getting a user to input a keyword and for this function to search for that keyword in the directories? Thankyou
#!/bin/bash
lookIn() {
echo $2
for d in $(find $1 -type d)
do
if [ "$d" != "$1" ]
echo "looking in $d"
lookIn $d
fi
done
}
lookIn
You only need find. find will traverse the entire directory. Assuming $1 points to the folder you want to search:
read -p "Enter file name to find: " KEYWORD
find $1 -type f -name "$KEYWORD"
If you want to find names that contain the keyword, then use:
find $1 -type f -name "*${KEYWORD}*"
Try this then you can work this into your bigger script (whatever it does).
TL;DR
Don't use recursion. It may work, but it's more work than necessary; Bash doesn't have tail-call optimization, and it's not a functional programming language. Just use find with the right set of arguments.
Parameterized Bash Function to Call Find
find_name() {
starting_path="$1"
filename="$2"
find "$1" -name "$2" 2>&-
}
Example Output
Make sure you quote properly, especially if using globbing characters like * or ?. For example:
$ find_name /etc 'pass?d'
/etc/passwd
/etc/pam.d/passwd
You don't really need find for recursive file search. grep -r (recursive) will work fine.
See below script:
#!/bin/bash
# change dir to base dir where files are stored for search
cd /base/search/dir
# accept input from user
read -p "Enter Search Keyword: " kw
# perform case insensitive recursive search and list matched file
grep -irl "$kw" *

Bash command to move only some files?

Let's say I have the following files in my current directory:
1.jpg
1original.jpg
2.jpg
2original.jpg
3.jpg
4.jpg
Is there a terminal/bash/linux command that can do something like
if the file [an integer]original.jpg exists,
then move [an integer].jpg and [an integer]original.jpg to another directory.
Executing such a command will cause 1.jpg, 1original.jpg, 2.jpg and 2original.jpg to be in their own directory.
NOTE
This doesn't have to be one command. I can be a combination of simple commands. Maybe something like copy original files to a new directory. Then do some regular expression filter on files in the newdir to get a list of file names from old directory that still need to be copied over etc..
Turning on extended glob support will allow you to write a regular-expression-like pattern. This can handle files with multi-digit integers, such as '87.jpg' and '87original.jpg'. Bash parameter expansion can then be used to strip "original" from the name of a found file to allow you to move the two related files together.
shopt -s extglob
for f in +([[:digit:]])original.jpg; do
mv $f ${f/original/} otherDirectory
done
In an extended pattern, +( x ) matches one or more of the things inside the parentheses, analogous to the regular expression x+. Here, x is any digit. Therefore, we match all files in the current directory whose name consists of 1 or more digits followed by "original.jpg".
${f/original/} is an example of bash's pattern substitution. It removes the first occurrence of the string "original" from the value of f. So if f is the string "1original.jpg", then ${f/original/} is the string "1.jpg".
well, not directly, but it's an oneliner (edit: not anymore):
for i in [0-9].jpg; do
orig=${i%.*}original.jpg
[ -f $orig ] && mv $i $orig another_dir/
done
edit: probably I should point out my solution:
for i in [0-9].jpg: execute the loop body for each jpg file with one number as filename. store whole filename in $i
orig={i%.*}original.jpg: save in $orig the possible filename for the "original file"
[ -f $orig ]: check via test(1) (the [ ... ] stuff) if the original file for $i exists. if yes, move both files to another_dir. this is done via &&: the part after it will be only executed if the test was successful.
This should work for any strictly numeric prefix, i.e. 234.jpg
for f in *original.jpg; do
pre=${f%original.jpg}
if [[ -e "$pre.jpg" && "$pre" -eq "$pre" ]] 2>/dev/null; then
mv "$f" "$pre.jpg" targetDir
fi
done
"$pre" -eq "$pre" gives an error if not integer
EDIT:
this fails if there exist original.jpg and .jpg both.
$pre is then nullstring and "$pre" -eq "$pre" is true.
The following would work and is easy to understand (replace out with the output directory, and {1..9} with the actual range of your numbers.
for x in {1..9}
do
if [ -e ${x}original.jpg ]
then
mv $x.jpg out
mv ${x}original.jpg out
fi
done
You can obviously also enter it as a single line.
You can use Regex statements to find "matches" in the files names that you are looking through. Then perform your actions on the "matches" you find.
integer=0; while [ $integer -le 9 ] ; do if [ -e ${integer}original.jpg ] ; then mv -vi ${integer}.jpg ${integer}original.jpg lol/ ; fi ; integer=$[ $integer + 1 ] ; done
Note that here, "lol" is the destination directory. You can change it to anything you like. Also, you can change the 9 in while [ $integer -le 9 ] to check integers larger than 9. Right now it starts at 0* and stops after checking 9*.
Edit: If you want to, you can replace the semicolons in my code with carriage returns and it may be easier to read. Also, you can paste the whole block into the terminal this way, even if that might not immediately be obvious.

Resources