Incorrect results from Md5 sum in bash shell script - linux

I'm missing some images that should have been archived when this script runs. I think this may be to do with my indentations or my Md5 sum. I have tried everything I can think of.
here is the code with out the correct indentations:
#!/bin/sh
if [ ! -d "$1" ]; then
echo Directory "$1" cannot be found. Please try again.
exit
fi
if [ $# -eq 1 ]; then
echo "usage: Phar image_path archive_path"
exit
fi
if [ -d "$2" ]; then
echo "archive exists"
else
echo "the directory 'archive' does't exist. Creating directory 'archive'."
mkdir -p ~/archive
fi
find $1 -iname "IMG_[0-9][0-9][0-9][0-9].JPG" | cat > list.txt
[ -f ~/my-documents/md5.txt ] && rm md5.txt || break
while read line;
do md5sum $line | xargs >> md5.txt
done < list.txt
sort -k 1,1 -u md5.txt | cat > uniquemd5.txt
cut -d " " -f 2- uniquemd5.txt > uniquelist.txt
sort uniquelist.txt -r -o uniquelist.txt
for line in $(cat uniquelist.txt)
do
file=$(basename $line) path="$2/file"
if [ ! -f $path ];
then
cp $line $2
else
cp $line $path.JPG
fi
rm uniquelist.txt md5.txt uniquemd5.txt list.txt
done

This loop
while read line;
do md5sum $line | xargs >> md5.txt
done < list.txt
should probably be
while read line;
do md5sum "$line"
done < list.txt > md5.txt
Quote parameter expansions, and it's unclear why you needed.

Related

Find using file for folder locations Linux Bash

I am trying to use a txt file to store folder locations to use in find command. But keep getting folder not found works with only one folder location in file
with "$addfolder"
found=$(find "$addfolder" ! -path "*/.bak/*" -type f -iname "*$ffind*" | sort)
and replacing \"
addfolder="$addfolder $Folder"
folder.txt :-
Main/Public
Main/General
Not Used
Old Backup Files
#!/bin/bash
addfolder=""
filename="Settings/folders.txt"
#Read Folder.txt for locations
while read -r Folder; do
if [ ! "$Folder" == "" ];then
if [ -d "$Folder" ]; then
addfolder="$addfolder \"$Folder\""
echo "$addfolder"
fi
fi
done < "$filename"
if [ "$addfolder" == "" ]; then
exit
fi
echo -e "\e[36mEnter Filename To Find :-\e[0m"
read -p "" ffind
echo -e "\e[92mSearching:\e[0m"
found=$(find $addfolder ! -path "*/.bak/*" -type f -iname "*$ffind*" | sort)
echo -e "\e[33m$found\e[0m"
echo "Press Enter To Exit"
read -s -n 1 -p ""
Regular variables should only hold single strings.
To hold lists of strings, use an array:
#!/bin/bash
addfolder=()
filename="Settings/folders.txt"
#Read Folder.txt for locations
while IFS= read -r Folder; do
if [ ! "$Folder" == "" ];then
if [ -d "$Folder" ]; then
addfolder+=( "$Folder" )
echo "${addfolder[#]}"
fi
fi
done < "$filename"
if [ "${#addfolder[#]}" == 0 ]; then
exit
fi
echo -e "\e[36mEnter Filename To Find :-\e[0m"
read -p "" ffind
echo -e "\e[92mSearching:\e[0m"
found=$(find "${addfolder[#]}" ! -path "*/.bak/*" -type f -iname "*$ffind*" | sort)
echo -e "\e[33m$found\e[0m"
echo "Press Enter To Exit"
read -s -n 1 -p ""

breaking the loop in bash programming

This code is working fine with me I am new to bash programming I want loop to run only once and I dont want loop to run and watermark the all videos. Instead I want to break the loop when first video is done.
#!/bin/bash
#!!!!!!VARIABLES!!!!!!
VIDEOS_PATH='/home/danny/public_html/videowork/'
LOGO_PATH='/home/danny/public_html/watermark_pics/'
DATABASE_INFORMATION='/home/danny/public_html/videowork/db.txt'
#!!!!!!!!!!!!!!!!!!!!!
if find $VIDEOS_PATH -name '*.mp4' 2> /dev/null
then
for file in $(find $VIDEOS_PATH -name '*.mp4')
do
echo "User: ".$(whoami)
echo "File: "$file" has been detected"
sitename=$(echo $file | awk -F $VIDEOS_PATH '{print $2}')
sitename=$(echo $sitename |awk -F '/' '{print $1}')
echo "File sitename is: "$sitename
logo=$LOGO_PATH$sitename.png
echo "Watermark picture has been located in: "$logo
echo "Encoding "$file" to /home/"$sitename"/public_html/yt/"$(basename $file)
ffmpeg -i $file -vcodec libx264 -acodec copy -vf "movie=$logo [watermark]; [in][watermark] overlay=10:10 [out]" /home/$sitename/public_html/yt/$(basename $file)
thumbnail=$(basename $file .mp4)
thumbnail=$VIDEOS_PATH$sitename"/"$(echo $thumbnail).jpg
mv $thumbnail /home/$sitename/public_html/yt/
rm $file
echo "/home/$sitename/public_html/yt/$(basename $file)"
for line in $(cat $DATABASE_INFORMATION)
do
database=$(echo $line | awk -F '|' '{print $1}')
password=$(echo $line | awk -F '|' '{print $2}')
echo "Database detected: "$database
echo "Password: "$password
videoid=$(basename $file .mp4)
if [[ $(echo $database) == $(echo $sitename) ]]
then
php -f /home/danny/public_html/videowork/database_job.php -- $(echo $database) $(echo $password) $(echo $videoid)
#echo "php -f database_job.php -- "$(echo $database)" "$(echo $password)" "$(echo $videoid)
fi
done
done
fi
If I add "break" before last fi? would it be fine? I tried but it didnt work well
Using a looping construct only to break after the first iteration is silly. Instead, don't use that construct at all. For instance, with bash 4.x or newer, you can completely avoid use of find entirely:
#!/bin/bash
shopt -s globstar nullglob
videos=( "$VIDEOS_PATH"/**/*.mp4 )
(( ${#videos[#]} )) || { echo "No videos found" >&2; exit 1; }
file=${videos[0]}
# ...transcode "$file" here
For compatibility with bash 3.x, one might change this a bit to still use find (albeit with -print0 and an appropriate read construct to handle filenames with unusual characters):
#!/bin/bash
IFS= read -r -d '' file < <(find "$VIDEOS_PATH" -name '*.mp4' -print0)
[[ $file ]] || { echo "No videos found" >&2; exit 1; }
# ...transcode "$file" here

Shell script randomly fails to execute file operations commands

I'm having a bit of trouble running this script. Every now and then, it will fail to execute a mv command or rm command referenced below. Then the next iteration will feel the full repercussions of that failure. Why is this happening, and how do I control for it? For reference, the syntax phyml -i $f [parameters] outputs the files $f_phyml_tree.txt and $f_phyml_stats.txt into the same directory as $f. I want to get both these files out of that directory while saving the tree somewhere else.
ber="_phyml_tree.txt"
for f in ~/randseqs/aa/*.txt;
do
echo $f
fpath=`echo $f | cut -d'/' -f 6`
if [ ! -f /home/mcb3421_10/phymlout/aa/$fpath$ber ] || [ ! -f /home/mcb3421_10/phymltimer/aa/$fpath ]; then
phyml -i $f -d aa -b 0 -m Blosum62 > ~/blown.txt
grep "Time used" < ~/blown.txt > ~/phymltimer/aa/$fpath
mv /home/mcb3421_10/randseqs/aa/*$ber phymlout/aa
if [ ! -f /home/mcb3421_10/phymlout/aa/$fpath$ber ]; then
echo $f failed to write, check the logfile /home/mcb3421_10/phymllogs/aa/$fpath
fi
rm ~/randseqs/aa/*_stat*
mv ~/blown.txt ~/phymllogs/aa/$fpath
fi
done
for f in ~/randseqs/nuc/*.txt;
do
echo $f
fpath=`echo $f | cut -d'/' -f 6`
if [ ! -f /home/mcb3421_10/phymlout/nuc/$fpath$ber ] || [ ! -f /home/mcb3421_10/phymltimer/nuc/$fpath ]; then
phyml -i $f -d nt -b 0 -m HKY85 > ~/blown.txt
grep "Time used" < ~/blown.txt > ~/phymltimer/nuc/$fpath
mv /home/mcb3421_10/randseqs/nuc/*$ber phymlout/nuc
if [ ! -f /home/mcb3421_10/phymlout/nuc/$fpath$ber ]; then
echo $f failed to write, check the logfile /home/mcb3421_10/phymllogs/nuc/$fpath
fi
rm ~/randseqs/nuc/*_stat*
mv ~/blown.txt ~/phymllogs/nuc/$fpath
fi
done

Batch file To check if the files are present in a folder comparing the names from a text file

I am trying to write a batch file which checks if the file names present in a text file are present in a folder.
I have a folder called PAN where i have a text file named as PRAS_filelist.txt which stores all the file names. And a folder PRAS in which all my files are present.
My requirement is it should scan the folder and at the end, it should populate a error message in a file PRAS_PP_ERROR_LOG. but now it is coming out if any one file is not present since i have used exit.
Below is the code
`rm -f ../SessLogs/PRAS_PP_ERROR_LOG`
for i in `cat ./PAN/PRAS_filelist.txt`
do
file_name=$i
if [ -e ./PAN/PRAS/"$file_name"_* ]
then
if [ -s ./PAN/PRAS/"$file_name"_* ]
then
a=`sed -n '1p' ./PAN/PRAS/"$file_name"_*|cut -d'|' -f1`
b=`sed -n '$p' ./PAN/PRAS/"$file_name"_*|cut -d'|' -f1`
if [ "$a" == "H" ] && [ "$b" == "T" ]
then
trailer_record=`sed -n '$p' ./PAN/PRAS/"$file_name"_*|cut -d'|' -f2`
record_count=`wc -l ./PAN/PRAS/"$file_name"_*|cut -d' ' -f1`
r_c=`expr $record_count - 1`
if [ $trailer_record -eq $r_c ]
then
`cp ./PAN/PRAS/"$file_name"_* ./PAN/PRAS/STANDARDIZE/"$file_name"`
`sed -i '$d' ./PAN/PRAS/STANDARDIZE/"$file_name"`
else
echo "ERROR in file $file_name: Count of records is $r_c and trailer is $trailer_record">../SessLogs/PRAS_PP_ERROR_LOG
fi
else
echo "ERROR in file $file_name: header or trailer is missing">../SessLogs/PRAS_PP_ERROR_LOG
exit 1
fi
else
echo "ERROR in file $file_name: empty file">../SessLogs/PRAS_PP_ERROR_LOG
exit 1
fi
else
echo "ERROR: $file_name doesnot exist">../SessLogs/PRAS_PP_ERROR_LOG
exit 1
fi
done
refactoring your code a bit:
One odd thing is that you claim to have a list of filenames, but then you append an underscore before you check the file exists. Do the actual filenames actually have underscores?
#!/bin/bash
shopt -s nullglob
logfile=../SessLogs/PRAS_PP_ERROR_LOG
rm -f $logfile
while read file_name; do
files=( ./PAN/PRAS/${file_name}_* )
if (( ${#files[#]} == 0 )); then
echo "ERROR: no files named ${file_name}_*"
continue
elif (( ${#files[#]} > 1 )); then
echo "ERROR: multiple files named ${file_name}_*"
continue
fi
f=${files[0]}
if [[ ! -s $f ]]; then
echo "ERROR in file $f: empty file"
continue
fi
a=$(sed '1q' "$f" | cut -d'|' -f1)
b=$(sed -n '$p' "$f" | cut -d'|' -f1)
if [[ "$a" != "H" || "$b" != "T" ]]; then
echo "ERROR in file $f: header or trailer is missing"
continue
fi
trailer_record=$(sed -n '$p' "$f" | cut -d'|' -f2)
r_c=$(( $(wc -l < "$f") - 1 ))
if (( trailer_record == r_c )); then
sed '$d' "$f" > ./PAN/PRAS/STANDARDIZE/"$file_name"
else
echo "ERROR in file $f: Count of records is $r_c and trailer is $trailer_record"
fi
done < ./PAN/PRAS_filelist.txt | tee $logfile

Renaming files in a folder (linux)

I have a different HTML files in a folder. How to rename the files so that they have the names of:
1.html
2.html
3.html
...
This can make it:
i=1
for file in /your/folder/*
do
mv $file ${i}.html
i=$((i+1)) #((i++)) was giving errors (see comments)
done
It loops through all files in /your/folder and renames them according to the number $i that keeps increasing.
Here is my script
#!/bin/sh
#
# batchrename - renames files like 01.ext, 02.ext ...
#
# format : batchrename <list of files>
# or: -r <extension> <<list of files> or <dir>>
# -r - recoursively
counter=0
extrec=""
if [ "$#" -lt "1" ]; then
echo -e "\n\t\tUsage:\n\tbatchrename [opt]\nopt:"
echo -e "-r <ext> <folder> (or file list) -- renames recoursively ALL files"
echo -e "\tin folder <folder> (or by file list given) with extension .<ext>"
echo -e "<folder> -- renames ALL files in folder given"
echo -e "<file list> -- renames ALL files of given filelist.\n\n"
exit 0
fi
Name="$*"
if [ "$1" = "-r" ]; then
extrec="$2"
shift
shift
Name="$*"
[ "$Name" = "" ] && Name="./"
fi
echo -e "\n\t\t\tRENAMING"
for file in $Name
do
file=`echo "$file" | sed "s/<>/ /g"`
if [ -d "$file" ];then
echo -e "\nDiving into \033[38m $file \033[39m"
cd "$file"
if [ "$extrec" != "" ]; then
batchrename -r $extrec `ls -1 | sed "s/\ /<>/g"`
else
batchrename `ls -1 | sed "s/\ /<>/g"`
fi
cd ../
continue
fi
ext=`ext "$file"`
if [ "$ext" = "ion" ]; then
continue
fi
if [ "$extrec" = "" -o "$ext" = "$extrec" ];then
counter=`expr $counter + 1`
echo -e "Progress: $counter files\r\c"
mv "$file" "rnmd$counter.$ext"
fi
done
echo -e "\n\n\t\t\tENDING"
digits=`echo $counter|awk '{print length ($0)}'`
cnt=1
while [ $digits -gt $cnt ]
do
f=`ls -S -1|grep "rnmd[0-9]\{$cnt\}\."`
rename rnmd rnmd0 $f
cnt=`expr $cnt + 1`
done
if [ "$counter" -gt "0" ]; then
rename rnmd "" rnmd*
fi
echo -e "\n\t\t\tDone !!!\n"
After renaming all your files will looks like 001.file, 002.file, ... and so on. Amount of leading zeros depends on amount of files. So, after renaming ls will show right order of files!
It use intermediate script ext:
#!/bin/sh
#
# ext - returns file suffix (case-unsensitive)
#
File="$*"
if [ -d "$File" ]; then
echo ""
exit 0
fi
EXT=`echo $File|sed 's/.\{1,\}\.//g'`
if [ "$EXT" = "$File" ]; then
EXT=""
fi
echo $EXT| tr '[:upper:]' '[:lower:]'
Here is a similar code, just to add to rename with the same prefix and append an incremental value
declare -i x=1
for f in $(find -type f); do
mv -v $f ${f%/*}/change_me_$x ;
x=$x+1;
done

Resources