I have some files stored on a server. I have to get all those files using a pattern and exclude the file which contains the current date as the file name.
Files are given below
/var/tomcat/logs/catalina.2022-05-11.log
/var/tomcat/logs/catalina.2022-05-13.log
/var/tomcat/logs/catalina.2022-05-14.log
/var/tomcat/logs/catalina.2022-05-16.log
/var/tomcat/logs/error_1.log
/var/tomcat/logs/error_2.log
/var/tomcat/logs/error_3.log
/var/tomcat/logs/error_4.log
For this, I have stored patterns in a file and I want to read the pattern of that file and find all files with the help of those patterns.
Pattern Input File content is given below, in below I have used '%Y-%m-%d' to identify the date format so that I can exclude the current date file.
/var/tomcat/logs/catalina.*.log;%Y-%m-%d
/var/tomcat/logs/error_*.log
I have developed a shell script which is given below
#!/bin/sh
pattern_filepath=$1
while IFS= read -r line || [ -n "$line" ]; do
pattern_var="$line"
echo pattern: "$pattern_var"
filepath=""
date_format=""
if [[ $pattern_var == *";"* ]];
then
echo "Semicolons ; separator is there"
filepath=($(echo "$pattern_var" | cut -f1 -d ';'))
echo filepath: "$filepath"
date_format=($(echo "$pattern_var" | cut -f2 -d ';'))
else
echo "Semicolons ; separator is not there"
filepath=$pattern_var
fi
echo "date_format: "$date_format
done < "$pattern_filepath"
Command to run the script
sh /var/monitoring/test.sh "/var/monitoring/pattern" > /var/monitoring/test.log
Inside the log file, I can see, that in the file path variable I am getting the value as a date but that should be with an asterisk instead of a date.
Log file 'test.log'
pattern: /var/tomcat/logs/catalina.*.log;%Y-%m-%d
Semicolons ; separator is there
filepath: /var/tomcat/logs/catalina.2022-05-11.log
date_format: %Y-%m-%d
pattern: /var/tomcat/logs/error_*.log
Semicolons ; separator is not there
date_format:
Please help me with this, how can I achieve this?
Simple, straight to the point. Achieves the requirement, without the complexity of patterns, or lists, or ...
#!/bin/bash
sourcedir="/var/tomcat/logs"
if [[ ! -d "$sourcedir" ]]
then
echo "ERROR: source directory $sourcedir does not exist."
exit 1
fi
targetdir="/tmp/somedir"
if [[ ! -d "$targetdir" ]]
then
echo "ERROR: target directory $targetdir does not exist."
exit 1
fi
# Get the catalina logs
cp "$sourcedir"/catalina.*.log "$targetdir"
# Remove the catalina log for today
filetoremove="$targetdir"/catalina.$(date +%Y-%m-%d).log
if [[ -f "$filetoremove" ]]
then
rm -f "$filetoremove"
fi
# Get the error logs
cp "$sourcedir"/error_*.log "$targetdir"
You can add error checking and recovery for the cp and rm commands.
Related
when using sed -e to update some parameters of a config file and pipe it to | tee (to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).
In Summary, this code is used for updating parameters:
# based on the provided linenumber, add some comments, add the new value, delete old line
sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile
I set up an script which calls this update command 100 times.
In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
behaviour occurs up to 50 times
In a Ubuntu VM (Parallels Desktop) on the
Ubuntu partition this behaviour occurs up to 40 times
On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times
Can someone explain why this is happening?
Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)
#!/bin/bash
# main function at bottom
#====================
#===HELPER METHOD====
#====================
# This method updates parameters with a new value. The replacement is performed linewise.
doUpdateParameterInFile()
{
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"
# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""
# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )
# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null
# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31
fi
}
#===============================
#=== Actual Update Function ====
#===============================
main_script()
{
echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi
echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi
echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi
}
#=================
#=== Main Loop ===
#=================
#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup
# Start the experiment and let it run 100 times
cnt=0
failSum=0
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
main_script
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
cnt=$((cnt+1))
sleep 0.5
done
regards
DonPromillo
The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.
If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like
tmpname=$(mktemp)
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"
or if you want to preserve the original permissions you could do
sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"
or use your tee approach like
sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"
I am looking for a script that recursively lists all files using export and read link and by not using ls options. I have tried the following code, but it does not fulfill the purpose. Please can you help.
My Code-
#!/bin/bash
for i in `find . -print|cut -d"/" -f2`
do
if [ -d $i ]
then
echo "Hello"
else
cd $i
echo *
fi
done
Here's a simple recursive function which does a directory listing:
list_dir() {
local i # do not use a global variable in our for loop
# ...note that 'local' is not POSIX sh, but even ash
# and dash support it.
[[ -n $1 ]] || set -- . # if no parameter is passed, default to '.'
for i in "$1"/*; do # look at directory contents
if [ -d "$i" ]; then # if our content is a directory...
list_dir "$i" # ...then recurse.
else # if our content is not a directory...
echo "Found a file: $i" # ...then list it.
fi
done
}
Alternately, if by "recurse", you just mean that you want the listing to be recursive, and can accept your code not doing any recursion itself:
#!/bin/bash
# ^-- we use non-POSIX features here, so shebang must not be #!/bin/sh
while IFS='' read -r -d '' filename; do
if [ -f "$filename" ]; then
echo "Found a file: $filename"
fi
done < <(find . -print0)
Doing this safely calls for using -print0, so that names are separated by NULs (the only character which cannot exist in a filename; newlines within names are valid.
I am pretty new to Unix and have little exposure to shell script. I need to come up with a script that converts the file names from certain string values to special characters. This needs to be run in such a way all files under sub-directories also gets renamed.
For Example:
From: abc(GE)xyz(PR).txt changes
To: abc>xyz%.txt
I m ok to set if condition for all required special characters, but im not sure what options to pass and how to do it for all sub-directories.
Thanks,
Jeel
Here's one approach:
# given a filename, execute any desired replacements.
update_name() {
local orig_name_var=$1
local dest_name_var=$2
local orig_name=${!orig_name_var}
local new_name="$orig_name"
new_name=${new_name//(GE)/">"}
new_name=${new_name//(PR)/"%"} # repeat for additional substitutions
printf -v "$dest_name_var" "$new_name"
}
while IFS= read -r -d '' orig_name; do
update_name orig_name new_name
[[ $orig_name = $new_name ]] && continue
if ! [[ -e $orig_name ]]; then
orig_dirname=${orig_name%/*}
orig_basename=${orig_name##*/}
update_name orig_dirname new_dirname
if [[ -e $new_dirname/$orig_basename ]]; then
# we already renamed the directory this file is in
orig_name=$new_dirname/$orig_basename
fi
fi
mv -- "$orig_name" "$new_name"
done < <(find . '(' -name '*(GE)*' -o -name '*(PR)*' ')' -print0)
I am trying to make a simple backup script and i have problem creating a folder with the curent date for name
My script is that and basically the problem is on the last line
drivers=$(ls /media/)
declare -i c=0
for word in $drivers
do
echo "($c)$word"
c=c+1
done
read -n 1 drive
echo
c=0
for word in $drivers
do
if [ $c -eq $drive ]
then
backuppath="/media/$word/backup"
fi
c=c+1
done
echo "doing back up to $backuppath"
cp -r /home/stefanos/Programming $backuppath/$(date +%Y-%m-%d-%T)
Ouput:
(0)0362-BA96
(1)Data
(2)Windows
0
doing back up to /media/0362-BA96/backup
cp: cannot create directory `/media/0362-BA96/backup/2012-12-05-21:58:37': Invalid argument
The path is triply checked that is existing until /media/0362-BA96/
SOLVED:
Did what janisz said the final script looks like
drivers=$(ls /media/)
declare -i c=0
for word in $drivers
do
echo "($c)$word"
c=c+1
done
read -n 1 drive
echo
c=0
for word in $drivers
do
if [ $c -eq $drive ]
then
backuppath="/media/$word/backup"
fi
c=c+1
done
echo "doing back up to $backuppath"
backup(){
time_stamp=$(date +%Y_%m_%d_%H_%M_%S)
mkdir -p "${backuppath}/${time_stamp}$1"
cp -r "${1}" "${backuppath}/${time_stamp}$1"
echo "backup complete in $1"
}
#####################The paths to backup####################
backup "/home/stefanos/Programming"
backup "/home/stefanos/Android/Projects"
backup "/home/stefanos/Dropbox"
Trying changing it to:
time_stamp=$(date +%Y-%m-%d-%T)
mkdir -p "${backuppath}/${time_stamp}"
cp -r /home/stefanos/Programming "${backuppath}/${time_stamp}"
: is not valid on FAT (it is used to specify disk). Some of M$ invalid character works on GNU/Linux systems but it is safer to avoid them (just replace with .). Use following date format
date +%Y_%m_%d_%H_%M_%S
It should works on most file systems but it could be too long for MS DOS FAT. More info you will find here.
I would like to write a shell script in which I'll take a line input from user,which will contain some xxx.cpp as filename.
I want to get that "xxx" in another variable.
e.g.
if user give input as:
some params path/to/file/xyz.cpp more p/ara/ms
I want to get xyz which occurs before".cpp" and after last occurance of "/" before ".cpp"
Use basename [param] [.ext].
echo `basename $1 .cpp`
where 1 is the index of path/to/file.xyz in the argument list.
Here's a sample bash script to prompt the user for a list of files and filter all the input and display only the base name of files that end in '.cpp' (with .cpp removed) and which are readable
#!/bin/bash
echo Enter list of files:
read list_of_files
for file in $(echo $list_of_files); do
if echo $file | grep '\.cpp$' > /dev/null; then
if [[ -r $file ]]; then
fn=$(basename ${file})
fn=${fn%.*}
echo $fn
fi
fi
done