I need to run script to parse the value from csv every five minutes but should not parse the all the values all the time it should be from the last point.
Really just putting some flesh on #hek2mgl's suggestion and implementing it for you.
I store the last known length of the logfile in another file called lastlength.
#!/bin/bash
LOGFILE=somelog.csv
LASTLEN=lastlength
# Pre-set seek to start of file...
seek=0
# ... but overwrite if there was a previously seen value
[ -f lastlength ] && seek=$(cat lastlength)
echo DEBUG: Starting from offset $seek
# Update last seen length into file - parameters to "stat" will differ on Linux
stat -f "%Dz" "$LOGFILE" > "$LASTLEN"
# Get last line of file starting from previous position
dd if="$LOGFILE" bs=$seek skip=1 2> /dev/null | tail -1
I am using OSX, so if you are using Linux, the parameters to the stat command in the second to last line will be different, probably
stat -c%s "$LOGFILE" > "$LASTLEN"
I'll leave you to put it into your crontab so it gets called every 5 minutes.
Related
i am writing a bash script to accomplish the following.
script runs wget every five minutes to download a small log from a static url.
script uses diff to see if there are any new entries made to the log file (new entries are made at the end of log file).
if new log entries are found - extract the new entries to a new file, format them properly, send me an alert, return to #1.
if no new log entries are found, go back to #1.
wget "https://url-to-logs.org" -O new_log
if diff -q new_log old_log; then
echo "no new log entries to send."
else
echo "new log entries found, sending alert."
diff -u new_log old_log > new_entries
#some logic i have to take the output of "new_entries", properly format the text and send the alert.
rm -rf old_log new_entries
cp new_log old_log
rm -rf new_log
fi
there is one additional thing - every night at midnight the server hosting the logs deletes all entries and displays a blank file until new log entries are made for the new day.
i guess i could always run a cron job at midnight to run "rm -rf" and "touch" the old_log file, but curious if an easier way to do this exists.
thanks in advance for any/all input and help.
If your logs are not rotating - i.e. the old log is guaranteed to be the prefix of the new log, you can just use tail to get the new suffix - something like this:
tail -n+$(( $(wc -l old_log) + 1 )) new_log > new_entries
If there are no new lines in new_log, the new_entries file will be empty, which you can check using stat or some other way.
If your logs are rotating, you should first use grep to check if the last line from the old log exists in the new log, and if not - assume the entire new log is new:
if ! egrep -q "^$(tail -n1 old_log)\$" new_log; then cat new_log > new_entries; fi
I am learning bash since 6 days I think I got some of the basics.
Anyway, for the wallpapers downloaded from Variety I've written two scripts. One of them moves downloaded photos older than 12 days to a folder and renames them all as "Aday 1,2,3..." and the other lets me select these and moves them to another folder and removes photos I didn't select. 1st script works just as I intended, my question is about the other
I think I should write the script down to better explain my problem
Script:
#!/bin/bash
#Move victors of 'Seçme-Eleme' to 'Kazananlar'
cd /home/eurydice/Bulunur\ Bir\ Şeyler/Dosyamsılar/Seçme-Eleme
echo "Select victors"
read vct
for i in $vct; do
mv -i "Aday $i.png" /home/eurydice/"Bulunur Bir Şeyler"/Dosyamsılar/Kazananlar/"Bahar $RANDOM.png" ;
mv -i "Aday $i.jpg" /home/eurydice/"Bulunur Bir Şeyler"/Dosyamsılar/Kazananlar/"Bahar $RANDOM.jpg" ;
done
#Now let's remove the rest
rm /home/eurydice/Bulunur\ Bir\ Şeyler/Dosyamsılar/Seçme-Eleme/*
In this script I originally intended to define another variable (let's call this "n") and so did I with copying and changing the variable from the first script. It was something like that
for i in $vct; do
n=1
mv "Aday $i.png" /home/eurydice/"Bulunur Bir Şeyler"/Dosyamsılar/Kazananlar/"Bahar $n.png" ;
mv "Aday $i.jpg" /home/eurydice/"Bulunur Bir Şeyler"/Dosyamsılar/Kazananlar/"Bahar $n.jpg" ;
n=$((n+1))
done
When I do that for the first time the script worked just as I intended. However, in my 2nd test run this script overwrote the files that already existed. I mean, for example in 1st run i had 5 files whose names are "Bahar 1,2,3,4,5" and the 2nd time I chose 3 files to add. I wanted their names to be "Bahar 6,7,8" but instead, my script made them the new 1,2 and 3. I tried many solutions and when I couldn't fix that I just assigned random numbers to them.
Is there a way to make this script work as I intended?
This command finds the biggest file name number amongst files in current directory. If no file is found, biggest number is assigned to 0.
biggest_number=$(ls -1 | sed -n 's/^[^0-9]*\([0-9]\+\)\(\.[a-zA-Z]\+\)\?$/\1/p' | sort -r -g | head -n 1)
[[ ! -z "$biggest_number" ]] || biggest_number=0
The regex in sed command assumes that there is no digit in filenames before the trailing number intended for increment.
As soon as you have found the biggest number, you can use it to start your loop to prevent overwrites.
n=$((biggest_number+1))
Can someone fix this for me.
It should copy a version log file to backup after moving to a repo directory
Then it automatically appends line given as input to the log file with some formatting.
That's it.
Assume existence of log file and test directory.
#!/bin/bash
cd ~/Git/test
cp versionlog.MD .versionlog.MD.old
LOGDATE="$(date --utc +%m-%d-%Y)"
read -p "MSG > " VHMSG |
VHENTRY="- **${LOGDATE}** | ${VHMSG}"
cat ${VHENTRY} >> versionlog.MD
shell output
virufac#box:~/Git/test$ ~/.logvh.sh
MSG > testing script
EOF
EOL]
EOL
e
E
CTRL + C to get out of stuck in reading lines of input
virufac#box:~/Git/test$ cat versionlog.MD
directly outputs the markdown
# Version Log
## version 0.0.1 established 01-22-2020
*Working Towards Working Mission 1 Demo in 0.1 *
- **01-22-2020** | discovered faker.Faker and deprecated old namelessgen
EOF
EOL]
EOL
e
E
I finally got it to save the damned input lines to the file instead of just echoing the command I wanted to enter on the screen and not executing it. But... why isn't it adding the lines built from the VHENTRY variable... and why doesn't it stop reading after one line sometimes and this time not. You could see I was trying to do something to tell it to stop reading the input.
After some realizing a thing I had done in the script was by accident... I tried to fix it and saw that the | at the end of the read command was seemingly the only reason the script did any of what it did save to the file in the first place.
I would have done this in python3 if I had know this script wouldn't be the simplest thing I had ever done. Now I just have to know how you do it after all the time spent on it so that I can remember never to think a shell script will save time again.
Use printf to write a string to a file. cat tries to read from a file named in the argument list. And when the argument is - it means to read from standard input until EOF. So your script is hanging because it's waiting for you to type all the input.
Don't put quotes around the path when it starts with ~, as the quotes make it a literal instead of expanding to the home directory.
Get rid of | at the end of the read line. read doesn't write anything to stdout, so there's nothing to pipe to the following command.
There isn't really any need for the VHENTRY variable, you can do that formatting in the printf argument.
#!/bin/bash
cd ~/Git/test
cp versionlog.MD .versionlog.MD.old
LOGDATE="$(date --utc +%m-%d-%Y)"
read -p "MSG > " VHMSG
printf -- '- **%s** | %s\n' "${LOGDATE}" "$VHMSG" >> versionlog.MD
I am trying to redirect the output of a continuously running program to a file say error_log.
The command I am using looks like this, where myprogram.sh runs continuously generating data and writing to /var/log/httpd/error_log
myprogram.sh >> /var/log/httpd/error_log
Now I am using logrotate to rotate this file per hour. I am using create command in logrotate so that it renames the original file and creates a new one.
The logrotate config looks like
/var/log/httpd/error_log {
# copytruncate
create
rotate 4
dateext
missingok
ifempty
.
.
.
}
But here redirection fails. What I want is myprogram.sh to write data to error_log file irrespective of it being rotated by logrotate, obviously to newly generated error_log file
Any idea how to make redirection work based on the file name and not the descriptor ?
OR
Any other way of doing it in bash ?
If I understood your problem, one solution (without modify myprogram.sh) could be:
$ myprogram.sh | while true; do head -10 >> /var/log/httpd/error_log; done
Explaining:
myprogram.sh writes to stdout
We redirect this output to while bash sentence through a pipe |.
while true is an infinite loop that will never end, nor even when myprogram.sh ends which should break the pipe.
In each loop head command is called to append the first 10 lines read from the pipe to the end of current /var/log/httpd/error_log file (that may be different from the last loop because of logrotate).
(You can change the number of lines being written in each loop)
And another way is:
$ myprogram.sh | while read line; do echo "$line" >> /var/log/httpd/error_log; done
That's very similar to the first one, but this ends the loop when myprogram.sh ends or closes it's stdout.
It works line by line instead in groups of lines.
Assuming a shell script(commands.sh) with few commands.
I need to write a script which sends the output of commands executed by commands.sh to a file f1.csv
if file size exceeds 1MB then the output flowing should go to file f2.csv
if the file size exceeds 1 mb again here,the output flowing should go to file f3.csv
if f3.csv exceeds the size 1mb,then the older f1 should be deleted and again new file f1 should be created,
output flowing should be to written to f1. This process should go on .
I can write the crontab file, just the shell script is a bit tricky
I have been experimenting:
#!/usr/bin/env bash
PREFIX="f"
# Maximum size after which you want a new file in bytes
MAX_SIZE=1048576
LAST_FILE=`ls "$prefix"*.csv | tail -1`
# Check if file exists and if it does not, create it.
if [[ -z "$LAST_FILE" ]]
then
LAST_FILE=$PREFIX"1.csv"
touch $LAST_FILE
fi
LAST_FILE_NO=`echo $LAST_FILE | sed s/$PREFIX/''/ | sed s/.csv/''/`
LAST_FILE_SIZE=`stat -c %s $LAST_FILE`
if [ `stat -c %s $LAST_FILE` -lt 200 ]
then
`/bin/sh ./sam.sh >> $LAST_FILE`
else
UPCOMING_FILE_NO=$((LAST_FILE_NO+1))
`/bin/sh ./sam.sh >> $PREFIX$UPCOMING_FILE_NO.csv`
fi
help is appreciated guys.
EDIT: Have got the secondary shell script to work too...
Now if anyone could help me with resetting after 3 files are done and starting from f1.
thanks
It sounds like you'd be better off using logrotate, depending on how your script is running. If you are running 'commands.sh' on a cron, you can have logrotate rotate out the logs. There is a good guide on logrotate here:
http://linuxers.org/howto/howto-use-logrotate-manage-log-files
If your commands.sh isn't going to be on a cron, meaning it's not a regular time interval that triggers it, you could manually set up a log rotation at the beginning of your script. I once had to do something similar. I found this guide really useful:
http://wazem.blogspot.com/2013/11/simple-bash-log-rotate-function.html