Delete lines prior to a specific date in a log file - linux

Question

I have a ffmpeg process log file in which log data get appended to it daily. I want to do a automatic maintainence of this log by going through the log and deleting lines belonging to a certain date.
I tried something but it didn't help.
How should i do it? Please help. Thanks.
Example. To delete all lines prior to Jun 01.
#!/bin/bash
var=`date -d "now-30 day" | awk '{print $3}'`
path="/home/pi/scripts/"
## Removing Entries before 30 Days ####
for logs in $path/process.log; do
awk -v rot=$var '{ if ($NF > rot ) {print $0}}' $logs > $path/temp.log
mv $path/temp.log $logs
done
Example of process.log file which contains data prior to june i want to delete all the previous data.
Oct 28 06:48:15 #####################################Start Time########################################
Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0: Device or resource busy
Oct 28 06:48:17 #####################################End Time########################################
Oct 29 07:38:13 #####################################Start Time########################################
Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0: Device or resource busy
Oct 29 07:38:13 #####################################End Time########################################"
This is the data i want to preserve which is of current month's data:
Jun 02 09:49:08 #####################################Start Time########################################
Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[video4linux2,v4l2 # 0x36e75e0] The driver changed the time per frame from 1/5 to 1/30
[mjpeg # 0x36e8530] EOI missing, emulating
Input #0, video4linux2,v4l2, from '/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0':
Duration: N/A, start: 13731.683520, bitrate: N/A
Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
Stream #0:0 -> #0:0 (mjpeg (native) -> mpeg4 (native))
Press [q] to stop, [?] for help

Answer 1

I'd use sed.
sed -i.bak '/^Jun 01/,/^Jun 01.*$\n/!d' log.txt
This will create a backup of log.txt named log.txt.bak. But still, I'd suggest you to make a backup first with a different name.
This assumes that Jun 01 exists in the log.txt.
EDITED
#!/bin/bash
target_file='log.txt'
target_date='Jun 01'
line=$(awk -v date="$target_date" '
date==$1" "$2 {
if(n++==0) { line=FNR
} else { n=0 }
} END {
print line-1
}' "$target_file")
if [[ "$line" > 1 ]]; then
sed -i.bak "1,${line}d" "$target_file"
fi
Similar as the above command, It will create a backup file named e.g. log.txt.bak. Change the target_file and target_date according to your needs.
This script will delete all lines prior date even though there are multiple duplicates.

Answer 2

We miss elements for coding waterproof. We need to know a.o. if it possible to find entries from past years in the log files, because the log files don't mention the year of the entries, only the month and the day.
We suppose that a date apparently in the future corresponds to entries of former years. If you run the script in the beginning of January, you will delete entries of December which are younger than 30 days.
This implies the problem can't be solved in a completely satisfactory way without more knowledge. Sometimes the name of the logfiles can contain some useful information about the year that we ignore here.
Anyway, the solution below may work, but it is not very optimal:
#!/bin/bash
#time stamp older entries to keep
ts_now=$(date +"%s" -d now)
ts_oldest_to_keep=$(date +"%s" -d "now - 30 days")
#adapt path to point to the directory where log files are living
path="."
for ts in $(grep "Start Time" "$path/"*".log" | awk '{print $1, $2}' | date +"%s" -f - | sort -u)
do
# get the timestamp corresponding to all the entries to be removed
if [ $ts -lt $ts_oldest_to_keep ] || [ $ts -gt $ts_now ]
then
# convert the timestamp into the string "%b %d" i.e. the abbreviated month
# name in English followed by space and zero padded day of month.
# LC_ALL=C to make sure you get an English name of month
ym=$(LC_ALL=C date +"%b %d" -d "#$ts")
# delete the block of lines beginning with "$ym" and ending with
# the same string
sed -i "/^$ym.*Start Time/,/^$ym.*End Time/d" "$path/"*".log"
fi
done
You first grab all the possible dates from the log files. You convert the date into a time stamp for making a numerical comparison of dates possible, and you use sed to delete block of lines without the need of a temporary file.
Looking for a solution in AWK in one pass shoud also be possible, but AWK misses a standard function to convert easily dates as Jun 01.
Anyway, backup your log files before messing around.