How to exclude ".gz" files in logrotate - linux

I am using logrotate version 3.12.3. How do I tell logrotate to exclude files that are already rotated/compressed?
For example, if I am rotating all files in /var/log as
"/var/log/*" {
compress
missingok
dateext
rotate 4
size=5000k
}
after a file is compressed, how do I tell logrotate to not rotate already rotated/compressed files? For example:
/var/log/file1
after logrotation it becomes
/var/log/file1.20211212.gz
I tried
tabooext + .gz
on top of definitions but it doesn't seem to take effect.
From man logrotate
include file_or_directory
Reads the file given as an argument as if it was included inline where the include directive appears. If a directory is given,
most of the files in that
directory are read in alphabetic order before processing of the including file continues. The only files which are ignored are
files which are not regular
files (such as directories and named pipes) and files whose names end with one of the taboo extensions, as specified by the
tabooext directive.
If something like below worked, that would have been good.
/var/log/*[!".gz"] {
}
Thank you.
EDIT
Maybe do something like
/var/log/*[!.][!g][!z] {
..
}
but it skips file named /var/log/test1gz. How do I match the . character with globbing?

Try to find all glob possibilities that does not accept *.gz. As you have three letters (., g, z), there are 2^3 = 8 possible combinations (3 characters, 2 states x and !x), minus one which is *.gz.
The seven other possibilities are the following:
All files that don't have an extension point as third last but end with gz (e.g. filegz)
/var/log/*[!.]gz
All files which 2-letters extension doesn't begin by g but ends by z (e.g. file.7z)
/var/log/*.[!g]z
All files which 2-letters extension begins by g but doesn't end by z (e.g. file.gg)
/var/log/*.g[!z]
All files that don't end with .gz but end with a z (e.g. file.ezz)
/var/log/*[!.][!g]z
All files that don't end with .gz but with g as second to last letter (e.g. file.cgi)
/var/log/*[!.]g[!z]
All files which 2-letters extension is not gz (e.g. file.hh)
/var/log/*.[!g][!z]
Finally, all files that don't finish by .gz (e.g. file.txt)
/var/log/*[!.][!g][!z]
So this gives us:
/var/log/*[!.]gz
/var/log/*.[!g]z
/var/log/*.g[!z]
/var/log/*[!.][!g]z
/var/log/*[!.]g[!z]
/var/log/*.[!g][!z]
/var/log/*[!.][!g][!z]
{
...
The combinations can be generated with python:
import itertools as it
suffix = '.gz'
states = ([x, '![%c]' % x] for x in suffix)
for res in it.product(*states):
print ''.join(res)
Hope it helps!

You don't need to specify a complete folder. You can define a file aswell. So first you should make sub folders for every service that you have a better overview. There you can place your logs into. You shouldn't put that on your complete log folder because there are some more rotate scripts located.
for example:
/var/log/yum.log {
missingok
notifempty
size 30k
yearly
create 0600 root root
}

Related

How do I write a python script to read through files in a Linux directory and perform certain actions?

I need to write a python script to read through files in a directory, retrieve the header record (which contains date)? I need to compare the date in the header record of each file with current date and if the difference is greater than 30 days. I need to delete such files.
I managed to come up with below code but not sure how to proceed since I am new to Python.
Example:
Sample file in the directory (/tmp/ah): abcdedfgh1234.123456
Header record : FILE-edidc40: 20200602-123539 46082 /tmp/ah/srcfile
I have below code for the list of files in the current directory. I need to pass the python equivalent of below actions on unix files
head -1 file|cut -c 15-22
Output: 20200206 (to compare with current date and if older than 30) then delete file(using rm command).
import os
def files in os.listdir(path):
for files in os.listdir(path):
if os.path.isfile(os.path.join(path,file)):
yield file
for file in files(".") : # prints the list of files

Log-rotate delete original file after compressing

I have following files in my someDir:
blacklistadm.out00009 blacklistadm.out00008 blacklistadm.out00007 blacklistadm.out00010 blacklistadm.out00025
I have following log rotation pattern in /etc/logrotate.d/:
someDir/blacklistadm.out*[0-9]{
weekly
missingok
compress
sharedscripts
postrotate
rm -f someDir/blacklistadm.out*[0-9]
endscript
}
When the log rotation script is run, it is somehow deleting all the files in someDir. What I want is to .gz all the files and after compressing delete the original file. I don't want to delete the .gz files.
The files are being deleted because your globbing is being used incorrectly.
blacklistadm.out*[0-9]
literally expands to any file starting with "blacklistadm.out" followed by any sequence of 0 or more characters, ending with a single character within the defined range of 0-9.
This is globbing on to everything obviously, because all your files start with "blacklistadm.out" and end in a number, so when you run your postrotate script with an identical glob you are matching everything in that directory and deleting it.

Is there a way to undo a batch-rename of file extensions?

Ok so I kinda dropped the ball. I was trying to understand how things work. I had a few html files on my computer that I was trying to rename as txt files. This was strictly a learning exercise. Following the instructions I found here using this code:
for file in *.html
do
mv "$file" "${file%.html}.txt"
done
produced this error:
mv: rename *.html to *.txt: No such file or directory
Long story short I ended up going rogue and renaming the html files, as well as a lot of other non html files as txt files. So now I have files labeled like
my_movie.mp4.txt
my_song.mp3.txt
my_file.txt.txt
This may be a really dumb question but.. Is there a way to check if a file has two extensions and if yes remove the last one? Or any other way to undo this mess?
EDIT
Doing this find . -name "*.*.txt" -exec echo {} \; | cat -b seems to tell me what was changed and where it is located. The cat -b part is not necessary but I like it. This still doesn't fix what I broke though.
I'm not sure if terminal can check for extensions "twice", but you can check for . in every name an if there's more than one occurence of ., then your file has more extensions. Then you can cut the extension off with finding first occurence of . in a string when going backwards... or last one if checking characters in string in a normal way.
I have a faster option for you if you can use python. You can strip the extension with:
for file in list_of_files:
os.rename(file,os.path.splitext(file)[0])
which can give you from your file.txt.txt your file.txt
Example:
You wrote that your command tells you what has changed, so just take those changed files and dump them into a file(path to file per line). Then you can easily run this:
with open('<path to list>') as f:
list_of_files = f.readlines()
for file in list_of_files:
os.rename(file.strip('\n'), os.path.splitext(file.strip('\n'))[0])
If not, then you'd need to get the list from python:
import os
results = []
for root, folder, filenames in os.walk(<your path to folder>):
for filename in filenames:
if filename.endswith('.txt.txt'):
results.append(os.path.join(root, filename))
With this you got a list of files ending with .txt.txt like this <your folder>\\<path_to_file>.
Get a path to your directory used in os.walk() without folder's name(it's already in list) so it'll be like this:
e.g. os.walk('/home/me/directory') -> path='/home/me/' and res is item already in a list, which looks like directory/...
for res in results:
path = '' # set the path here
file = os.path.join(path,r)
os.rename(file, os.path.splitext(file)[0])
Depending on what files you want to find change .txt.txt in if filename.endswith('...') to whatever you like and os.rename() will take file's name without extension which in your case means it strips the additional extension you don't want to have.

Create a text file containing list of (relative paths to) files in a directory?

Suppose I am standing on a directory. Inside there's another directory called inside_dir containing a huge number of files. I want to create a file containing a list of all the files inside inside_dir, listed as the relative path to the files. That is, if there is a file called file1 inside inside_dir, the corresponding line in the list file should be inside_dir/file1.
Since the number of files is huge, just doing ls inside_dir/* > list.txt won't work because it will complain about having too many arguments.
find inside_dir -type f > list.txt

Replacing files in one folder and all its subdirectories with modified versions in another folder

I have two folders, one called 'modified' and one called 'original'.
'modified' has no subdirectories and contains 4000 wav files each with unique names.
The 4000 files are copies of files from 'original' except this folder has many subdirectories inside which the original wav files are located.
I want to, for all the wav files in 'modified', replace their name-counterpart in 'original' wherever they may be.
For example, if one file is called 'sound1.wav' in modified, then I want to find 'sound1.wav' in some subdirectory of 'original' and replace the original there with the modified version.
I run Windows 8 so command prompt or cygwin would be best to work in.
As requested, I've written the python code that does the above. I use the 'os' and 'shutil' modules to first navigate over directories and second to overwrite files.
'C:/../modified' refers to the directory containing the files we have modified and want to use to overwrite the originals.
'C:/../originals' refers to the directory containing many sub-directories with files with the same names as in 'modified'.
The code works by listing every file in the modified directory, and for each file, we state the path for the file. Then, we look through all the sub-directories of the original directory, and where the modified and original files share the same name, we replace the original with the modified using shutil.copyfile().
Because we are working with Windows, it was necessary to change the direction of the slashes to '/'.
This is performed for every file in the modified directory.
If anyone ever has the same problem I hope this comes in handy!
import os
import shutil
for wav in os.listdir('C:/../modified'):
modified_file = 'C:/../modified/' + wav
for root, dirs, files in os.walk('C:/../original'):
for name in files:
if name == wav:
original_file = root + '/' + name
original_file = replace_file.replace('\\','/')
shutil.copyfile(modified_file, original_file)
print wav + ' overwritten'
print 'Complete'

Resources