I have one file. It's not a log file. Every day I need to copy it somewhere else adding timestamp to its name. I need to keep last ten (10) days copies.
I am considering using logrotate service (server is running CentOS).
My question is if there is some limit on logrotate to rotate only log files or if I can use any other file and location. That is may I specify some other location where to put copies with timestamps and to hold only last 10 (days) copies?
Thanks in advance for your hints.
You can rotate any file in any location e.g. take a look at /etc/logrotate.d/samba (if installed) which rotates log.smbd. Just create a new file in /etc/logrotate.d/ and configure for your file to be rotated. A nice description is here for centOS not different (here).
The problem is the timestamp in the name. If you want this it is no longer rotating. With logrotate the latest archived file is always number 1 and thus all older once are getting renamed. But they keep their time of last modification stored in the file system untouched. But this time is not in the name.
As an alternative you can just create a new cron job by adding a file in one of the \etc\cron.*\ directories. This cron job can move you file adding a timestamp e.g. by using date +%y%m%d_%H:%M and create a new file using touch. No need for logrotate.
Related
I have a logrotate configuration which is given below.
/var/log/Test.log {
size 50k
missingok
rotate 10
notifempty
create 0640 root root
}
The file is rotating successfully once it reached the size of 50kb in the below format
Test.log.1
Test.log.2
Test.log.3
What I can do if I want to rotate files in the below format
Test_1.log
Test_2.log
Test_3.log
Try this one,it should help you.
/var/log/Test.log {
...
postrotate
/usr/bin/mv Test.log.1 Test_1.log
...
endscript
}
OK, so the problem here is that logrotate uses these filename extensions to keep track of its rotations. So if you have x.log, it inherently needs a predictable, sequential extension to keep track of what is what; i.e. x.log.1 is the first rotation, x.log.2 is the second, etc.
By renaming the file like that, you're moving the extension to the middle of the file and subsequently leaving each file with the same extension: .log. So as far as logrotate is concerned, it will think those are all unrelated files, not rotations of a previous log.
E.g., if you rename Test.log.1 to Test_1.log, and leave that file in the same folder, logrotate won't know that it's a rotation of Test.log and create a new Test.log.1 upon next rotation. And so on, and so on.
So the short answer is no, there is no logrotate directive that will rename a file like you want. You will have to create this functionality yourself using a postrotate script, and you will also have to (at some point) move these files out of the var/log/ directory once they're renamed, using either said postrotate script or the olddir directive.
The dateext directive might also be useful, as in my opinion date extensions are typically more user-friendly than numerical ones.
I a series of applications on Linux systems that I need to basically constantly 'stream' out or even just 'tail' out but the challenge is the filenames are constantly rolling and changing.
The are all date encoded (dates being in different formats) and each then have different incremented formats.
Most of them start with one and increase, but one doesn't have an extension and then adds an extension past the first file and the other increments a number but once hitting 99 rolls to increment a alpha and returns the numeric to 01 and then up again as it rolls so quickly.
I just have the OS level shell scripting, OS command line utilities, and perl available to me to handle this situation for another application to pickup and read these logs.
The new files are always created right when it starts writing to the new file and groups of different logs (some I am reading some I am not) are being written to the same directory so I cannot just pickup anything hitting the directory.
If I simply 'tail -n 1000000 -f |' them today this works fine for the reader application I am using until the file changes and I cannot setup file lists ranges within the reader application, but can pre-process them so they basically appear as a continuous stream to the reader vs. the reader directly invoking commands to read them. A simple Perl log reader like this also work fine for a static filename but not for dynamic ones. It is critical I don't re-process any logs lines and just capture new lines being written to the logs.
I admit I am not any form a Perl guru, and the best answers / clue I've been able to find so far is the use of Perl's Glob function to possibly do this but the examples I've found basically reprocess all of the files on each run then seem to stop.
Example File Names I am dealing with across multiple apps I am trying to handle..
appA_YYMMDD.log
appA_YYMMDD_0001.log
appA_YYMMDD_0002.log
WS01APPB_YYMMDD.log
WS02APPB_YYMMDD.log
WS03AppB_YYMMDD.log
APPCMMDD_A01.log
APPCMMDD_B01.log
YYYYMMDD_001_APPD.log
As denoted above the files do not have the same inode and simply monitoring the directory for change is not possible as a lot of things are written there. On the dev system it has more than 50 logs being written to the directory and thousands of files and I am only trying to retrieve 5. I am seeing if multitail can be made available to try that suggestion but it is not currently available and installing any additional RPMs in the environment is generally a multi-month battle.
ls -i
24792 APPA_180901.log
24805 APPA__180902.log
17011 APPA__180903.log
17072 APPA__180904.log
24644 APPA__180905.log
17081 APPA__180906.log
17115 APPA__180907.log
So really the root of what I am trying to do is simply a continuous stream regardless if the file name changes and not have to run the extract command repeatedly nor have big breaks in the data feed while some script figures out that the file being logged to has changed. I don't need to parse the contents (my other app does that).. Is there an easy way of handling this changing file name?
How about monitoring the log directory for changes with Linux inotify, e.g. Linux::inotify2? Then you could detect when new log files are created, stop reading from the old log file and start reading from the new log file.
Try tailswitch. I created this script to tail log files that are rotated daily and have YYYY-MM-DD on their names. To use this script, you just say:
% tailswitch '*.log'
The quoting prevents the shell from interpreting the glob pattern. The script will perform glob pattern from time to time to switch to a newer file based on its name.
I have a batch that integrates an xml file time to time but could happens daily. After it integrates it puts in a folder like /archives/YYMMDD(current day). The problem is if the same file is integrated twice. So I need a script what verifys the file (with diff command its possible but risky to make a bottleneck) but the problem is I can't find to resolve how to make to give the second files location.
P.S. I can't install on the server anything.
Thanks in advance.
I am using file as input for logs in logstash . My log files are rotated daily so , I wanted to ask how can we configure file plugin of logstash so that it work with the files that are rotated daily. And adding to this, is log rotation available with file beat as well.
I am trying to answer your questions in part.
First - log rotation.
From the docs:
Note that the rotated filename will be treated as a new file so if
start_position is set to beginning the rotated file will be
reprocessed.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
That means, that if you have a rename in your file rotation, you will likely double your file (unless the path excludes the renamed file I believe).
If your path excludes your renamed file, then it should be fine.
I fixed this in a different way (in java and python accordingly).
I disable renaming of files and instead name the log file with the date prefix. So for me, in my java app, the file name is:
my-server-log-%h-%d.log
Since I am working in a distributed environment, I incorporate the hostname into my logfile name.
%h = hostname
%d = date
This ends up in my file being named:
my-server-log-pandaadb-2016-06-20.log
This file is never renamed. I modified my rotation algorithm to simply not rename and instead at midnight create a new file and leave the previous file untouched.
This has the effect that logstash (correctly) knows that it has read all lines in the previous file. It picks up the new file since I am using wildcards in my input. No logs are duplicated.
This also works quite well in combination with rsync by the way.
I hope that helps,
Artur
Edit: I have not worked with filebeat so far, so I can't comment on that part.
Recently I was asked the following question in an interview.
Suppose I try to create a new file named myfile.txt in the /home/pavan directory.
It should automatically create myfileCopy.txt in the same directory.
A.txt then it automatically creates ACopy.txt,
B.txt then BCopy.txt in the same directory.
How can this be done using a script? I may know that this script should run in crontab.
Please don't use inotify-tools.
Can you explain why you want to do?
Tools like VIM can create a backup copy of a file you're working on automatically. Other tools like Dropbox (which works on Linux, Windows, and Mac) can version files, so it backs up all the copies of the file for the last 30 days.
You could do something by creating aliases to the tools you use for creating these file. You edit a file with the tools you tend to use, and the alias could create a copy before invoking a tool.
Otherwise, your choice is to use crontab to occasionally make backups.
Addendum
let me explain suppose i have directory /home/pavan now i create the file myfile.txt in that directory , immediately now i should automatically generate myfileCopy.txt file in the same folder
paven
There's no easy user tool that could do that. In fact, the way you stated it, it's not clear exactly what you want to do and why. Backups are done for two reasons:
To save an older version of the file in case I need to undo recent changes. In your scenario, I'm simply saving a new unchanged file.
To save a file in case of disaster. I want that file to be located elsewhere: On a different computer, maybe in a different physical location, or at least not on the same disk drive as my current file. In your case, you're making the backup in the same directory.
Tools like VIM can be set to automatically backup a file you're editing. This satisfy reason #1 stated above: To get back an older revision of the file. EMACs could create an infinite series of backups.
Tools like Dropbox create a backup of your file in a different location across the aether. This satisfies reason #2 which will keep the file incase of a disaster. Dropbox also versions files you save which also is reason #1.
Version control tools can also do both, if I remember to commit my changes. They store all changes in my file (reason #1) and can store this on a server in a remote location (reason #2).
I was thinking of crontab, but what would I backup? Backup any file that had been modified (reason #1), but that doesn't make too much sense if I'm storing it in the same directory. All I would have are duplicate copies of files. It would make sense to backup the previous version, but how would I get a simple crontab to know this? Do you want to keep the older version of a file, or only the original copy?
The only real way to do this is at the system level with tools that layer over the disk IO calls. For example, at one location, we used Netapps to create a $HOME/.snapshot directory that contained the way your directory looked every minute for an hour, every hour for a day, and every day for a month. If someone deleted a file or messed it up, there was a good chance that the version of the file exists somewhere in the $HOME/.snapshot directory.
On my Mac, I use a combination of Time Machine - which backs up the entire drive every hour, and gives me a snapshot of my drive that stretches back over a year and a half) and Dropbox which keeps my files stored in the main Dropbox server somewhere. I've been saved many times by that combination.
I now understand that this was an interview question. I'm not sure what was the position. Did the questioner want you to come up with a system wide way of implementing this, like a network tech position, or was this one of those brain leaks that someone comes up with at the spur of the moment when they interview someone, but were too drunk the night before to go over what they should really ask the applicant?
Did they want a whole discussion on what backups are for, and why backing up a file immediately upon creation in the same directory is a stupid idea non-optimal solution, or were they attempting to solve an issue that came up, but aren't technical enough to understand the real issue?