Is it possible to trigger a command with every new line in to a file?
For example: I have a log file say maillog. I want to get every new entry in to the log file as a mail.
If a new entry like " Mail Sent " added in to maillog file then my script should grep the new entry and send me a mail with the entry(data).
I know its crazy but i want to automate my Linux box with these kind of things.
Regards,
Not so crazy. Check periodically (once per hour, per day, what you like) the file for new parts by storing the original length of the file, compare the length, in case it has grown, handle the part which was appended:
length=0
while sleep 3600 # use wanted delay here
do
new_length=$(find "$file" -printf "%s")
if [ $length -lt $new_length ]
then
tail --bytes=$[new_length-length] "$file" | handle_part
fi
length=$new_length
done
Now you only have to write that handle_part function which could for instance mail its input somewhere.
Using this way (instead of the obvious tail -f) has the advantage that you can store the current length into a file and later on restarting your script read that length again. So you won't get the whole file after a restart of your script (e. g. due to a machine reboot).
If you want a faster response you could have a look at inotify which is a facility on Linux to monitor file actions; so that polling could be replaced.
Use tail -f, that watches a file and sents whatever is appended to it to stdout. If you have a script that performs the desired action, say mail_per_line, then you can set it up as
tail -f maillog | mail_per_line
In this case, mail_per_line runs once and gets all the lines. If you want to spawn a separate process each time a line comes in, use the shell built-in read:
tail -f maillog | while IFS='' read line; do
send_a_message "$line"
done
To counter the effect described by Alfe, that a restart of this program will cause all the previous logs to be processed again, consider using logrotate.
Related
I have read most questions regarding this topic, but can't get an answer to my specific question:
I have a number of files in a directory, and I want to apply a command to each of these files and then create a new file with the outpot for every single file. I can only manage to write it into one file alltogether. As i expect to have ~ 500.000 files, i also would need the script to be as efficient as possible.
for f in *.bed; do sort -k1,1 -k2,2n; done
This command sorts each file accordingly and writes the ouput in the Shell - But i cannot manage to write to file in the for-loop without appending it with ">>" .
I'm thankful for any answer providing an approach or an already answered question on this topic!
You can use script like this:
for f in *.bed
do
sort -k1,1 -k2,2n $f >>new_filename
done
If you want to be sure new_filename is empty before run the loop you can clear the content in file with command (before for loop):
>new_filename
Currently I write a simple logger to log messages from my bash script. The logger works fine and I simply write the date plus the message in the log file. Since the log file will increase, I would like to set the limit of the logger to for example 1000 lines. After reaching 1000 lines, it doesn't delete or totally clear the log file. It should truncate the first line and replace it with the new log line. So the file keeps 1000 lines and doesn't increase further. The latest line should always be at the top of the file. Is there any built in method? Or how could I solve this?
Why would you want to replace the first line with the new message thereby causing a jump in the order of messages in your log file instead of just deleting the first line and appending the new message, e.g. simplistically:
log() {
tail -999 logfile > tmp &&
{ cat tmp && printf '%s\n' "$*"; } > logfile
}
log "new message"
You don't even need a tmp file if your log file is always small lines, just save the output of the tail in a variable and printf that too.
Note that unlike a sed -i solution, the above will not change the inode, hardlinks, permissions or anything else for logfile - it's the same file as you started with just with updated content, it's not getting replaced with a new file.
Your chosen example may not be the best. As the comments have already pointed out, logrotate is the best tool to keep log file sizes at bay; furthermore, a line is not the best unit to measure size. Those commenters are both right.
However, I take your question at face value and answer it.
You can achieve what you want by shell builtins, but it is much faster and simpler to use an external tool like sed. (awk is another option, but it lacks the -i switch which simplifies your life in this case.)
So, suppose your file exists already and is named script.log then
maxlines=1000
log_msg='Whatever the log message is'
sed -i -e1i"\\$log_msg" -e$((maxlines))',$d' script.log
does what you want.
-i means modify the given file in place.
-e1i"\\$log_msg" means insert $log_msg before the first (1) line.
-e$((maxlines))',$d' means delete each line from line number $((maxlines)) to the last one ($).
I need to be able to read the second last line of all the files within a specific directory.
These files are log files and, that specific line contains the status of tasks that ran, 'successful', 'fail', 'warning'.
I need to pull this to dump it after in reports.
At this stage i am looking only to pull the data, so the entire line, and will worry about the handling after.
As the line numbers are not set, they are irregular, I am looking at doing it with a 'while' loop, so it goes through the whole thing, but i am actually not getting the last 2 lines read, and also, i can read 1 file not all of them.
Any ideas on a nice little script to do this?
And anyone knows if this can be just done with just a linux command?
Use the tail command to get the last 2 lines, and then the head command to get the first of these:
for file in $DIR/*; do
tail -2 "$file" | head -1
done
Using bash on Linux.
I'm running find piped into xargs which launches a second bash script to do some processing on each file. I would like to maintain a "running tally" of file sizes and how many errors are encountered by the second script. In other words, every time the second script runs, it calculates the file size and adds it to the total so far, and also same thing if it encounters an error in its processing of the file. And I need this information to be available to the parent script after the find | xargs finishes.
I can do this by having the second script save and update a text file — a crude way to maintain a "global variable" — but I'm wondering if there's a nicer, more efficient way.
Can you use a pipe or Process Substitution to get the information back from the second script?
find ... | xargs second_script |
while read information
do
something useful with it
done
Or:
while read information
do
something useful with it
done < <(find ... | xargs second_script)
I am in the middle of writing a shell script to nullify/truncate a file if it reaches certain size. Also the file is being opened/written by a process all the time. Now every time when I nullify the file, will the file pointer be repositioned to the start of the file or will it remain in its previous position? Let me know if we could reset the file pointer once the file has been truncated?
The position of the file pointer depends on how the file was opened by the process that has it open. If it was opened in append mode, then truncating the file will mean that new data will be written at the end of the file, which is actually the beginning too the first time it writes after the file is truncated. If it was not opened in append mode, then truncating the file will simply mean that there is a series of virtual zero bytes at the start of the file, but the real data will continue to be written at the same point as the last write finished. If the file is being reopened by the other process, rather than being held open, then roughly the same rules apply, but there is a better chance that the file will be written at the beginning. It all depends on how the first process to write to the file after the truncation is managing its file pointer.
You can't reset the file pointer of another process, AFAIK.
A cron job or something like this will do the task; it will find every files bigger than 4096bytes then nullified the files
$ find -type f -size 4096c -print0 | while IFS= read -r -d $'\0' line; do cat /dev/null > $line; done
enter link description here