I can tail -f the last modified file using ls --sort=time | head -1 | xargs tail -f but I'm looking for something that would continuously run the first part as well, i.e. if a new files is created while the tail command is running it switches to tailing the new file. I thought about using watch command but that doesn't seem to work nicely with tail -f
It seems you need something like this:
#!/bin/bash
TAILPID=0
WATCHFILE=""
trap 'kill $(jobs -p)' EXIT # Makes sure we clean our mess (background processes) on exit
while true
do
NEWFILE=`ls --sort=time | head -n 1`
if [ "$NEWFILE" != "$WATCHFILE" ]; then
echo "New file has been modified"
echo "Now watching: $NEWFILE";
WATCHFILE=$NEWFILE
if [ $TAILPID -ne 0 ]; then
# Kill old tail
kill $TAILPID
wait $! &> /dev/null # supress "Terminated" message
fi
tail -f $NEWFILE &
TAILPID=$! # Storing tail PID so we could kill it later
fi
sleep 1
done
Related
I'm basically trying to make debugging easier for other scripts of mine.
(Centos 7.6)
What I need is a script doing :
tail -f the last file entry in a directory
if a new file appears in this directory, it logs this new file, smoothly
if I send a SIGINT (Ctrl+C), it doesn't leave orphans
with the less possible add-ons for the maximum portability
This is my non working solution :
CURRENT_FILE=`ls -1t | head -n1`
tail -n 100 -f "$CURRENT_FILE" &
PID=$!
while [ true ];
do
#is there a new file in the directory ?
NEW_FILE=`ls -1t | head -n1`
if [[ "$CURRENT_FILE" != "$NEW_FILE" ]]; then
#yes, so kill last tail
kill -9 $PID
clear
#tail on the new file
CURRENT_FILE=$NEW_FILE
tail -n 100 -f "$CURRENT_FILE"
PID=$!
fi
sleep 1s
done
The problem with this solution is that when I'm sending SIGINT (Ctrl+C), what I normally do when exiting a "tail -f", it leaves an orphan child in the background.
I've searched solution with "trap" but I don't get it well, and it doesn't seem to work with an eternal process like "tail -f".
I'll be glad to here your thoughts about that and get into advanced bash programming.
You can trap whenever the script exits and kill the process then. You don't need -9 to kill your tail though, that's overkill.
You can also use inotify to tell you when something happens in the directory instead of sleeping and rechecking. Here's a basic building block. inotify has a lot of events you can wait for. You can add detection if the file was moved/renamed so you don't have to restart the tail in those cases etc.
#!/bin/bash
killpid() {
if [[ -n $PID ]]; then
kill $PID
PID=""
fi
}
trap killpid EXIT
DIR="."
CURRENT_FILE="$(ls -1t "$DIR" | head -n1)"
tailit() {
echo "::: $CURRENT_FILE :::"
tail -n 100 -f "$CURRENT_FILE" &
PID=$!
}
tailit
# wait for any file to be created, modified or deleted
while EVENT=$(inotifywait -q -e create,modify,delete "$DIR"); do
# extract event
ev=$(sed -E "s/^${DIR}\/ (\S+) .+$/\1/" <<< "$EVENT")
# extract the affected file
NEW_FILE=${EVENT#${DIR}/ $ev }
case $ev in
MODIFY)
# start tailing the file if we aren't tailing it already
if [[ $NEW_FILE != $CURRENT_FILE ]]; then
killpid
CURRENT_FILE="$NEW_FILE"
tailit
fi
;;
CREATE)
# a new file, tail it
killpid
CURRENT_FILE="$NEW_FILE"
tailit
;;
DELETE)
# stop tailing if the file we are tailing was deleted
if [[ $NEW_FILE == $CURRENT_FILE ]]; then
echo "::: $CURRENT_FILE removed :::"
CURRENT_FILE=""
killpid
fi
;;
esac
done
You can use trap solution at the beginning of your shell.
#! /bin/bash
trap ctrl_c INT
function ctrl_c() {
if [[ -n "$PID" ]]; then
kill -9 $PID
fi
exit 0
}
CURRENT_FILE=`ls -1t | head -n1`
tail -n 100 -f "$CURRENT_FILE" &
PID=$!
while [ true ];
do
#is there a new file in the directory ?
NEW_FILE=`ls -1t | head -n1`
if [[ "$CURRENT_FILE" != "$NEW_FILE" ]]; then
#yes, so kill last tail
kill -9 $PID
clear
#tail on the new file
CURRENT_FILE=$NEW_FILE
tail -n 100 -f "$CURRENT_FILE" &
PID=$!
fi
sleep 1s
done
For some monitoring purposes, for a given list of files (that change path daily) I want to launch a tail -f in the background searching for specific strings
How do I properly get the exact pid of the tail commands generated by the for loop, so I could kill them later ?
Here's what I'm trying to do which I think gives me the pid of the script and not the tail process.
for f in $(find file...)
do
tail -f $f | while read line
do case $line in
*string_to_search*) echo " $line" | mutt -s "string detected in file : $1" mail#mail.com;
;;
esac
done &
echo $! >> pids.txt
done
I have a shell script that does a backup. I set this script in a cron but the problem is that the backup is heavy so it is possible to execute a second rsync before the first ends up.
I thought to launch rsync in a script and then get PID and write a file that script checks if the process exist or not (if this file exist or not).
If I put rsync in background I get the PID but I don't know how to know when rsync ends up but, if I set rsync (no background) I can't get PID before the process finish so I can't write a file whit PID.
I don't know what is the best way to "have rsync control" and know when it finish.
My script
#!/bin/bash
pidfile="/home/${USER}/.rsync_repository"
if [ -f $pidfile ];
then
echo "PID file exists " $(date +"%Y-%m-%d %H:%M:%S")
else
rsync -zrt --delete-before /repository/ /mnt/backup/repositorio/ < /dev/null &
echo $$ > $pidfile
# If I uncomment this 'rm' and rsync is running in background, the file is deleted so I can't "control" when rsync finish
# rm $pidfile
fi
Can anybody help me?!
Thanks in advance !! :)
# check to make sure script isn't still running
# if it's still running then exit this script
sScriptName="$(basename $0)"
if [ $(pidof -x ${sScriptName}| wc -w) -gt 2 ]; then
exit
fi
pidof finds the pid of a process
-x tells it to look for scripts too
${sScriptName} is just the name of the script...you can hardcode this
wc -w returns the word count by words
-gt 2 no more than one instance running (instance plus 1 for the pidof check)
if more than one instance running then exit script
Let me know if this works for you.
Test both for presence of pid file and status of the running process like this:
#!/bin/bash
pidfile="/home/${USER}/.rsync_repository"
is_running =0
if [ -f $pidfile ];
then
echo "PID file exists " $(date +"%Y-%m-%d %H:%M:%S")
previous_pid=`cat $pidfile`
is_running=`ps -ef | grep $previous_pid | wc -l`
fi
if [ $is_running -gt 0 ];
then
echo "Previous process didn't quit yet"
else
rsync -zrt --delete-before /repository/ /mnt/backup/repositorio/ < /dev/null &
echo $$ > $pidfile
fi
Hope this helps!!!
I want to run the command:
nc localhost 9998
Then I want my script to monitor a file and echo the contents of the file to this sub process whenever the file changes.
I can't work out the re-direction scheme. How can get access to the STDIN of the subprocess?
How about
tail -f $file |nc localhost 9998
Edit:
Since you already have a buffer, then you can try something like this:
while [ 1 ]; do
# Your stuff here.
buf=yourfunctionhere
buffer=$buffer$buf
if [ ! -z $buffer ]; then
echo $buffer |nc localhost 9998
# Empty buffer on success.
if [ $? -eq 0 ]; then
buffer="";
fi
fi
done
mkfifo X
some_program <X >output &
create_input >X
some_program will block reading X until create_input writes to it.
Two solutions that I found acceptable:
1) use coprocess, this way we have access to stdin and stdout of the process created by the coprocess command via the COPROC[0/1] array.
2) What I ultimately did is separate my application into two code blocks as shown below. The first block writes to stdout, that is then piped to the stdin of the second block. This gives me a clean way to buffer data when there are issues with netcat in the second code block:
{ while true;
write to STDOUT; } |
{ while true
nc localhost 9998 }
(in actuality the script is far more complex as the second command provides to-disk buffering when netcat is unable to connect, but the use of the pipe provides buffering so that data isn't lost when a network issue interrupts netcat)
I found a solution using diff and a simple bash script.
The following script execute cat $file > $namedpipe when file change. This is the script I made check-file.sh:
#!/bin/bash
file=$1
tmp=`mktemp`
cp "$file" "$tmp"
namedpipe=`mktemp`
rm -rf $namedpipe
mkfifo $namedpipe
function cleanup() {
echo "end of program"
rm -rf $tmp
rm -rf $namedpipe
exit 1;
}
trap cleanup SIGINT
tail -f $namedpipe 2> /dev/null | netcat localhost 9998 &
while true; do
diff=$(diff "$file" "$tmp")
if [ ! -z "$diff" ]; then
cat $file > $namedpipe
cp $file $tmp
fi
sleep 1
done
This script take as an input the name of a file. For example try these commands in your environment (whit netcat -l 9998 running):
touch /tmp/test
bash check-file.sh /tmp/test &
echo "change 1" > /tmp/test
sleep 1
echo "change 2" > /tmp/test
sleep 1
echo "change 3" > /tmp/test
Note: The temp file get cleaned up by the trap, so you can interrupt this script gracefuly.
I work with some log system which creates a log file every hour, like follows:
SoftwareLog.2010-08-01-08
SoftwareLog.2010-08-01-09
SoftwareLog.2010-08-01-10
I'm trying to tail to follow the latest log file giving a pattern (e.g. SoftwareLog*) and I realize there's:
tail -F (tail --follow=name --retry)
but that only follow one specific name - and these have different names by date and hour. I tried something like:
tail --follow=name --retry SoftwareLog*(.om[1])
but the wildcard statement is resoved before it gets passed to tail and doesn't re-execute everytime tail retries.
Any suggestions?
I believe the simplest solution is as follows:
tail -f `ls -tr | tail -n 1`
Now, if your directory contains other log files like "SystemLog" and you only want the latest "SoftwareLog" file, then you would simply include a grep as follows:
tail -f `ls -tr | grep SoftwareLog | tail -n 1`
[Edit: after a quick googling for a tool]
You might want to try out multitail - http://www.vanheusden.com/multitail/
If you want to stick with Dennis Williamson's answer (and I've +1'ed him accordingly) here are the blanks filled in for you.
In your shell, run the following script (or it's zsh equivalent, I whipped this up in bash before I saw the zsh tag):
#!/bin/bash
TARGET_DIR="some/logfiles/"
SYMLINK_FILE="SoftwareLog.latest"
SYMLINK_PATH="$TARGET_DIR/$SYMLINK_FILE"
function getLastModifiedFile {
echo $(ls -t "$TARGET_DIR" | grep -v "$SYMLINK_FILE" | head -1)
}
function getCurrentlySymlinkedFile {
if [[ -h $SYMLINK_PATH ]]
then
echo $(ls -l $SYMLINK_PATH | awk '{print $NF}')
else
echo ""
fi
}
symlinkedFile=$(getCurrentlySymlinkedFile)
while true
do
sleep 10
lastModified=$(getLastModifiedFile)
if [[ $symlinkedFile != $lastModified ]]
then
ln -nsf $lastModified $SYMLINK_PATH
symlinkedFile=$lastModified
fi
done
Background that process using the normal method (again, I don't know zsh, so it might be different)...
./updateSymlink.sh 2>&1 > /dev/null
Then tail -F $SYMLINK_PATH so that the tail hands the changing of the symbolic link or a rotation of the file.
This is slightly convoluted, but I don't know of another way to do this with tail. If anyone else knows of a utility that handles this, then let them step forward because I'd love to see it myself too - applications like Jetty by default do logs this way and I always script up a symlinking script run on a cron to compensate for it.
[Edit: Removed an erroneous 'j' from the end of one of the lines. You also had a bad variable name "lastModifiedFile" didn't exist, the proper name that you set is "lastModified"]
I haven't tested this, but an approach that may work would be to run a background process that creates and updates a symlink to the latest log file and then you would tail -f (or tail -F) the symlink.
#!/bin/bash
PATTERN="$1"
# Try to make sure sub-shells exit when we do.
trap "kill -9 -- -$BASHPID" SIGINT SIGTERM EXIT
PID=0
OLD_FILES=""
while true; do
FILES="$(echo $PATTERN)"
if test "$FILES" != "$OLD_FILES"; then
if test "$PID" != "0"; then
kill $PID
PID=0
fi
if test "$FILES" != "$PATTERN" || test -f "$PATTERN"; then
tail --pid=$$ -n 0 -F $PATTERN &
PID=$!
fi
fi
OLD_FILES="$FILES"
sleep 1
done
Then run it as: tail.sh 'SoftwareLog*'
The script will lose some log lines if the logs are written to between checks. But at least it's a single script, with no symlinks required.
We have daily rotating log files as: /var/log/grails/customer-2020-01-03.log. To tail the latest one, the following command worked fine for me:
tail -f /var/log/grails/customer-`date +'%Y-%m-%d'`.log
(NOTE: no space after the + sign in the expression)
So, for you, the following should work (if you are in the same directory of the logs):
tail -f SoftwareLog.`date +'%Y-%m-%d-%H'`
I believe the easiest way is to use tail with ls and head, try something like this
tail -f `ls -t SoftwareLog* | head -1`