I am trying to tail all the log files present under a directory and it's sub-directories recursively using below command
shopt -s globstar
tail -f -n +2 /app/mylogs/**/* | awk '/^==> / {a=substr($0, 5, length-8); next} {print a":"$0}'
and the output is below:
/app/mylogs/myapplog10062020.log:Hi this is first line
/app/mylogs/myapplog10062020.log:Hi this is second line
which is fine, but problem is when I add a new log file under /app/mylogs/,directory after I fire above tail command. tail will not take that new file into consideration.
Is there a way to get this done?
When you start your the tail process, you pass to it a (fixed) list of the files which tail is suppoed to follow, as you can see from the tail man page. This is different to, say, 'find', where you can in its options pass a file name pattern. After the process has been started, tail has no way of knowing that you suddenly want it to follow another file too.
If you want to have a feature like this, you would have to program your own version of tail, which gets passed for instance a directory to scan, and either periodically checks the directory content for change, or using a service such as inotify to be informed by directory changes.
Related
I have only two folders under a given directory. Is there any method to choose the second directory based on the order and not on the folder name?
Example: (I want to enter under doc2)
#ls
doc1 doc2
If you really want to use ls,
cd "$(ls -d */ | sed -n '2p')"
selects enters the second directory listed by it, independently of the number of directories provided by ls.
Parsing ls output is not a good idea generally, although it will work in most cases and will cause no harm if you are just using it in your interactive shell for fast navigation. You should not use this for serious programming.
You can use the tail command to get the last line
ls |tail -1
I have a service "A" which generates some compressed files comprising of the data it receives in requests. In parallel there is another service "B" which consumes these compressed files.
The trick is "B" shouldn't consume any of the files unless they are written completely. The service deduces this information by looking for a ".ready" file created by service "A" with name exactly same as the file generated along with the extension mentioned; once the compression is done. Service "B" uses Apache Camel to do this filtering.
Now, I am writing a shell script which needs the same compressed files and this would need the same filtering be implemented in shell. I need help writing this script. I am aware of find command but a naive shell user, so have very limited knowledge.
Example:
Compressed file: sumit_20171118_1.gz
Corresponding ready
file: sumit_20171118_1.gz.ready
Another compressed file: sumit_20171118_2.gz
No ready file is present for this one.
Of the above listed files only the first should be picked up as it has a corresponding ready file.
The most obvious way would be to use a busy loop. But if you are on GNU/Linux you can do better than that (from: https://www.gnu.org/software/parallel/man.html#EXAMPLE:-GNU-Parallel-as-dir-processor)
inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |
parallel -uj1 echo Do stuff to file {}
This way you do not even have to wait for the .ready file: The command will only be run when writing to the file is finished and the file is closed.
If, however, the .ready file is only written much later then you can search for that one:
inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |
grep --line-buffered '\.ready$' |
parallel -uj1 echo Do stuff to file {.}
I have a database server that it basic work is to import some specific files, do some calculations and provide data in a web interface.
It's planned for next weeks a hardware replacement, it needs to migrate the database. But there's one problem in it: the actual database is corrupted and show some errors in web interface. This is due to server freezing while importing/calculating, that's why the replacement.
So I'm not willing to just dump the db and restore in the new server. Doesn't make sense to still use the corrupted database and while dumping the old server goes really slow. I have a backup from all files to be imported (the current number is 551) and I'm working on a script to "re-import" all of them and have a nice database again.
The actual server takes ~20 minutes to import each new file. Let's say that new server takes 10 for each file due to its power... It's a long time! And here comes the problem: it receives new file hourly, so there will be more files when it finishes the job.
Restore script start like this:
for a in $(ls $BACKUP_DIR | grep part_of_filename); do
Question is: does this "ls" will have new file names when they come? File names are timestamp based, so they will be in the end of the list.
Or does this "ls" is execute once and results goes to a temp var?
Thanks.
ls will execute once, at the beginning, and any new files won't show up.
You can rewrite that statement to list the files again at the start of each loop (and, as Trey mentioned, better to use find, not ls):
while all=$(find $BACKUP_DIR/* -type f | grep part_of_filename); do
for a in $all; do
But this has a major problem: it will repeatedly process the same files over and over again.
The script needs to record which files are done. Then it can list the directory again and process any (and only) new files. Here's one way:
touch ~/done.list
cd $BACKUP_DIR
# loop while f=first file not in done list:
# find list the files; more portable and safer than ls in pipes and scripts
# fgrep -v -f ~/done.list pass through only files not in the done list
# head -n1 pass through only the first one
# grep . control the loop (true iff there is something)
while f=`find * -type f | fgrep -v -f ~/done.list | head -n1 | grep .`; do
<process file $f>
echo "$f" >> ~/done.list
done
how can we write a shell script in linux to copy a newly added content from a file and append it to another file.
I have a log file where errors will be stored and i am supposed to retrieve the new errors and store it in a database table. I will run a cron job invoking the shell script in a certain interval.
Edited:
Sample Log
140530 13:48:57 [ERROR] Event Scheduler: [root#%][test.event] Table 'test.test_event' doesn't exist
140530 13:48:57 [Note] Event Scheduler: [root#%].[test.event] event execution failed.
140530 13:49:57 [ERROR] Event Scheduler: [root#%][test.event] Table 'test.test_event' doesn't exist
140530 13:49:57 [Note] Event Scheduler: [root#%].[test.event] event execution failed.
initially i copied this into a file using cat but later some more error will be logged, only newly added lines should be logged.how can i do it in a routine basis.
Kindly help! Thanks in advance !
Simplest case
You can use tail -f to keep retrieving data from a file whenever it is appended to, then use >> (appending redirect) to append it to your second file.
tail -f file1.txt >> file2.txt
will "watch" file1.txt and append new content to file2.txt.
To test that it works, open another terminal and do:
echo "Hello!" >> file1.txt
You should see "Hello!" appear in file2.txt.
Please note that this will only work if the underlying I/O operation on file1.txt was an actual append. It won't work if you open file1.txt in a text editor and change its content, for instance. It also won't work as a cron job, because it needs to run continuously.
With cron
To periodically check for appends, you could do a diff on an earlier version of the file you saved somewhere, then use sed to get only those lines that were appended in the meantime:
diff file1_old.txt file1_current.txt | \
sed -r -e '/^[^>]/ d' -e 's/^> //' >> file2.txt
But then you have to take care of storing the earlier versions somewhere etc. in your cron job as well.
If you need to append (catenate) one file with another, use the "cat" command:
cat file1.txt file2.txt > fileall.txt
But if you need to modify the contents of a file, I recommend you to use "sed", or "grep" if what you need is a filter.
Sorry, your specification is a bit loose, so I cannot give you a more exact answer.
BTW. Database table? Can you please explain?
I've been trying to find a way to make tail wait for files to be created and then start tailing them.
For example, let's assume I have a logs directory which hasn't been created yet, but my application will create it. Let's say I have another process in which I'd like to run tail -F logs/*.log and have it start tailing all the files as soon as they are created. How would I go about doing that?
And another thing: if tail is tailing some file which is deleted, is it possible to have it start tailing it again, if it's re-created?
With some roughness, this works:
while inotifywait -q .; do kill $PID; tail -F -n 0 * & PID=$!; done
Once anything in a directory changes, the tail command gets restarted.
And another thing: if tail is tailing some file which is deleted, is it possible to have it start tailing it again, if it's re-created?
While you are tailing some file which is truncated and re-created; tail command handles it automatically. You don't need to re-issue the tail command for that particular file.