Suppose some logs is being rotated by size 50MB each .
I did 'grep' some string and my string is present in log file, log_3 and I want to copy content of log_3 before it gets rotated(renamed) to log_4 .
Please suggest how to take backup of content of log_3 before it is rotated .
I just need the content of log_3 . I don't want like I copied log_3 (by cp -p log_3 log_3_backup) but by that time logs got rotated and now log_3_backup contains content of log_2 .Is there any way we can do to avoid this . Working on an automation project I need a solution to this . Thank you very much for your suggestions in advanced . You May share python or shell script .
You can get the file inode number that won't change if renamed, then reference the file by that name
for f in *.log; do
# get inode of file
iname=$(ls -i $f)
# test file contents for pattern presence
if grep -q 'some pattern' $f; then
# the file contains the searched pattern, let's do something
# find by inode number and move it
find -inum $iname -exec mv {} {}.bak ';'
fi
done
Perhaps getting a backup of the file is not necessary anymore, let's grep it again
find -inum $iname -print0 | xargs -r0 grep 'some pattern'
Related
I'm writing a shell script on a Linux machine to be run via a crontab which is meant to move all files older than the current day to a new folder, and then tar and zip the entire folder. Seems like a simple task but for some reason, I'm running into all kinds of roadblocks. I'm new to this and self-taught so any help or redirection would be greatly appreciated.
Specific criteria for which files to archive:
All log files are in /home/tech/logs/ and all pdfs are in /home/tech/logs/pdf
All files are over a day old as indicated by the file name (file name does not include $CURRENT_DATE)
All files must be *.log or *.pdf (i.e. don't archive files that don't include $CURRENT_DATE if it isn't a log or pdf file.
Filename formatting specifics:
All the log file names are in home/tech/logs in the format NAME 00_20180510.log, and all the pdf files are in a "pdf" subdirectory (home/tech/logs/pdf) with the format NAME 00_20180510_00000000.pdf ("20180510" would be whenever the file was created and the 0's would be any number). I need to use the name rather than the file metadata for the creation date, and all files (pdf/log) whose name does not include the current date are "old". I also can't just move all files that don't contain $CURRENT_DATE in the name because it would take any non-*.pdf or *.log files with it.
Right now the script creates a new folder with a new pdf subdir for the old files (mkdir -p /home/tech/logs/$ARCHIVE_NAME/pdf). I then want to move the old logs into $ARCHIVE_NAME, and move all old pdfs from the original pdf subdirectory into $ARCHIVE_NAME/pdf.
Current code:
find /home/tech/logs -maxdepth 1 -name ( "*[^$CURRENT_DATE].log" "*.log" ) -exec mv -t "$ARCHIVE_NAME" '{}' ';'
find /home/tech/logs/pdf -maxdepth 1 -name ( "*[^$CURRENT_DATE]*.pdf" "*.pdf" ) -exec mv -t "$ARCHIVE_NAME/pdf" '{}' ';'
This hasn't been working because it treats the numbers in $CURRENT_DATE as a list of numbers to exclude rather than a literal string.
I've considered just using tar's exclude options like this:
tar -cvzPf "$ARCHIVE_NAME.tgz" --directory /home/tech/logs --exclude="$CURRENT_DATE" --no-unquote --recursion --remove-files --files-from="/home/tech/logs/"
But a) it doesn't work, and b) it would theoretically include all files that weren't *.pdf or *.log files, which would be a problem.
Am I overcomplicating this? Is there a better way to go about this?
I would go about this using bash's extended glob features, which allow you to negate a pattern:
#!/bin/bash
shopt -s extglob
mv /home/tech/logs/*!("$CURRENT_DATE")*.log "$ARCHIVE_NAME"
mv /home/tech/logs/pdf/*!("$CURRENT_DATE")*.pdf "$ARCHIVE_NAME"/pdf
With extglob enabled, !(pattern) expands to everything that doesn't match the pattern (or list of pipe-separated patterns).
Using find it should also be possible:
find /home/tech/logs -name '*.log' -not -name "*$CURRENT_DATE*" -exec mv -t "$ARCHIVE_NAME" {} +
Building on #tom-fenech answer, optimized to avoid many mv invocations:
find /home/tech/logs -maxdepth 1 -name '*.log' -not -name "*_${CURRENT_DATE?}.log" | \
xargs mv -t "${ARCHIVE_NAME?}"
An interesting feature, from processing the file thru pipes, is the ability to filter them with extra tools (aka grep :), which can (arguably) become more readable i.e. ->
find /home/tech/logs -maxdepth 1 -name '*.log' | fgrep -v "_${CURRENT_DATE?}" | \
xargs mv -t "${ARCHIVE_NAME?}"
Then similarly for the pdf ones, BTW you can "dry-run" above by just replacing mv by echo mv.
--jjo
Im trying to look for the text Elapsed time inside a specific log file names vsim.log. Im not familiar with grep, but after some googling I found that grep -r will allow me to do recursively searches and grep -r "Elapsed time" will do recursive searches for that phrase within all files in my directory. According to this link, I can then do grep -r "Elapsed time" ./vsim* to recursively search through the directories for files starting with vsim and look inside those files for Elapsed time. However, when i tried this i get grep: No match. which i know is not true since I know the files exist there with those keywords. What am i messing up?
Continuing from my comment, you can use find to locate the file vsim.log if you do not know its exact location and then use the -execdir option to find to grep the file for the term Elapsed time, e.g.
find path -type f -name "vsim.log" -execdir grep -H 'Elapsed time' '{}' +
That will return the filename along with the matched text which you can simply parse to isolate the filename if desired. You can process all files that match if you anticipate more than one by feeding the results of the find command into a while read -r loop, e.g.
while read -r match; do
# process "$match" as desired
echo "Term 'Elapsed time' found in file ${match%:*}"
done < <(find path -type f -name "vsim.log" -execdir grep -H 'Elapsed time' '{}' +)
Where:
find is the swiss-army knife for finding files on your system
path can be any relative or absolute path to search (e.g. $HOME or /home/dorojevets) to search all files in your home directory
the option -type f tells find to only locate files (see man find for link handling)
the option -name "foo" tell find to only locate files named foo (wildcards allowed)
the -exec and -execdir options allow you to execute the command that follows on each file (represented by '{}')
the grep -H 'Elapsed time' '{}' being the command to execute on each filename
the + being what tells find it has reached the end of the command (\; used with -exec)
finally, the ${match%:*} parameter expansion on the variable $match is used to parse the filename from filename:Elapsed time returned by grep -H (the %:* simply being used to trim everything to the first : from the right of $match)
Give that a try and compare the execution time to a recursive grep of the file tree. What you may be missing in this discussion, is that you use find if you know some part of the filename (or file mod time, or set of permissions, etc) that contains the information you need. It can search millions of files in a file tree vastly quicker than you can recursively grep every single file. If you have no clue what file may contain the needed info -- then use grep and just wait...
Try:
grep -r "Elapsed time" * --include vsim.log
Or this answer Use grep --exclude/--include syntax to not grep through certain files.
The following works just in case if you are using Mac:
To search UUID in *.c files recursively under given folder "/home" use the following:
grep -r "UUID" --include "*.c" /home/
To recursively search UUID in all main.c files in multiple projects under current folder use the following:
grep -r "UUID" --include "main.c" .
I am building a JAVA app that saves the output of bash commands into a list.
#root ~/a $ zipinfo -1 data.zip
a.txt
b.txt
test/
test/c.txt
root# ~/a $ find .
.
./test
./test/c.txt
./b.txt
./data.zip
./a.txt
The idea is to compare files and directories from the zip to what is on the disk and remove any differences from the disk. In this example test/c.txt should only be removed.
As you can see the format is different. Which command do I need to have the same style as zipinfo -1?
I tried commands like:
ls -R
ls -LR | grep ""
find . -exec ls -dl \{\} \; | awk '{print $9}'
One way to remove the . prefix is to use sed. For example:
find . ! -name . | sed -e 's|^\./||'
The ! -name . removes the . entry from the list. The sed part removes the ./ from the beginning of each line.
It should give you pretty much the same format as your zipinfo, though perhaps not the same order.
Note: the above suggestion is compatible with both Linux and other unix versions such as MacOS X. On Linux specifically you can achieve the same result with:
find . ! -name . -printf "%P\n"
The -printf predicate is specific to the Linux findutils, and the %P format prints each found file, removing the search directory from its beginning.
The easiest thing might be to ask zip itself for the list. Its -sf option means to show the files it would add if it were going to, without actually creating a zip file. So:
$ zip add fake.zip -r -sf *
zip warning: name not matched: fake.zip
Would Add/Update:
a.txt
b.txt
test/
test/c.txt
Total 4 entries (0 bytes)
You Java code would then have to skip the extraneous header/footer lines that it also outputs, but the file list itself would be in the same format that you're getting from your other execution of zip.
I'm having a really hard time finding an answer for this because most people want to find a list of files, then do the xargs.
I have a file with a list of file names. I would like to go through the list of file names in that file and one by one search a single directory (not recursive) and any time it finds the file, move it to a sub folder.
cat test.txt | xargs find . -type f -iname
At this point, I get this error:
find: paths must precede expression: file1
Why don't you use something like:
for i in `cat test.txt`
do
test -e $i && mv <src> <dst>
done
I've seen numerous examples for replacing one string with another among multiple files but what I want to do is a bit different. Probably a lot simpler :)
Find all the files that match a certain string and replace them completely with the contents of a new file.
I have a find command that works
find /home/*/public_html -name "index.php" -exec grep "version:1.23" '{}' \; -print
This finds all the files I need to update.
Now how do I replace their entire content with the CONTENTS of /home/indexnew.txt (I could also name it /home/index.php)
I emphasize content because I don't want to change the name or ownership of the files I'm updating.
find ... | while read filename; do cat static_file > "$filename"; done
efficiency hint: use grep -q -- it will return "true" immediately when the first match is found, not having to read the entire file.
If you have a bunch of files you want to replace, and you can get all of their names using wildcards you can try piping output to the tee command:
cat my_file | tee /home/*/update.txt
This should look through all the directories in /home and write the text in my_file to update.txt in each of those directories.
Let me know if this helps or isn't what you want.
I am not sure if your command without -l and then print it is better than to add -l in grep to list file directly.
find /home/*/public_html -name "index.php" -exec grep -l "version:1.23" '{}' \; |xargs -i cp /home/index.php {}
Here is the option -l detail
-l, --files-with-matches
Suppress normal output; instead print the name of each input
file from which output would normally have been printed. The
scanning will stop on the first match. (-l is specified by
POSIX.)