Assume before clear the log file, file size is 1.5M xxx.log.
I'm using sudo cp /dev/null xxx.log command to clear the log file.
After running the script, file size change to '0'.
But If i make some action, log size increase to 1.5 xxx.log with a lot of white space.
When I googling it need to stop writing process first!
But I don't want to stop the writing process, is there is any other ways.
When the log size increases, the logical size increases. It creates a sparse file. You could check using ls -ls xxx.log or du xxx.log to check. The file on disk does not use 1.5MB.
(edited because of negatives feedback)
What I tried to explain, is when you cp /dev/null xxx.log, you truncated the file. But the application, will continue to write on it, but at its current position. So you create a sparse file, it is empty from the start of the file to the last write of the application.
The ls -lh will print what I called the logical size, but du -hs will only print the number of block really allocated.
Related
I am trying to the setup log for the httpd server then I write a script in /etc/logrotate.d/apache.conf
/var/log/httpd/* {
daily
rotate 3
size 20K
compress
delaycompress
}
what I understood from this file
/var/log/httpd/* = where all the logs the stored and you want to logrotate them
daily = when you want to rotate the log
rotate= only 3 rotated logs should be kept
size = when your log file size meet with this condition
compress= make a zip file of rotated logs
delaycompress = kind of compress don't know much
so I hit to apache server that generates a lot of logs
after the log generated
where is my log are store how it is run only on when size condition matches or else
thanks for any guidance or help
one more thing when and how log rotate run why some people suggested to use cron job with logrotate
where is my log are store
Unless you specify the olddir directive, they are rotated within the same directory that they exist.
how it is run only on when size condition matches or else
If you specify the size directive, logs are only rotated if they are larger than that size:
size size
Log files are rotated only if they grow bigger then size bytes.
Files that do not meet the size requirement are ignored (https://linux.die.net/man/8/logrotate).
why some people suggested to use cron job with logrotate
logrotate is just an executable; it does not handle any facet of how or when it is executed. cron is typically how logrotate's execution is scheduled. For example, on CentOS, inside the /etc/cron.daily directory is an executable shell script:
#!/bin/sh
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
This script is executed one per day by cron and is how logrotate's execution is actually initiated.
Couple other problems:
/var/log/httpd/* - Unless you're rotating files out of the original directory with the olddir directive, never end your logrotate's directory definition with a wildcard (*). This definition is going to glom on to every file in that directory, including the files that you've already rotated. Logrotate has no way of keeping track of what files are actual logs and which are stored rotations. Your directory definition should be something like /var/log/httpd/*_log instead.
You should be reloading httpd after you rotate the log files, else it will probably continue to log into the rotated file because you never closed its handle.
sharedscripts
postrotate
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
endscript
Again this is a CentOS-specific example.
I am trying to copy a list of file in varying directories based on their sample name using the following script. Although the files are copied, the file are only partially copied. I have 64k lines in each file, but only exactly 40k lines are copied.
while read sample
do
echo copying ${sample}
cp ${sample_dir}/*${sample}*/file.tsv ${output_dir}/${sample}.file.tsv
done < ${input_list}/sample_list.txt
Am I missing something obvious here? Does the cp command have limits on how many lines it can copy?
Cheers,
I don't think CP command has limit on file size (unless ulimit has restrictions) to copy but it has limit on number of files to copy or check if it is related to new file creation with larger size.
check limits across the system using command ulimit,
ulimit -a
And verify the file size is not limited to 40k and if it is unlimited then no issue (like, file size (blocks, -f) unlimited).
Try with rsync command if it works for you,
rsync -avh source destination
Could you verify few things,
1) Verify if it's not file read error, like cat the file and save the output to another file (either manually/redirect).
cat *.tsv > /tmp/verify-size
And then verify the size on that file,
du -h /tmp/verify-size ---> This should be 64k
2) Create large dummy file with size > 40k (or exact size of .tsv (64k))
dd if=/dev/zero of=/tmp/verify-newfile-size bs=64000 count=1
du -h /tmp/verify-newfile-size ---> This should be 64k
If this new file creation success (if you are able to create a file with any size) then try CP command and verify the size
-OR- Try with dd command
dd if=/tmp/verify-newfile-size of=/tmp/verify-newfile-size2 bs=64000 count=1
3) Try with an atomic command,
mv /tmp/verify-newfile-size /tmp/verify-newfile-size3
I have a script start.sh which runs another script run.sh. run.sh starts my executable.
I wanted to record what run.sh does and I used a tee command to log it into a file loglink, start.sh:
exec run.sh | tee -a loglink
loglink is a soft linked file.
I have a logic where I have 3 log files log1.txt log2.txt log3.txt, I need each file to have a max size of only 1024 bytes. So in my code I keep checking every 5 seconds if log1.txt has reached max size, if reached max size, I change the softlink loglink to point to log2.txt, same with log2.txt and log3.txt in circular way.
As per my understanding, when I change the softlink from log1.txt to log2.txt, tee should print to log2.txt, but strange, tee is still saving output to log1.txt and not log2.txt
And to add on,
I see the softlink changed in ls -l
I tried something ls-l | tee loglink, it does to log2.txt.
Why the tee in script start.sh is not recognising this link change?
Am I missing some logic here?
In short, a filename or symbol link is just a proxy for program to tell the kernel setup the reading or writing path for the real file representation in kernel.
tee used file descriptor to represent files, as its source code(from freebsd) explains:
for (exitval = 0; *argv; ++argv)
if ((fd = open(*argv, append ? O_WRONLY|O_CREAT|O_APPEND :
O_WRONLY|O_CREAT|O_TRUNC, DEFFILEMODE)) < 0) {
warn("%s", *argv);
exitval = 1;
} else
add(fd, *argv);
once a file is opened, in your case, the symbol link is followed and open the target log file, after then, the path for writing do file is opened, and symbol link or filename is not need anymore.
A program which opens a file keeps that file link. If you change the link from outside, the program is not impressed and keeps writing to (or reading from) the original file.
Only if your program closes the file and reopens it, it will be using the new link.
You may, for example, open a file in vlc and play it, then, while playing, move it to a different directory. No problem. Then delete it. You now can't open it with a new program, but the old one is using it until the file is closed by that program.
Its a normal behaviour, as rightly explained in other answers.
As a solution you should periodically open and close output file in your run.sh or use
very nice utility for runtime change of the other process output:
reredirect -m newfile.log `pidof run.sh`
watch -n 3 du -sh >> log
this command may update the value every 3 seconds, but only latest size of current directory is stored in file log, the old values are simplely overwrite, so how to reserve the old values, and store it to the file named log?
watch does not overwrite the file. In fact, it is not possible to overwrite a file in the middle of a redirection.
What happens is watch only saves the differences between successive screens (using ANSI codes). It was not designed to be used to log something (therefore it is called "watch", anyway).
Use xxd to see the real content of the log file.
Perhaps this might do more what you want:
while sleep 3
do
du -sh
done >> log &
tail -F log
If I launch the following command to recover lost file on linux:
grep -a -B 150 -A 600 "class SuperCoolClass" /dev/sda10 > /tmp/SuperCoolClass.repair
Do I really need the "-a"? We need to recover from "sda10" some erased files (sabotage) and we have a bunch of them to recover and I believe removing the -a would be faster.
I believe the files to be on disk but not in binary.
thx
The file you are working on is /dev/sda10 which grep would assume to contain binary data. In order to treat it as text (which you are looking for) you need the -a otherwise grep will just print Binary file /dev/sda10 matches
In addition since the task is IO rather than CPU bound it would not be a big performance gain in any case.
In the future it's quite easy to test something like this by yourself:
create dummy 10Mb disk: dd if=/dev/zero of=testfs bs=1024 count=10000
create filesystem: mkfs.ext4 testfs
mount via loopback: mount -o loop ./testfs /mnt/test/
copy some stuff on the dummy filesystem
unmount: umount /mnt/test
run grep on the test file with different options
EDIT
it just occurred to me that maybe you are looking for the command '/usr/bin/strings' instead
something like:
extract all printable strings from ruined disk: /usr/bin/strings -a /dev/sda10 > /tmp/recovery
grep on the text only many times for different strings: grep "whatever" /tmp/recovery > /tmp/recovery.whatever
To recover a text file (only a text file) you accidently deleted / overwrote (provided you remember a phrase in that text file)
Ensure the safety of files by unmounting the directory with
umount /home/johndoe.
Find which hard disk partition the folder is at, say sda3
Switch to terminal as root.
Run
grep -a -A800 -B800 'search this phrase' /dev/sda3 | strings>recovery_log.txt
This will take a while. You can go through the file recovery_log.txt using any text editor, even while the command is running.