my first question - Please Keep calm ;-)
our Problem is, that log4j uses the wrong logfile. Our configuration is a simple DailyRollingFileAppender
log4j.appender.dx4wsa=org.apache.log4j.DailyRollingFileAppender
log4j.appender.dx4wsa.File=${env.WFL_DIR}/log/dx4wsa-agents.log
log4j.appender.dx4wsa.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.dx4wsa.layout=org.apache.log4j.PatternLayout
log4j.appender.dx4wsa.layout.ConversionPattern=%d{dd.MM.yyyy HH:mm:ss}: %5p %-30c{1} %-50x %m%n
What we see is, that sometimes the logger uses old logfiles after rolling over. For example here
dx4wsa-agents.log.2016-10-12-18:12.10.2016 20:39:03: INFO VorgangLoeschen 21974690 Start executing agent on process instance = 21974690 and work item = 14f27076-f31f-48a7-849d-669189918730
You can see here, that this step is from 20:39:03 and the Logfile already rotated - it's the 18:00 Logfile. An "ls -ltr" Shows you the last Access.
-rw-r--r-- 1 tpdx4wf2 gpdoxis4 61944 Oct 12 17:59 dx4wsa-agents.log.2016-10-12-17
-rw-r--r-- 1 tpdx4wf2 gpdoxis4 51668039 Oct 12 17:59 dx4wsa-agents.log.2016-10-12-16
-rw-r--r-- 1 tpdx4wf2 gpdoxis4 40437528 Oct 12 19:59 dx4wsa-agents.log.2016-10-12-19
-rw-r--r-- 1 tpdx4wf2 gpdoxis4 1463292 Oct 12 20:54 dx4wsa-agents.log
-rw-r--r-- 1 tpdx4wf2 gpdoxis4 67702368 Oct 12 20:54 dx4wsa-agents.log.2016-10-12-18
Today we had a release and, stopped the Server, killed all Java-Threads deployed new Jar-Files and restarted the Server at 19:00!!! And log4j logged in the 18:00 file?
Related
I have a test application which runs every hour and uses unique log file at each execution. To clean the logs the following logrotate configuration has been set:
{
# Daily rotation with 1 week of backlog
daily
rotate 7
maxage 7
dateext
compress
}
The first day the log file compressed (which is ok) but a empty file is left and every other day that files is "emptied" and compressed. And that makes 6 files of every logfiles which fills the inodes table of the FS. Here's two examples :
-rw-r--r-- 1 root root 1752 Feb 11 01:36 J20190211013601_Status.txt-20190212.gz
-rw------- 1 root root 20 Feb 12 03:33 J20190211013601_Status.txt-20190213.gz
-rw------- 1 root root 20 Feb 13 03:37 J20190211013601_Status.txt-20190214.gz
-rw------- 1 root root 20 Feb 14 03:10 J20190211013601_Status.txt-20190215.gz
-rw------- 1 root root 20 Feb 15 03:12 J20190211013601_Status.txt-20190216.gz
-rw------- 1 root root 20 Feb 16 03:36 J20190211013601_Status.txt-20190217.gz
-rw------- 1 root root 20 Feb 17 03:44 J20190211013601_Status.txt-20190218.gz
-rw------- 1 root root 0 Feb 18 03:24 J20190211013601_Status.txt
-rw-r--r-- 1 root root 1752 Feb 11 02:36 J20190211023601_Status.txt-20190212.gz
-rw------- 1 root root 20 Feb 12 03:33 J20190211023601_Status.txt-20190213.gz
-rw------- 1 root root 20 Feb 13 03:37 J20190211023601_Status.txt-20190214.gz
-rw------- 1 root root 20 Feb 14 03:10 J20190211023601_Status.txt-20190215.gz
-rw------- 1 root root 20 Feb 15 03:12 J20190211023601_Status.txt-20190216.gz
-rw------- 1 root root 20 Feb 16 03:36 J20190211023601_Status.txt-20190217.gz
-rw------- 1 root root 20 Feb 17 03:44 J20190211023601_Status.txt-20190218.gz
-rw------- 1 root root 0 Feb 18 03:24 J20190211023601_Status.txt
How can i correct this, in order to delete the files after being compressed
Thanks for time and help,
This is how logrotate is supposed to function; your issue stems from the fact that you're using unique filenames every time your appplication runs.
When logrotate runs for the first time on each log, it's moving the log file from "J20190211023601_Status.txt" to "J20190211023601_Status.txt-20190212.gz" and then creating a new file named J20190211023601_Status.txt.
Logrotate has no inherent idea that those filenames are unique and thus will never be populated again; all it sees is a log it's rotated in the past, so it must be rotated again as per your configuration.
Your easiest solution here is to pass the nocreate directive for this logrotation; this will prevent that new log file from being created and subsequently rotated while still respecting the 7-day age limit on previously rotated files:
{
daily
maxage 7
dateext
compress
nocreate
}
I am testing kafka integration with spark as consumer. For debugging , have set-up log.retention.minutes = 2 in server.properties which cleans up .log file every 2mins. But .index file is not cleaned up
[cloudera#quickstart airline1-1]$ ls -l
total 0
-rw-r--r-- 1 root root 10485760 Apr 29 15:08 00000000000000000101.index
-rw-r--r-- 1 root root 0 Apr 29 15:08 00000000000000000101.log
-rw-r--r-- 1 root root 10485756 Apr 29 15:08 00000000000000000101.timeindex
Wondering why .index files are not cleaned up. Any insight would be helpful to understand what's happening in the background.
Also please share recommended approach to clean up log and index during testing. Found many google links referring to stop kakfa server -> remove topic partition files -> Restart kafka. But not inclined towards this approach , it could impact offsets state maintained in zookeeper.
Thanks very much!
$ pm2 --version
1.1.1
Started a shell script with pm2. It was gonna run for about 12 hours. Went back to check on it the next day and it was only halfway done. But I checked the logs and it looked like it had started over. Checked pm2 ls and yes, it had, 3 times. That seemed strange since all it was doing was sending some curls and had been doing fine for hours.
So I went to look at the error output:
[collumj#serverName ~]$ ll /opt/xyz/.pm2/logs | grep util-perf-testing
-rw-r--r-- 1 collumj alskdjflkas 0 Oct 12 21:27 util-perf-testing-error-54.log
-rw-r--r-- 1 collumj alskdjflkas 0 Oct 12 21:38 util-perf-testing-error-59.log
-rw-r--r-- 1 collumj alskdjflkas 6.8K Oct 12 21:36 util-perf-testing-out-54.log
-rw-r--r-- 1 collumj alskdjflkas 15K Oct 13 00:00 util-perf-testing-out-59__2016-10-13_00-00-00.log
-rw-r--r-- 1 collumj alskdjflkas 166K Oct 14 00:00 util-perf-testing-out-59__2016-10-14_00-00-00.log
-rw-r--r-- 1 collumj alskdjflkas 384K Oct 14 17:16 util-perf-testing-out-59.log
The two error logs are completely empty.
Does PM2 keep any sort of internal logs about restarts? The output of the shell script doesn't show any reason for it to restart and the error logs are empty. I'm looking at a complete mystery as to why this shell script would have restarted 3 times.
PM2 will store the reasons for resets in its log, which can be accessed like so: tail --lines 100 ~/.pm2/pm2.log.
I am trying to obtain a backup of 'newly' added files to a Fedora system. Files can be copied through a Windows Samba share and appear to retain the original created timestamp. However, because it retains this timestamp I am having issues identifying which files were newly added to the system.
Currently, the only way I can think of doing this is to have a master list snapshot of all the files on the system at a specific time. Then when I perform the backup I compare the previous snapshot with a current snapshot. It would detect files that were removed from the system but it seems excessive and I was thinking there must be an easier way to backup newly added files.
Terry
Try using find. Something like this:
find . -ctime -10
That will give you a list of files and directories, starting from within your current directory, that has had its state changed within the last 10 days.
Example:
My Downloads directory looks like this:
kobus#akira:~/Downloads$ ll
total 2025284
drwxr-xr-x 4 kobus kobus 4096 Nov 4 11:25 ./
drwxr-xr-x 41 kobus kobus 4096 Oct 30 09:26 ../
-rw-rw-r-- 1 kobus kobus 8042383 Oct 28 14:08 apache-maven-3.3.3- bin.tar.gz
drwxrwxr-x 2 kobus kobus 4096 Oct 14 09:55 ELKImages/
-rw-rw-r-- 1 kobus kobus 1469054976 Nov 4 11:25 Fedora-Live-Workstation-x86_64-23-10.iso
-rw------- 1 kobus kobus 351004 Sep 21 14:07 GrokConstructor-master.zip
drwxrwxr-x 11 kobus kobus 4096 Jul 11 2014 jboss-eap-6.3/
-rw-rw-r-- 1 kobus kobus 183399393 Oct 19 16:26 jboss-eap-6.3.0-installer.jar
-rw-rw-r-- 1 kobus kobus 158177216 Oct 19 16:26 jboss-eap-6.3.0.zip
-rw-rw-r-- 1 kobus kobus 71680110 Oct 13 13:51 jre-8u60-linux-x64.tar.gz
-rw-r--r-- 1 kobus kobus 4680 Oct 12 12:34 nginx-release-centos-7-0.el7.ngx.noarch.rpm
-rw-r--r-- 1 kobus kobus 3479765 Oct 12 14:22 ngx_openresty-1.9.3.1.tar.gz
-rw------- 1 kobus kobus 16874455 Sep 15 16:49 Oracle_VM_VirtualBox_Extension_Pack-5.0.4-102546.vbox-extpack
-rw-r--r-- 1 kobus kobus 7505310 Oct 6 10:29 sublime_text_3_build_3083_x64.tar.bz2
-rw------- 1 kobus kobus 41467245 Sep 7 10:37 tagspaces-1.12.0-linux64.tar.gz
-rw-rw-r-- 1 kobus kobus 42658300 Nov 4 10:14 tagspaces-2.0.1-linux64.tar.gz
-rw------- 1 kobus kobus 70046668 Sep 15 16:49 VirtualBox-5.0-5.0.4_102546_el7-1.x86_64.rpm
Here's what the find returns:
kobus#akira:~/Downloads$ find . -ctime -10
.
./tagspaces-2.0.1-linux64.tar.gz
./apache-maven-3.3.3-bin.tar.gz
./Fedora-Live-Workstation-x86_64-23-10.iso
kobus#akira:~/Downloads$
Most unices do not have a concept of file creation time. You can't make ls print it because the information is not recorded. If you need creation time, use a version control system: define creation time as the check-in time.
If your unix variant has a creation time, look at its documentation. For example, on Mac OS X (the only example I know of¹), use ls -tU. Windows also stores a creation time, but it's not always exposed to ports of unix utilities, for example Cygwin ls doesn't have an option to show it. The stat utility can show the creation time, called “birth time” in GNU utilities, so under Cygwin you can show files sorted by birth time with stat -c '%W %n' * | sort -k1n.
Note that the ctime (ls -lc) is not the file creation time, it's the inode change time. The inode change time is updated whenever anything about the file changes (contents or metadata) except that the ctime isn't updated when the file is merely read (even if the atime is updated). In particular, the ctime is always more recent than the mtime (file content modification time) unless the mtime has been explicitly set to a date in the future.
"Newly added files, Fedora" : The below examples will show a list with date and time.
Example, all installed packages : $ rpm -qa --last
Example, the latest 100 packages : $ rpm -qa --last | head -100
Example, create a text file : $ rpm -qa --last | head -100 >> last-100-packages.txt
I have changed the startWeblogic.sh script to give read access to the log files.
Here are examples:
-rw-r--r--. 1 oracle oinstall 81586 Apr 15 22:43 access.log
-rw-r--r--. 1 oracle oinstall 700087 Apr 15 22:45 localhost.log
-rw-r-----. 1 oracle oinstall 20553 Apr 15 22:49 localhost.out
From the above other users got read permissions for access.log and localhost.log, but its not assigning read access to localhost.out log file. Please suggest which configuration file to change.
set umask 022 in startweblogic.sh and startnodemanager.sh and wlscontrol.sh worked