I trust you all doing good.
we planning to implement log rotation for below file.
stdout.log
we use below log rotation configuration file.
/usr/local/rms/kafka/kafka-connect-fluentd/stdout.log {
daily
rotate 7
maxsize 100M
minsize 10M
copytruncate
delaycompress
compress
notifempty
missingok
}
we have noticed the file is rotating and file is truncated.But application does not write logs to new file.
we tried to send the HUP signal and it did not work.
-rw-r--r-- 1 appuser appuser 8.2M Feb 20 03:11 stdout.log.4.gz
-rw-r--r-- 1 appuser appuser 4.0M Feb 20 23:48 stdout.log.3.gz
-rw-r--r-- 1 appuser appuser 7.6M Feb 20 23:49 stdout.log.2.gz
-rw-r--r-- 1 appuser appuser 2.1G Feb 21 03:39 stdout.log.1
-rw-r--r-- 1 appuser appuser 2.2G Feb 21 14:15 stdout.log
The application itself do not have a reload option, We stop the application and start the application when we need to reload or restart the application.
we use below command to bring up the application
nohup connect-standalone ${BASE}/connect-standalone.properties
${BASE}/FluentdSourceConnector.properties >& ${BASE}/stdout.log &
we use below command to kill the application
kill -9 <processid>
How do we implement a log rotating mechanism for this situation ?
>& FILE
Is the obsolete syntax for:
> FILE 2>&1
The > FILE redirects standard output of the command to file named FILE. However, before that happens, "if the file does not exist it is created; if it does exist it is truncated to zero size".
So each time you restarted your command, your file was (properly) truncated by the shell. What you want is to append to the file. Do that by using >> redirection. Including that you want to redirect both stdout and stderr, use:
2>&1 >>FILE
The 2>&1 redirects stderr to stdout and the >>FILE appends stdout to FILE.
Related
There are two directories that contains these files:
First one /usr/local/nagios/etc/hosts
[root#localhost hosts]$ ll
total 12
-rw-rw-r-- 1 apache nagios 1236 Feb 7 10:10 10.80.12.53.cfg
-rw-rw-r-- 1 apache nagios 1064 Feb 27 22:47 10.80.12.62.cfg
-rw-rw-r-- 1 apache nagios 1063 Feb 22 12:02 localhost.cfg
And the second one /usr/local/nagios/etc/services
[root#localhost services]$ ll
total 20
-rw-rw-r-- 1 apache nagios 2183 Feb 27 22:48 10.80.12.62.cfg
-rw-rw-r-- 1 apache nagios 1339 Feb 13 10:47 Check usage _etc.cfg
-rw-rw-r-- 1 apache nagios 7874 Feb 22 11:59 localhost.cfg
And I have a script that goes through file in Hosts directory and paste some lines from that file in the file in the Services directory.
The script is ran like this:
./nagios-contacts.sh /usr/local/nagios/etc/hosts/10.80.12.62.cfg /usr/local/nagios/etc/services/10.80.12.62.cfg
How can I achieve that another script calls my script and goes through every file in the Hosts directory and does its job for the files with the same name in the Service directory?
In my script I´m pulling out contacts from the 10.80.12.62.cfg in the Hosts directory and appending them to the file with the same name in the Service directory.
Don't use ls output as an input to for loop instead use the built-in wild-cards. See why it's not a good idea.
for f in /usr/local/nagios/etc/hosts/*.cfg
do
basef=$(basename "$f")
./nagios-contacts.sh "$f" "/usr/local/nagios/etc/services/${basef}"
done
It sounds like you just need to do some iteration.
echo $(pwd)
for file in $(ls); do ./nagious-contacts.sh $file; done;
So it will loop over all files in the current directory.
You can also modify it as well by doing something more absolute.
abspath=$1
for file in $(ls $abspath); do ./nagious-contacts.sh $abspath/$file; done
which would loop over all files in a set directory, and then pass the abspath/filename into your script.
Every time after newsyslog rotates the log file, syslog stops logging into the file. Until a syslogd restart is done.
(myserver:wheel)# logger -p local1.info -t myprocess "hello thiru"; ll myfile.log; cat myfile.log
-rw-r--r-- 1 root wheel 0B Nov 10 11:26 myfile.log
(myserver:wheel)# /etc/rc.d/syslogd restart
Stopping syslogd.
Starting syslogd.
(myserver:wheel)# logger -p local1.info -t myprocess "hello thiru"; ll myfile.log; cat myfile.log
-rw-r--r-- 1 root wheel 44B Nov 10 12:04 myfile.log
Nov 10 12:04:31 myserver myprocess: hello thiru
(myserver:wheel)#
On Linux (which uses logrotate), we can solve this by doing syslog/rsyslog restart in the postrotate section of the logroate conf.
Is there something similar to postrotate in newsyslog?
Edit:
Syslog and newsyslog conf files:
(TPC-E11-36:wheel)# cat /etc/newsyslog.d/newsyslog-myprocess.conf
/var/log/myfile.log 644 20 10000 * Z
(TPC-E11-36:wheel)# cat /etc/syslog.d/syslog-myprocess.conf
!myprocess
local1.info /var/log/myfile.log
(TPC-E11-36:wheel)#
Add the path of the pidfile into the /etc/newsyslog.conf and if required a signal for example 1 that stands for SIGHUP, more info about newsyslog here: https://www.freebsd.org/doc/handbook/configtuning-syslog.html
The format/example is this:
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
/var/log/your-app.log root:wheel 600 7 * #T00 GBJC /var/run/your-app.pid 1
Also see this answer: https://serverfault.com/a/701563/94862
There is actually something similar to postrotate in newsyslog, it's the R flag in the configuration file in combination with path_to_pid_cmd_file (that's what it's called in the FreeBSD manual entry of newslog.conf(5)). With this flag set, the path provided will not be interpreted as a path to a PID file, but as a path to a script (for example a shell script) which will be executed after the log rotation is finished.
You can even pass arguments to the script, but for this you'll have to use the IFS-variable (separated by a space, the argument would be interpreted as a signal and likely throw an error).
# logfilename [owner:group] mode count size when flags [/cmd_file]
/var/log/myfile.log root:wheel 644 20 10000 * ZR /path/to/my/script${IFS}myargument
I was trying to redirect the TOP command output in the particular file in every 5 minutes with the below command.
top -b -n 1 > /var/tmp/TOP_USAGE.csv.$(date +"%I-%M-%p_%d-%m-%Y")
-rw-r--r-- 1 root root 0 Dec 9 17:20 TOP_USAGE.csv.05-20-PM_09-12-2015
-rw-r--r-- 1 root root 0 Dec 9 17:25 TOP_USAGE.csv.05-25-PM_09-12-2015
-rw-r--r-- 1 root root 0 Dec 9 17:30 TOP_USAGE.csv.05-30-PM_09-12-2015
-rw-r--r-- 1 root root 0 Dec 9 17:35 TOP_USAGE.csv.05-35-PM_09-12-2015
Hence i made a very small (1 line) shell script for this, so that i can run in every 5 minutes via cronjob.
Problem is when i run this script manually then i can see the output in the file, however when this script in running automatically, file is generating in every 5 minutes but there is no data (aka file is empty)
Can anyone please help me on this?
I now modified the script and still it's the same.
#!/bin/sh
PATH=$(/usr/bin/getconf PATH)
/usr/bin/top -b -n 1 > /var/tmp/TOP_USAGE.csv.$(date +"%I-%M-%p_%d-%m-%Y")
I met the same problem as you.
Top command with -b option must be added.Saving top output to variable before we use it.
the scripts are below
date >> /tmp/mysql-mem-moniter.log
MEM=/usr/bin/top -b -n 1 -u mysql
echo "$MEM" | grep mysql >> /tmp/mysql-mem-moniter.log
Most likely the environment passed to your script from cron is too minimal. In particular, PATH may not be what you think it is (no profiles are read by scripts started from cron).
Place PATH=$(/usr/bin/getconf PATH) at the start of your script, then run it with
/usr/bin/env -i /path/to/script
Once that works without error, it's ready for cron.
Background
I have a test script that creates and destroys file systems on the fly, used in a suite of performance tests.
To avoid running the script as root, I have a disk device /dev/testdisk that is owned by a specific user testuser, along with a suitable entry in /etc/fstab:
$ ls -l /dev/testdisk
crw-rw---- 1 testuser testuser 21, 1 Jun 25 12:34 /dev/testdisk
$ grep testdisk /etc/fstab
/dev/testdisk /mnt/testdisk auto noauto,user,rw 0 0
This allows the disk to be mounted and unmounted by a normal user.
Question
I'd like my script (which runs as testuser) to programmatically create a btrfs file system on /dev/testdisk such that the root directory is owned by testuser:
$ mount /dev/testdisk /mnt/testdisk
$ ls -la /mnt/testdisk
total 24
drwxr-xr-x 3 testuser testuser 4096 Jun 25 15:15 .
drwxr-xr-x 3 root root 4096 Jun 23 17:41 ..
drwx------ 2 root root 16384 Jun 25 15:15 lost+found
Can this be done without running the script as root, and without resorting to privilege escalation (use of sudo) within the script?
Comparison to other file systems
With ext{2,3,4} it's possible to create a filesystem whose root directory is owned by the current user, with the following command:
mkfs.ext{2,3,4} -F -E root_owner /dev/testdisk
Workarounds I'd like to avoid (if possible)
I'm aware that I can use the btrfs-convert tool to convert an existing (possibly empty) ext{2,3,4} file system to btrfs format. I could use this workaround in my script (by first creating an ext4 filesystem and then immediately converting it to brtfs) but I'd rather avoid it if there's a way to create the btrfs file system directly.
I have a script in /etc/cron.daily/backup.sh file is allowed to execute and run but do not start happening, I read the manual and used the search but not mastered decision.
ls -l /etc/cron.daily/
total 52
-rwxr-xr-x 1 root root 8686 2009-04-17 10:27 apt
-rwxr-xr-x 1 root root 314 2009-02-10 19:45 aptitude
-rwxr-xr-x 1 root root 103 2011-05-22 19:08 backup.sh
-rwxr-xr-x 1 root root 502 2008-11-05 03:43 bsdmainutils
-rwxr-xr-x 1 root root 89 2009-01-27 00:55 logrotate
-rwxr-xr-x 1 root root 954 2009-03-19 16:17 man-db
-rwxr-xr-x 1 root root 646 2008-11-05 03:37 mlocate
The cron job filename can't have a period in it on certain ubuntus. See this. Particularly, this quote within:
Although the directories contain periods in their names, run-parts
will not accept a file name containing a period and will fail silently
when encountering them
Properly, this is a problem with run-parts, which the ubuntu cron runs, and not with cron itself. Still, it's what bit me.
pls check:
1a) is the script executable and has correct owner/group settings?
1b) does it start with the correct #! line? and do you specify the full path to the shell you're using,
e.g. #!/bin/bash ?
2) does the script generate an error while being executed?
e.g. can you write to a log file from it, and do you see the log messages?
also: check in the email inbox of the user who owns the crontab -- errors are emailed to the user / e.g. root
what does the output of ls -l /etc/cron.daily/ look like? can you post that?
NOTE:
you can always create a crontab entry for this yourself, outside of those cron.xxx directory structure ;-)
See: man 5 crontab
10 1 * * * /somewhere/backup.sh >> /somewhere/backup.log 2>&1
this has the advantage that you get to pick the exact time when it runs (e.g. 1:10am here), and you can redirect STRERR and STDOUT to append to a log file for that particular script
For testing purposes you could run it ever 10 minutes, like this:
0,10,20,30,40,50 * * * * /somewhere/backup.sh >> /somewhere/backup.log 2>&1
do touch /somewhere/backup.log to make sure it exists