We have an application on Linux that used the syslog mechanism. After a week spent trying to figure out why this application was running slower than expected, we discovered that if we eliminated syslog, and just wrote directly to a log file, performance improved dramatically.
I understand why syslog is slower than direct file writes. But I was wondering: Are there ways to configure syslog to optimize its performance?
You can configure syslogd (and rsyslog at least) not to sync the log files after a log message by prepending a "-" to the log file path in the configuration file. This speeds up performance at the expense of the danger that log messages could be lost in a crash.
There are several options to improve syslog performance:
Optimizing out calls with a macro
int LogMask = LOG_UPTO(LOG_WARNING);
#define syslog(a, ...) if ((a) & LogMask ) syslog((a), __VA_ARGS__)
int main(int argc, char **argv)
{
LogMask = setlogmask(LOG_UPTO(LOG_WARNING));
...
}
An advantage of using a macro to filter syslog calls is that the entire call is
reduced to a conditional jump on a global variable, very helpful if you happen to
have DEBUG calls which are translating large datasets through other functions.
setlogmask()
setlogmask(LOG_UPTO(LOG_LEVEL))
setlogmask() will optimize the call by not logging to /dev/log, but the program will
still call the functions used as arguments.
filtering with syslog.conf
*.err /var/log/messages
"check out the man page for syslog.conf for details."
configure syslog to do asynchronous or buffered logging
metalog used to buffer log output and flushed it in blocks. stock syslog and syslog-ng
do not do this as far as I know.
Before embarking in new daemon writing you can check if syslog-ng is faster (or can be configured to be faster) than plain old syslog.
One trick you can use if you control the source to the logging application is to mask out the log level you want in the app itself, instead of in syslog.conf. I did this years ago with an app that generated a huge, huge, huge amount of debug logs. Rather than remove the calls from the production code, we just masked so that debug level calls never got sent to the daemon. I actually found the code, it's Perl but it's just a front to the setlogmask(3) call.
use Sys::Syslog;
# Start system logging
# setlogmask controls what levels we're going to let get through. If we mask
# them off here, then the syslog daemon doesn't need to be concerned by them
# 1 = emerg
# 2 = alert
# 4 = crit
# 8 = err
# 16 = warning
# 32 = notice
# 64 = info
# 128 = debug
Sys::Syslog::setlogsock('unix');
openlog($myname,'pid,cons,nowait','mail');
setlogmask(127); # allow everything but debug
#setlogmask(255); # everything
syslog('debug',"syslog opened");
Not sure why I used decimal instead of a bitmask... shrug
Write your own syslog implementation. :-P
This can be accomplished in two ways.
Write your own LD_PRELOAD hook to override the syslog functions, and make them output to stderr instead. I actually wrote a post about this many years ago: http://marc.info/?m=97175526803720 :-P
Write your own syslog daemon. It's just a simple matter of grabbing datagrams out of /dev/log! :-P
Okay, okay, so these are both facetious answers. Have you profiled syslogd to see where it's choking up most?
You may configure syslogd's level (or facility) to log asynchronously, by putting a minus before path to logfile (ie.: user.* [tab] -/var/log/user.log).
Cheers.
The syslog-async() implementation may help, at the risk of lost log lines / bounded delays at other times.
http://thekelleys.org.uk/syslog-async/
Note: 'asynchronous' here refers to queueing log events within your application, and not the asynchronous syslogd output file configuration option that other answers refer to.
Related
in rsyslog configuration file we configured like all application logs are to be write in /var/log/messages but the logs get written at very high rate, how can i decrease the level of logging at application level
Hope this is what you are looking for.
Open the file in a text editor:
/etc/rsyslog.conf
change the following parameter to what you think is good for you:
$SystemLogRateLimitInterval 3
$SystemLogRateLimitBurst 40
restart rsyslogd
service rsyslog restart
$InputFilePollInterval equivalent to: “PollingInterval”
PollingInterval seconds
Default: 10
This setting specifies how often files are to be polled for new data.
The time specified is in seconds. During each polling interval, all
files are processed in a round-robin fashion.
A short poll interval provides more rapid message forwarding, but
requires more system resources. While it is possible, we stongly
recommend not to set the polling interval to 0 seconds
.
There are a few approaches to this, and it depends on what exactly you're looking to do, but you'll likely want to look into separating your facilities into separate output files, based on severity. This can be done using RFC5424 severity priority levels in your configuration file.
By splitting logging into separate files by facility and/or severity, and setting the stop option, messages based on severity can be output to as many or few files as you like.
Example (set in the rsyslog.conf file):
*.*;auth,authpriv,kern.none /var/log/syslog
kern.* /var/log/kern.log
kern.debug stop
*.=debug;\
auth,authpriv.none;\
news.none;mail.none /var/log/debug
This configuration:
Will not output output any kern facility messages to syslog (due to kern.none)
Will output all debug level logging of kern to kern.log and "stop" there
Will output any other debug logs that are not excluded by .none to debug
How you separate things out is up to you, but I would recommend looking over the first link I included. You also may want to look into the different local facilities that can be used as separate log pipelines.
So the syslog log is made up of a header (timestamp + hostname) and a message (tag + content). Is any of this customizable? I mean how can i decide the format of timestamp or whether I want hostname to be logged.
Is there someway to do this from the syslog system call or syslog.conf file?
UPDATE:
I'm using syslogd and FREEBSD8
It's pretty much hardcoded for security reasons (you don't want faked messages which look like they came from host X at time T but came from x at t).
If you want to mess with it, you'd have to hack usr.sbin/syslogd/syslogd.c, functions printline() and logmsg().
Syslog's date format, limited as it is, is defined in RFC 5424, which FreeBSD's built-in Syslog follows accurately.
If you want more flexibility or detail, you'll either need to post-process your logs, or switch to a different syslog daemon. Note that by doing so, you may make it impossible for standard log analysis tools to interpret your logs.
One favourite alternative is syslog-ng, which is available in the ports tree.
cd /usr/ports/sysutils/syslog-ng && make install clean
I have two user processes A and B. Both use syslog using facility LOG_USER.
I want to have different threshold levels for them:
For A, only messages of priority ERR-and-above must be logged
For B, only messages of priority CRIT-and-above must be logged
I found that if I setup /etc/syslog.conf as
user.err /var/log/messages
then messages of ERR-and-above are logged, but, from both A and B.
How can I have different minimum threshold levels for different processes?
Note: I am exploring if there is a config file based solution. Otherwise, there is another approach that works. In each process, we can use setlogmask() to install process specific priority mask.
EDIT (Nov 18): I want to use syslog and some portable solution.
A config file based solution is available. I think CentOS by default ships with rsyslog and even if it does not, you can always install rsyslog with yum. This solution works only with rsyslog and nothing else.
The is a catch, though. You can not separate log messages with rsyslog (or pretty much any syslog daemon implementation) between processes with same name ie. the same executable path. However, rsyslog does allow you to filter messages based on program name. Here lies a possible solution: most programs call openlog(3) using argv[0], ie. the executable name, as the first argument. Now since you don't reveal the actual program you're running, there is no way to determine this for you, but you can always read the sources of your own program, I guess.
In most cases the executable path is the program name, though some daemons do fiddle with argv[0] (notable examples are postfix and sendmail). Rsyslog on the other hand provides a filtering mechanism which allows one to filter messages based on the name of the sending program (you can now probably see how this is all connected to how openlog(3) is called). So, instead of trying to filter directly processes, we can do filtering on program names. And that we can affect by creating symbolic links.
So, this solution only works given following conditions: a) the process you're running does not fiddle with argv[0] after beginning execution; b) you can create symlinks to the binary, thus creating two different names for the same program; c) your program is calling openlog(3) using argv[0] as the first parameter to the call.
Given those two conditions, you can simply filter messages on /etc/rsyslog.conf like this (example directly from rsyslog documentation):
if $programname == 'prog1' then {
action(type="omfile" file="/var/log/prog1.log")
}
if $programname == 'prog2' then {
action(type="omfile" file="/var/log/prog2.log")
}
E.g. if your program is called /usr/bin/foobar and you've created symbolic links /usr/bin/prog1 and /usr/bin/prog2 both pointing at /usr/bin/foobar, the above configuration file example will then direct messages from processes started as "prog1" and "prog2" to different log files respectively. This example will not fiddle with anything else, so all those messages are still going to general log files, unless you filter them out explicitly.
This tutorial http://www.freebsd.org/cgi/man.cgi?query=syslog.conf&sektion=5 helped me. The following seem to work:
# process A: log only error and above
!A
*.err /var/log/messages
# process B: log only critical and above
!B
*.critical /var/log/messages
# all processes other than A and B: log only info and above
!-A,B
*.info /var/log/messages
I am debugging a driver for linux (specifically ubuntu server 9.04), and there are several printf statements in the code.
Where can I view the output of these statements?
EDIT1: What i'm trying to do is write to kernel using the proc file-system.
The print code is
static int proc_fractel_config_write(struct file *file, const char *argbuf, unsigned long count, void *data)
{
printk(KERN_DEBUG "writing fractel config\n");
...
In kern.log, I see the following message when i try to overwrite the file /proc/net/madwifi/ath1/fractel_config (with varying time of course).
[ 8671.924873] proc write
[ 8671.924919]
Any explainations?
Many times KERN_DEBUG level messages are filtered and you need to explicitly increase the logging level. You can see what the system defaults are by examining /proc/sys/kernel/printk. For example, on my system:
# cat /proc/sys/kernel/printk
4 4 1 7
the first number shows the console log level is KERN_WARNING (see proc(5) man pages for more information). This means KERN_NOTICE, KERN_INFO, and KERN_DEBUG messages will be filtered from the console. To increase the logging level or verbosity, use dmesg
$ sudo dmesg -n 7
$ cat /proc/sys/kernel/printk
7 4 1 7
Here, setting the level to 7 (KERN_DEBUG) will allow all levels of messages to appear on the console. To automate this, add loglevel=N to the kernel boot parameters where N is the log level you want going to the console or ignore_loglevel to print all kernel messages to the console.
It depends on the distribution, but many use klogd(8) to get the messages from the kernel and will either log them to a file (sometimes /var/log/dmesg or /var/log/kernel) or to the system log via syslog(3). In the latter case, where the log entries end up will depend on the configuration of syslogd(8).
One note about the dmesg command: Kernel messages are stored in a circular buffer, so large amounts of output will be overwritten.
You'll get the output with the command dmesg
dmesg outputs all the messages from the kernel. Finding your desired messages would be difficult. Better use dmesg and grep combination and use a driver specific label in all your printk messages. That will ease in eliminating all the unwanted messages.
printk("test: hello world")
dmesg | grep test
I had this problem on Ubuntu 11.10 and 10.04 LTS, on the former I edited /etc/rsyslog.d/50-default.conf, then restarted rsyslog using "sudo service rsyslog restart" to restart rsyslogd. Then it worked.
Note that Ubuntu uses *r*syslogd, not syslogd.
You might try a higher level than KERN_DEBUG, for example KERN_INFO. Depending on your configuration the lowest priority messages might not be displayed.
In centos (Atleast in centos 6.6) the output will be in /var/log/messages
Let's say I have 10 programs (in terminals) working in tandem: {p1,p2,p3,...,p10}.
It's hard to keep track of all STDOUT debug statements in their respective terminal. I plan to create a GUI to keep track of each STDOUT such that, if I do:
-- Click on p1 would "tail" program 1's output.
-- Click on p3 would "tail" program 4's output.
It's a decent approach but there may be better ideas out there? It's just overwhelming to have 10 terminals; I'd rather have 1 super terminal that keeps track of this.
And unfortunately, linux "screen" is not an option. RESTRICTIONS: I only have the ability to either: redirect STDOUT to a file. (or read directly from STDOUT).
If you are looking for a creative alternative, I would suggest that you look at sockets.
If each program writes to the socket (rather than STDOUT), then your master terminal can act as a server and organize the output.
Now from what you described, it seems as though you are relatively constrained to STDOUT, however it could be possible to do something like this:
# (use netcat (or nc on some systems) to write to a socket on the provided port)
./prog1 | netcat localhost 12312
I'm not sure if this fits in the requirements of what you are doing (and it might be more effort than it is worth!), but it could provide a very stable solution.
EDIT: As was pointed out in the comments, netcat does exactly what you would need to make this work.