I have different logs that are written to our moutend nfs share that i need to send to our syslog-server (graylog) they are located outside /var/log folder.
So i add some extra conf in /etc/rsyslog.d/
For this example i have two files with following config:
atlassian-application-confluence-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/confluence/logs")
# This is the main application log file
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence.log"
Tag="atlassian"
PersistStateInterval="200"
)
# This file contains entries related to the search index.
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence-index.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
atlassian-application-jira-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/jira/log")
# Contains logging for most of Jira, including logs that aren’t specifically written elsewhere
input(type="imfile"
File="/atlassian/test/jira/log/atlassian-jira.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
So to my problem.
When i check the Rsyslogd configuration with following command:
rsyslogd -N1 -f /etc/rsyslog.d/atlassian-application-confluence-log.conf
It says it is valid.
When i restart the rsyslog service i get the following errors:
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workdirectory' specified more than once - one instance is ignored. Fix config [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]>
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workDirectory' not known -- typo in config file? [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
[origin software="rsyslogd" swVersion="8.2102.0-10.el8" x-pid="379288" x-info="https://www.rsyslog.com"] start
imjournal: journal files changed, reloading... [v8.2102.0-10.el8 try https://www.rsyslog.com/e/0 ]
How can i get rid of the warnings?
I have already tried to put the two modules in /etc/rsyslog.conf
I get following errors from that config:
parameter 'PersistStateInterval' not known
parameter 'Tag' not known
parameter 'File' not known
If there are multiple configuration files, they are processed in ascending sort order of the file name (numerically/alphabetically), See: $IncludeConfig.
Therefore you don't have to include any configuration parameters (modules, work directories, rulesets etc.) multiple times. You can include them once in the config which is loaded first.
Related
I'm upgrading from SoapUI 5.4.0 to 5.7.0 and trying to put the log files in a specific directory. Note: The alternate error logs directory was working prior to the upgrade.
I have both the following specified in my JAVA_OPTS for SoapUITestCaseRunner
-Dsoapui.logroot="%SOAPUI_LOGSDIR%"
-Dsoapui.log4j.config="%SOAPUI_HOME%/soapui-log4j.xml"
In my soapui-log4j.xml I specify the error file as:
<RollingFile name="ERRORFILE"
fileName="${soapui.logroot}/soapui-errors.log"
filePattern="${soapui.logroot}/soapui-errors.log.%i"
append="true">
The error file then gets created without resolving ${soapui.logroot} e.g.
$ find . -name "*errors*"
./${soapui.logroot}/soapui-errors.log
I also tried it as lookup but ended up with this:
ERROR Unable to create file ${sys:soapui.logroot}/soapui-errors.log java.io.IOException: The filename, directory name, or volume label syntax is incorrect
Am I missing anything? Any ideas for next steps?
I tried replacing
fileName="${soapui.logroot}/soapui-errors.log"
with
fileName="${sys:soapui.logroot}/soapui-errors.log"
and it worked for me.
I no longer see unresolved '${soapui.logroot}' directory created.
A
I have a folder with log files from 2016-present and setup filebeat with "ignore_older: 48h". All the files get rotated so that "log" is always the new one, "log.1" is the next etc.
Logs are on linux NFS partition mounted on the logstash host.
I expect filebeat to get only log files that where changed in the last 24h and ignore the older ones.
The above happens except from time to time it also gets older files in no specific order.
I ran "stat" command on one of the older file from 2018 and i see the following:
Access: 2019-03-02 03:15:32.254460960 +0000
Modify: 2018-09-06 13:12:00.331460890 +0000
Change: 2019-02-28 03:34:33.946462475 +0000
I run filebeat version 6.4.2
Is this data confusing Logstash? What is it actually looking at when checking if a file has changed. How can i stop it from taking older files.
UPDATE:
My filebeat configuration looks like this:
- type: log
enabled: true
paths:
- /path/to/my/log/file/log*
fields:
logname: "log.name"
include_lines: ["SOME_TEXT"]
ignore_older: 48h
Logs are in CSV format.
On another host i do the same but with logstash directly, the input config is like this:
input {
file {
path => "/path/to/my/log/file/log*"
mode => "tail"
start_position => "beginning"
close_older => "24h"
ignore_older => "2w"
}
}
I have the same issue here.
You can try to do two things, one is to remove the * after log in the path like this
- /path/to/my/log/file/log
Since filebeat will read a rotated log file even after it is moved until it reaches a certain age.
Or for logstash the path parameter is an array and you create a list of files to be read, if you know how often the files get rotated:
path => [ "path/to/my/log/file.log", "/path/to/my/log/file1.log", "path/to/my/log/file2.log"]
I like to configure syslog. It seems that are more than one way to set up syslog. I am asking for the common way/steps to do that.
I have several use cases. To simplify I like to ask how to configure syslog to write an infinity long log file in /var/log/.
Following steps:
1.) configure what messages
1.1) create own "syslog.conf" (define /var/log/myLog)
1.2) append it to "recipes-core/busybox"
2.) configure how to log
??
I found two possible places to do that:
#meta-poky
-> "meta-poky/recipes-core/busybox/busybox/poky-tiny/defconfig"
#
# System Logging Utilities
#
CONFIG_SYSLOGD=y
CONFIG_FEATURE_ROTATE_LOGFILE=y
CONFIG_FEATURE_REMOTE_LOG=y
CONFIG_FEATURE_SYSLOGD_DUP=y
CONFIG_FEATURE_SYSLOGD_CFG=y
CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256
CONFIG_FEATURE_IPC_SYSLOG=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=16
CONFIG_LOGREAD=y
CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y
CONFIG_KLOGD=y
CONFIG_FEATURE_KLOGD_KLOGCTL=y
CONFIG_LOGGER=y
add/change:
"CONFIG_FEATURE_ROTATE_LOGFILE=n" by adding that line to meta-mylayer/conf/layer.conf"
etc.
???
# "/etc/syslog-startup.conf"
# This configuration file is used by the busybox syslog init script,
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.
DESTINATION=file # log destinations (buffer file remote)
LOGFILE=/var/log/messages # where to log (file)
REMOTE=loghost:514 # where to log (syslog remote)
REDUCE=no # reduce-size logging
DROPDUPLICATES=no # whether to drop duplicate log entries
#ROTATESIZE=0 # rotate log if grown beyond X [kByte]
#ROTATEGENS=3 # keep X generations of rotated logs
BUFFERSIZE=64 # size of circular buffer [kByte]
FOREGROUND=no # run in foreground (don't use!)
#LOGLEVEL=5 # local log level (between 1 and 8)
In the systemV init script "/etc/init.d/syslog.bussybox" the file "/etc/syslog-startup.con" is read and used for configuration.
System behaviour:
When running my system, the log wraps when the logfile reaches 200kBytes. One logfile + one log-rotate file is generated.
Any ideas how to archive that syslog writes an infinite long log-file?
I am working on the Yocto krogoth branch + meta-atmel / meta_openembedded (# krogoth too).
By checking the sources of syslog and busybox I found a possible solution. This solution shows how to configure syslog to log in two logs with max 10MByte:
1.) get valid syslog build config
1.1) download busybox -> git/busybox
1.2) build busybox via bitbake -> bitbake busybox
1.3) copy defconfig file to downloaded busybox -> cp /defconfig git/busybox/
1.4) make menueconfig
1.5) goto "System Logging Utilities"
1.6) deselect klogd because it can colide with printk
1.7) save to "defconfig"
#
# System Logging Utilities
#
# CONFIG_KLOGD is not set
# CONFIG_FEATURE_KLOGD_KLOGCTL is not set
CONFIG_LOGGER=y
CONFIG_LOGREAD=y
CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y
CONFIG_SYSLOGD=y
CONFIG_FEATURE_ROTATE_LOGFILE=y
CONFIG_FEATURE_REMOTE_LOG=y
CONFIG_FEATURE_SYSLOGD_DUP=y
CONFIG_FEATURE_SYSLOGD_CFG=y
CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256
CONFIG_FEATURE_IPC_SYSLOG=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=64
CONFIG_FEATURE_KMSG_SYSLOG=y
2.) setup your log config
Create "syslog.conf" and enter the rules:
This is an example:
#
# /etc/syslog.conf Configuration file for busybox's syslogd utility
#
kern.notice /var/log/messages
#
# my software messages
#
user.err /var/log/mySWError
user.* /var/log/mySWFull
local0.* /var/log/mySWFull
local0.err /var/log/mySWError
#
#this prevents from logging to default log file (-O FILE or /var/log/messages)
#
*.* /dev/null
3.) modify configuration for the busybox syslog deamon
This example logs to files which are limited to 10 MBytes. If "ROTATESIZE" is not set syslog set the log filesize automatic to 200 kBytes. The content of "syslog-startup.conf" looks like:
# This configuration file is used by the busybox syslog init script,
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.
DESTINATION=file # log destinations (buffer file remote)
#LOGFILE=/var/log/messages # where to log (file)
REMOTE=loghost:514 # where to log (syslog remote)
REDUCE=no # reduce-size logging
DROPDUPLICATES=no # whether to drop duplicate log entries
ROTATESIZE=10000 # rotate log if grown beyond X [kByte]
#ROTATEGENS=3 # keep X generations of rotated logs
BUFFERSIZE=64 # size of circular buffer [kByte]
FOREGROUND=no # run in foreground (don't use!)
#LOGLEVEL=5 # local log level (between 1 and 8)
4.) get the configuration into yocto build
4.1) create following directory structure in your own layer(meta-custom):
meta-custom/recipes-core/
meta-custom/recipes-core/busybox/
meta-custom/recipes-core/busybox/busybox
4.2) copy into "meta-custom/recipes-core/busybox/busybox":
defconfig
syslog.conf
syslog-startup.conf
4.3) create in "meta-custom/recipes-core/busybox/" "busybox_1.24.1.bbappend". If you using an older/newer version of busybox you need to change the "1.24.1" number to yours. You can find your version in "/poky/meta/recipes-core/busybox/"
Add this two lines to this file:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}/poky-tiny:"
5.) build your custom defined syslog
bitbake busybox
and transfer it into the rootfs of the image
bitbake core-image-minimal
Now it should work!
Adding to the Stefan Jaritz comment, you can skip the download step by just issuing
bitbake busybox -c devshell
Then running make menuconfig and getting the new config file from it. Note that this will use Yocto's defconfig by default, so you don't need to do worry about "what's this default config".
When I try to create a JSCS config file:
C:\Blog\BlogWeb>jscs --auto-configure "C:\Blog\BlogWeb\temp.jscs"
I get the following error:
safeContextKeyword option requires string or array value
What parameter am I supposed to pass? What is a safecontextkeyword?
New to NPM and JSCS, please excuse ignorance.
JSCS was complaining that I didn't have a config file, so I was trying to figure out how to create one.
JSCS looks for a config file in these places, unless you manually specify it with the --config option:
jscs it will consequentially search for jscsConfig option in package.json file then for .jscsrc (which is a just JSON with comments) and .jscs.json files in the current working directory then in nearest ancestor until it hits the system root.
I fixed this by:
Create a new file named .jscsrc. Windows Explorer may not let you do this, so may need to use the command line.
Copy the following into it. It doesn't matter if this is the preset you want to use or not. The command will overwrite it.
{
"preset": "jquery",
"requireCurlyBraces": null // or false
}
Verify that it works by running a command such as:
run the command
jscs --auto-configure .jscsrc
I would like to parse a directory of logs files with logstash.
When the logs are formatted like this :
server-20140604.log
server-20140603.log
server-20140602.log
There is no problem, I am using globs like this :
input {
file {
path=>["D:/*.log"]
}
}
But my logs are formatted like this :
server.log
server.log.1
server.log.2
client.log
client.log.1
client.log.2
So I would like to know how to tell to logstash to parse in the folder all the files starting with "server" expression in their names. I really need to do it like that, because I have other files in the folder (i.e client logs) that I don't want to parse but also cannot remove from the folder.
With this configuration I can only parse all the log files start with prefix server.
input {
file {
path => ["D:/server*"]
}
}
output {
stdout {
codec => rubydebug
}
}
I think the possible problem you have meet is the start_position config. It means that where does logstash start to read the logs. Please refer to here. Remember this option only modifies first contact situations where a file is new and not seen before. If a file has already been seen before, this option has no effect.
When you stop logstash, logstash will save a .sincedb* in your home directory. Next time you start it, logstash will start read the file according to .sindb*. If you do not input new logs to server.log, logstash will never parse the old logs.
What you can try to do is delete all the .sincedb before you start logstash and add start_posistion to your config. In your comment you have say if you overwrite the server.log logstash can parse the file from beginning, it is because logstash detect it as a new file and the .sincedb* do not save any information about this file. So logstash will parse it! You can try to find out your .sincedb and try to delete it.