Sending user specific logs from one host to other using rsyslog - linux

I am using rsyslog server running on localhost(centos) and remote machine(ubuntu).I am able to send the logs from localhost to remote server using TCP connection and UDP connection able to see the logs in remote server.
My localhost config :
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
*.* ##192.168.122.50:514
My remote server config:
/etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
$ModLoad imtcp
$InputTCPServerRun 514
$AllowedSender TCP, 192.168.0.0/8
#$ModLoad immark # provides --MARK-- message capability
$template TmplAuth, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.* ?TmplAuth
*.info;mail.none;authpriv.none;cron.none ?TmplMsg
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
authpriv.* /var/log/secgw-siglogs;RSYSLOG_FileFormat
$template RemoteStore, "/var/log/remote/%HOSTNAME%/%timegenerated:1:10:date-rfc3339%"
:source, !isequal, "localhost" -?RemoteStore
:source, isequal, "last" ~$template RemoteStore, "/var/log/remote/%HOSTNAME%/%timegenerated:1:10:date-rfc3339%"
:source, !isequal, "localhost" -?RemoteStore
:source, isequal, "last" ~
Now i have to send some user specific logs not all kernel logs,auth logs which are present in /var/log location,Is there any configuration need to be modified ?

Related

Issue with multiple logs files forwarding with rsyslog?

I have a use case, where I need to forward multiple log files to remote server.
//Log Files
1. /var/log/boot.log
2. /path/to/tomcat/tomcat.log
3. /path/to/app/app.log
I have added 3 conf files in /etc/rsyslog.d/
#/etc/rsyslog.d/boot.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /var/log/boot.log
$InputFileTag boot-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
#/etc/rsyslog.d/tomcat.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /path/to/tomcat.log
$InputFileTag tomcat-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
#/etc/rsyslog.d/app.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /path/to/app.log
$InputFileTag app-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
To include these file I've added this line $IncludeConfig /etc/rsyslog.d/*.conf.
but in /var/log/syslog just boot logs are adding up rest of the configs are getting ignored.
RsysLog Version: 8.1907.0 (aka 2019.07)
Maybe the answer should be in the rsyslog's documentation:
$ModLoad imfile # needs to be done just once
# File 1
$InputFileName /path/to/file1
$InputFileTag tag1:
$InputFileStateFile stat-file1
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor
# File 2
$InputFileName /path/to/file2
$InputFileTag tag2:
$InputFileStateFile stat-file2
$InputRunFileMonitor
# ... and so on ...
# check for new lines every 10 seconds $InputFilePollingInterval 10
In that way, you have re-defined the module load itself, that should be defined before the file includes.

remote logging using rsyslog server is not working

I have 2 linux machines, both of them have rsyslog. I need to send logs from client machine to server machine. I have done these steps but still I am not able to send the message. What am I doing wrong? I am using UDP port 514
SERVER MACHINE : (192.16.72.239)
updated /etc/rsyslog.conf, uncommented 2 lines after comment
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
`restarted rsyslog`
executed below commands as well:
[root#mysystem/]# iptables -A INPUT -p udp --dport 514 -j ACCEPT
[root#mysystem/]#nc -l -p 514 -4
Ncat: bind to 0.0.0.0:514: Address already in use. QUITTING.
CLIENT MACHINE :
added this rule(third line) at the end of the file:
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* ##remote-host:514
*.* ##192.16.72.239:514
# ### end of the forwarding rule ###
now I am using command on client machine logger "sending message from client to server". But I can see above message is present in client's /var/log/messages but it is not present in server's /var/log/messages.
Am I doing anything wrong?

rsyslog doesnt send any information

Good day, everyone!
I'm trying to send log files from one Red Hat based server to another Graylog server by using rsyslog.
So I cant do it, cause rsyslog doesn't send anything.
I would really appreciate if someone helps!
Graylog recieves messages by:
echo "Hello Graylog, let's be friends." | nc -w 1 -u "my-graylog-ip" 13101
I had configured rsyslog.config:
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* #http://"my-graylog-ip":13101
# ### end of the forwarding rule ###
then restarted rsyslog
systemctl restart rsyslog
in /etc/rsyslog.d/ located grid.conf:
# File 1
$ModLoad imfile
$InputFileName /home/ucp/current-envelope/log/envelope.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
# File 2
ModLoad imfile
$InptFileName /home/ucp/current-envelope/log/envelope-err.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
# File 3
$ModLoad imfile
$InputFileName /home/ucp/current-envelope/log/performance.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
journalctl -b | grep rsyslog shows next messages:
Mar 23 10:24:01 grid101 kernel: type=1130 audit(1490253841.179:135915): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 23 10:45:02 grid101 kernel: type=1131 audit(1490255102.506:135925): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 23 10:45:02 grid101 kernel: type=1130 audit(1490255102.512:135926): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
The IP:port configuration is not right. It must be changed to something like this for TCP:
*.* ##my-graylog-ip:13101
I wonder why your Graylog server is using the 13101 TCP port. I guess, it is behind a NAT or something like that. I also wonder why you want to use the http:// thing in your configuration.

What is the criteria for rsyslogd to create symbolic link to my own file similar to /var/log/messages

When I check the file descriptors opened by rsyslogd, i see the process have created symbolic link to /var/log/messages and similar files,
root#blr09> ll /proc/16635/fd
total 0
lr-x------. 1 root root 64 Jan 4 08:29 0 -> /dev/null
l-wx------. 1 root root 64 Jan 4 08:29 1 -> /dev/null
l-wx------. 1 root root 64 Jan 4 08:29 10 -> **/var/log/authlog**
lr-x------. 1 root root 64 Jan 4 08:29 11 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system.journal
l-wx------. 1 root root 64 Jan 4 08:29 2 -> /dev/null
lrwx------. 1 root root 64 Jan 4 08:29 3 -> socket:[4422054]
l-wx------. 1 root root 64 Jan 4 08:29 4 -> **/var/log/messages**
lr-x------. 1 root root 64 Jan 4 08:29 5 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system#3a558a8cce7b45a6bf810fe33c7a89d6-0000000000011e3c-0005453c395dc7aa.journal
lr-x------. 1 root root 64 Jan 4 08:29 6 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system#3a558a8cce7b45a6bf810fe33c7a89d6-0000000000000001-00054520bb84848d.journal
lr-x------. 1 root root 64 Jan 4 08:29 7 -> anon_inode:inotify
l-wx------. 1 root root 64 Jan 4 08:29 8 -> **/var/log/secure**
l-wx------. 1 root root 64 Jan 4 08:29 9 -> **/var/log/cron**
root#blr09>
Below is rsyslog.conf
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* ##remote-host:514
# ### end of the forwarding rule ###
auth.info /var/log/authlog
*.* #127.0.0.1:10514
My requirement is to create a similar file of /var/log/messages for some application messages so that rsyslog will send all the messages to port 10514 as configured in rsyslog.conf.
Could you please let me know how this can be achieved.

rsyslog - Property-based filtering not working

I almost hate to submit a topic for this, but I haven't been able to figure it out on my own. I'm running a Federoa 17 server, and I'm attempting to log dropped packets from iptables to a separate log file via rsyslog, but it keeps sending them to /var/log/messages instead.
Snippet from my firewall script:
#!/bin/bash
iptables -F
# My accepted rules would be here
iptables -A INPUT -j LOG --log-prefix "iptables: "
iptables -A FORWARD -j LOG --log-prefix "iptables: "
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables-save > /etc/sysconfig/iptables
service iptables restart
iptables -L -v
The config file that SHOULD be catching the messages from iptables:
[root#fc17 ]# cat /etc/rsyslog.d/iptables.conf
:msg, startswith, "iptables: " /var/log/iptables.log
& ~
Snippet from my rsyslog.conf file:
#### GLOBAL DIRECTIVES ####
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
# I put this in here too to see if it would work; it doesn't
:msg, startswith, "iptables: " /var/log/iptables.log
& ~
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
I've restarted both iptables and rsyslog multiple times since making the changes, and no matter what, it will only only log dropped packets from iptables to /var/log/messages.
I heard running rsyslog in compatibility mode can cause various problems. Could this be the case here? Here are its run-options on my system:
[root#fc17 ]# ps -ef | grep rsyslog
root 3571 1 0 00:59 ? 00:00:00 /sbin/rsyslogd -n -c 5
startswith comparison operator didn't work,because msg didn't begin with iptables: when i checked my logs.
[root#localhost ~]# cat /etc/rsyslog.d/test.conf
:msg, startswith, "iptables:" /var/log/iptables.log
but contains comparison operator worked on my FC18
[root#localhost ~]# cat /etc/rsyslog.d/test.conf
:msg, contains, "iptables:" /var/log/iptables.log
Ref: Rsyslog site
you should add the following two line in your "/etc/rsyslogd.conf" in directives part
$klogParseKernelTimestamp on
$klogKeepKernelTimestamp off
This will remove the kernel timestamp which appears in the begining of every kernel message like "[6448.546951]" in the following log
Mar 31 14:36:14 localhost kernel: [ 6448.546951] iptables: IN=ppp0 OUT= MAC= SRC=
2019 solution. Tested with rsyslogd 8.32.0 on Ubuntu18.04.
You can still use startswith,
[root#localhost ~]# cat /etc/rsyslog.d/test.conf
:msg, startswith, " iptables:" /var/log/iptables.log
by changing the line in /etc/rsyslogd.conf
module(load="imklog" ParseKernelTimestamp="on" KeepKernelTimestamp="off")
I'm using rsyslogd 5.8.10 over centos 6, my log report show this way:
Aug 12 11:50:41 node2 kernel: [10256396.525411] IPTables-Dropped: IN=eth0 OUT= MAC=00:25:90:c3:05:40:00:24:13:10:8c:00:08:00 SRC=212.237.40.56 DST=37.153.1.29 LEN=45 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=UDP SPT=45661 DPT=53413 LEN=25
I tried to disabled the timestamp with:
$klogParseKernelTimestamp on
$klogKeepKernelTimestamp off
But show:
Aug 12 11:50:22 node2 rsyslogd-3003: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]
In modules have this:
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
Thank you advance.

Resources