rsyslog.conf file stopped receiving logs - linux

Question

I am working on a school project to get logs from a printer sent to splunk. This is a project done by a few batches of students. I initially used the original copy of the rsyslog.conf file done by the previous batch student but was unable to retrieve data in Splunk. I had already set the printer to send logs to the IP address and port of the linux server I am using. Configuration in Splunk has also been made to listen to port 2048. I was told by my supervisor that the copy done by the previous student should work.
Original Copy:
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 2048
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
$template RemoteLogs,"/var/log/syslog/%HOSTNAME%/%FROMHOST-IP%.log"
*.* ?RemoteLogs
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually
not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* ##remote-host:514
# ### end of the forwarding rule ###
As I was unable to get the printer to send logs to Splunk with the above configuration, I went to research online and made a few modifications which I managed to get the printer to send logs to Splunk.
Modified Copy:
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 2048
*$InputUDPServer BindRuleset remote
$UDPServerRun 2048*
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
$template RemoteLogs,"/var/log/syslog/%HOSTNAME%/%FROMHOST-IP%.log"
*.* ?RemoteLogs
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually
not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
**.* action(type="omfwd" target="172.**.***.156" port="2048" protocol="udp"
action.resumeRetryCount="100"
queue.type="linkedList" queue.size="10000")*
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
*remote host is: 172.**.***.43:2048
*.* ##remote-host:514*
# ### end of the forwarding rule ###
I was able to receive logs from the printer to Splunk after I had made the above modifications. However, after a few days when I tried to generate logs from the printer I no longer receive new logs in Splunk. I did not make any changes to the modified copy of the rsyslog.conf file that I have been using. Hence I don't get why Splunk no longer receives logs from the printer.
I changed the configuration to another port number but still did not receive new logs from the printer, so I changed back to port 2048. I also double checked that I had made the corresponding changes in the printer's EWS and Splunk correctly. I have been researching on this and double checking for the whole day but to no avail.
Note:
-2048 is the port used to retrieve logs from the printer
-172.** . ***.43 is the IP address of the printer (it represents the actual IP address, was told by another user to censor it, i did not put astrids as part of my IP address in the real config file)
-172.** . ***.156 is the IP address of the Linux server I am using (it represents the actual IP address, was told by another user to censor it, i did not put astrids as part of my IP address in the real config file)
-I am using the Linux server as my main workpoint, in the modified copy I added a paragraph with the destination stating my current IP address (linux server), is it the cause why the printer stopped sending logs to my server since I have been directing the logs to "myself"-config done using the same Linux server as destination host-also Linux server)
-The text with astrids are the ones I added to the original copy
-EWS stands for embedded web service, basically a web portal of the printer.
-How I trigger logs from printer: logging in to the EWS/making failed logins. Details of logins will be sent to Splunk

Answer 1

You need to decouple rsyslog and splunk and focus on one at a time.
Assuming:
Printer is 172.x.x.43 and configured to send syslog over udp 2048 to 172.x.x.156.
Rsyslog is running on 172.x.x.156 an listening on udp 2048.
Verify you are able to generate the syslog messages from the printer correctly.
(from 172.x.x.156 command line)
tcpdump -nn | grep 2048
You should at least see the traffic coming into the system.
I would modify the rsyslog.conf to only log the printer information to a file and then throw it away. You are currently putting everything into a file and forwarding everything to yourself but on a port you aren't listening too...
something like:
if ($fromhost-ip == '172.x.x.43') then {
action(type="omfile" file="/var/log/remotehosts/printers/%fromhost-ip%.log")
stop
}
test rsyslog config for errors:
rsyslogd -N1
If good restart rsyslogd.
Then you should configure splunk forwarder to consume the file.
inputs.conf
[monitor:///var/log/remotehosts/printers/*.log]
sourcetype = printer # or whatever