Issue with multiple logs files forwarding with rsyslog? - linux

I have a use case, where I need to forward multiple log files to remote server.
//Log Files
1. /var/log/boot.log
2. /path/to/tomcat/tomcat.log
3. /path/to/app/app.log
I have added 3 conf files in /etc/rsyslog.d/
#/etc/rsyslog.d/boot.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /var/log/boot.log
$InputFileTag boot-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
#/etc/rsyslog.d/tomcat.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /path/to/tomcat.log
$InputFileTag tomcat-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
#/etc/rsyslog.d/app.conf
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /path/to/app.log
$InputFileTag app-log
$InputRunFileMonitor
$InputFilePersistStateInterval 100
To include these file I've added this line $IncludeConfig /etc/rsyslog.d/*.conf.
but in /var/log/syslog just boot logs are adding up rest of the configs are getting ignored.
RsysLog Version: 8.1907.0 (aka 2019.07)

Maybe the answer should be in the rsyslog's documentation:
$ModLoad imfile # needs to be done just once
# File 1
$InputFileName /path/to/file1
$InputFileTag tag1:
$InputFileStateFile stat-file1
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor
# File 2
$InputFileName /path/to/file2
$InputFileTag tag2:
$InputFileStateFile stat-file2
$InputRunFileMonitor
# ... and so on ...
# check for new lines every 10 seconds $InputFilePollingInterval 10
In that way, you have re-defined the module load itself, that should be defined before the file includes.

Related

Slurm says drained Low RealMemory

I want to install slurm on localhost. I already installed slurm on similar machine, and it works fine, but on the other machine i got this:
transgen#transgen-4:~/galaxy/tools/melanoma_tools$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
transgen-4-partition* up infinite 1 drain transgen-4
transgen#transgen-4:~/galaxy/tools/melanoma_tools$ sinfo -Nel
Fri Jun 25 17:42:56 2021
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
transgen-4 1 transgen-4-partition* drained 48 1:24:2 541008 0 1 (null) Low RealMemory
transgen#transgen-4:~/galaxy/tools/melanoma_tools$ srun -n8 sleep 10
srun: Required node not available (down, drained or reserved)
srun: job 5 queued and waiting for resources
^Csrun: Job allocation 5 has been revoked
srun: Force Terminated job 5
I found the advice to do so:
sudo scontrol update NodeName=transgen-4 State=DOWN Reason=hung_completing
sudo systemctl restart slurmctld slurmd
sudo scontrol update NodeName=transgen-4 State=RESUME
, but it had no effect.
slurm.conf:
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
SlurmctldHost=localhost
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/spool/slurm.state
SwitchType=switch/none
TaskPlugin=task/cgroup
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
#
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=cluster
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
#SlurmctldDebug=info
#SlurmctldLogFile=
#SlurmdDebug=info
#SlurmdLogFile=
#
#
# COMPUTE NODES
NodeName=transgen-4 NodeAddr=localhost CPUs=48 Sockets=1 CoresPerSocket=24 ThreadsPerCore=2 RealMemory=541008 State=UNKNOWN
PartitionName=transgen-4-partition Nodes=transgen-4 Default=YES MaxTime=INFINITE State=UP
cgroup.conf:
###
# Slurm cgroup support configuration file.
###
CgroupAutomount=yes
CgroupMountpoint=/sys/fs/cgroup
ConstrainCores=no
ConstrainDevices=yes
ConstrainKmemSpace=no #avoid known Kernel issues
ConstrainRAMSpace=no
ConstrainSwapSpace=no
TaskAffinity=no #use task/affinity plugin instead
How can i get slurm working?
Thanks in advance.
This could be that RealMemory=541008 in slurm.conf is too high for your system. Try lowering the value. Lets suppose you have indeed 541 Gb of RAM installed: change it to RealMemory=500000, do a scontrol reconfigure and then a scontrol update nodename=transgen-4 state=resume.
If that works, you could try to raise the value a bit.

slurmctld.service: Can't open PID file No such file or directory

I have the following error message after trying to start slurm on Ubuntu 18.04
slurmctld.service: Can't open PID file /var/run/slurm-llnl/slurmctld.pid (yet?) after start: No such file or directory
here's the ownership of the slurmllnl directory :
drwxr-xr-x 2 slurm slurm 60 juin 22 11:06 slurm-llnl
And in this directory i have slurmd.pid but i don't have slurmctld.pid
Here is my slurm.conf file :
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=daoud
#ControlAddr=
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/linuxproc
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/spool/slurm-llnl
SwitchType=switch/none
TaskPlugin=task/none
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/backfill
#SchedulerPort=7321
SelectType=select/cons_res
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/filetxt
ClusterName=cluster
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
#SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
#SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
#
# COMPUTE NODES
NodeName=daoud CPUs=64 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN
PartitionName=standard Nodes=daoud Default=YES MaxTime=INFINITE State=UP
This is a message issued by systemd, not Slurm, and is caused by using PIDfile in the systemd unit. Slurmctld should keep the Slurmctld from starting.
Newer versions of Slurm switched to Type=simple, therefore not needing a PIDfile anymore

Sending user specific logs from one host to other using rsyslog

I am using rsyslog server running on localhost(centos) and remote machine(ubuntu).I am able to send the logs from localhost to remote server using TCP connection and UDP connection able to see the logs in remote server.
My localhost config :
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
*.* ##192.168.122.50:514
My remote server config:
/etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
$ModLoad imtcp
$InputTCPServerRun 514
$AllowedSender TCP, 192.168.0.0/8
#$ModLoad immark # provides --MARK-- message capability
$template TmplAuth, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.* ?TmplAuth
*.info;mail.none;authpriv.none;cron.none ?TmplMsg
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
authpriv.* /var/log/secgw-siglogs;RSYSLOG_FileFormat
$template RemoteStore, "/var/log/remote/%HOSTNAME%/%timegenerated:1:10:date-rfc3339%"
:source, !isequal, "localhost" -?RemoteStore
:source, isequal, "last" ~$template RemoteStore, "/var/log/remote/%HOSTNAME%/%timegenerated:1:10:date-rfc3339%"
:source, !isequal, "localhost" -?RemoteStore
:source, isequal, "last" ~
Now i have to send some user specific logs not all kernel logs,auth logs which are present in /var/log location,Is there any configuration need to be modified ?

rsyslog doesnt send any information

Good day, everyone!
I'm trying to send log files from one Red Hat based server to another Graylog server by using rsyslog.
So I cant do it, cause rsyslog doesn't send anything.
I would really appreciate if someone helps!
Graylog recieves messages by:
echo "Hello Graylog, let's be friends." | nc -w 1 -u "my-graylog-ip" 13101
I had configured rsyslog.config:
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* #http://"my-graylog-ip":13101
# ### end of the forwarding rule ###
then restarted rsyslog
systemctl restart rsyslog
in /etc/rsyslog.d/ located grid.conf:
# File 1
$ModLoad imfile
$InputFileName /home/ucp/current-envelope/log/envelope.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
# File 2
ModLoad imfile
$InptFileName /home/ucp/current-envelope/log/envelope-err.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
# File 3
$ModLoad imfile
$InputFileName /home/ucp/current-envelope/log/performance.log
$InputFileTag def-grid101-envelope
$InputFileFacility local0
$InputRunFileMonitor
journalctl -b | grep rsyslog shows next messages:
Mar 23 10:24:01 grid101 kernel: type=1130 audit(1490253841.179:135915): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 23 10:45:02 grid101 kernel: type=1131 audit(1490255102.506:135925): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 23 10:45:02 grid101 kernel: type=1130 audit(1490255102.512:135926): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
The IP:port configuration is not right. It must be changed to something like this for TCP:
*.* ##my-graylog-ip:13101
I wonder why your Graylog server is using the 13101 TCP port. I guess, it is behind a NAT or something like that. I also wonder why you want to use the http:// thing in your configuration.

What is the criteria for rsyslogd to create symbolic link to my own file similar to /var/log/messages

When I check the file descriptors opened by rsyslogd, i see the process have created symbolic link to /var/log/messages and similar files,
root#blr09> ll /proc/16635/fd
total 0
lr-x------. 1 root root 64 Jan 4 08:29 0 -> /dev/null
l-wx------. 1 root root 64 Jan 4 08:29 1 -> /dev/null
l-wx------. 1 root root 64 Jan 4 08:29 10 -> **/var/log/authlog**
lr-x------. 1 root root 64 Jan 4 08:29 11 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system.journal
l-wx------. 1 root root 64 Jan 4 08:29 2 -> /dev/null
lrwx------. 1 root root 64 Jan 4 08:29 3 -> socket:[4422054]
l-wx------. 1 root root 64 Jan 4 08:29 4 -> **/var/log/messages**
lr-x------. 1 root root 64 Jan 4 08:29 5 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system#3a558a8cce7b45a6bf810fe33c7a89d6-0000000000011e3c-0005453c395dc7aa.journal
lr-x------. 1 root root 64 Jan 4 08:29 6 -> /run/log/journal/3da3ce2773004947b9a8d40578a1fb8b/system#3a558a8cce7b45a6bf810fe33c7a89d6-0000000000000001-00054520bb84848d.journal
lr-x------. 1 root root 64 Jan 4 08:29 7 -> anon_inode:inotify
l-wx------. 1 root root 64 Jan 4 08:29 8 -> **/var/log/secure**
l-wx------. 1 root root 64 Jan 4 08:29 9 -> **/var/log/cron**
root#blr09>
Below is rsyslog.conf
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* ##remote-host:514
# ### end of the forwarding rule ###
auth.info /var/log/authlog
*.* #127.0.0.1:10514
My requirement is to create a similar file of /var/log/messages for some application messages so that rsyslog will send all the messages to port 10514 as configured in rsyslog.conf.
Could you please let me know how this can be achieved.

Resources