All Logstash errors logs in /var/log/logstash/* , but all the error logs of Logstash logs in /var/log/syslog too.
Is there any way to disable logging errors of Logstash in /var/log/syslog?
I suppose you run logstash from systemd. For unknown reason logstash print it's logs on stdout/stderr, so journald is forwarding console output to syslog.
Redirect stdout/err in systemd unit file like this:
[Service]
Type=simple
Restart=always
WorkingDirectory=/data/logstash
ExecStart=/usr/bin/sudo -u logstashuser bash -c "/opt/logstash/bin/logstash --path.settings /opt/logstash/conf >/dev/null 2>/logs/logstash/logstash.error.log"
then
systemctl daemon-reload
and
systemctl stop logstash.service
systemctl start logstash.service
Another solution is to modify /etc/systemd/system/logstash.service and change those two option as following
StandardOutput=null
StandardError=null
# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
StandardOutput=null
StandardError=null
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity
[Install]
WantedBy=multi-user.target
Then of course
#systemctl daemon-reload
#systemctl restart logstash
Related
I am using the nohup command with Python and Flask for background process. After I close the terminal it is working fine but after 1 or 2 days the process stops. Can someone tell me how to keep the background process running? I am using below command:
screen
space
nohup python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443 &
ctrl+a+d
Let's assume all your Flask code resides in the folder /home/abc_user/flask_app.
Steps
Create a file flask-server.service in /etc/systemd/system.
[Unit]
Description=Flask server
After=network.target
[Service]
User=abc_user
Group=abc_user
WorkingDirectory=/home/abc_user/flask_app
ExecStart=python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443
Restart=always
[Install]
WantedBy=multi-user.target
Run sudo systemctl daemon-reload.
Start the service using systemctl start flask-server.service.
Check that it has started by systemctl status flask-server.service. Status should say "running".
If you want your flask server to auto-start after reboots, run systemctl enable flask-server.service
Some common operations
Check current status - systemctl status flask-server.service
Start the service - systemctl start flask-server.service
Stop the service - systemctl stop flask-server.service
Check logs - journalctl -u flask-server.service
Stream logs - journalctl -f -u flask-server.service
Check logs in past 1 hour - journalctl -u flask-server.service --since "1 hour ago"
Try nohup python -m flask run --cert local.crt --key local.key --host=0.0.0.0 --port=443 >/dev/null 2>&1&
Use nohup , you should redirct you print to /dev/null or log, Otherwise it will be create a file nohup.out occupy disk space.
Most times we use gunicorn and supervisor to manager flask application.
Did you maybe shut down the computer the flask server is running on ?
If so, the problem will be solved by either not shutting down your computer or starting the flask server again after shutting down !
nohup is a POSIX command to ignore the HUP (hangup) signal. The HUP signal is, by convention, the way a terminal warns dependent processes of logout.
Output that would normally go to the terminal goes to a file called nohup.out if it has not already been redirected.
See nohup.out for searching errors in ./ or executed directory. It is no nohup error. Look nohup.out and google error and will refresh question.
My systemd unit file looks something like this
start() {
java -jar server.jar &
}
reload() {
#do some application specific reload stuff
start
}
I'm realizing that right when the systemd call to reload finishes, the process running java -jar is actually dead. Systemd seems to think that my reload process is done and kills it. Is there any way to configure and cancel this process killing behavior? I have tried disowning the process, and messing with KillMode=blah and Type=blah in the service file, but no result.
Running on CentOS 7.4
I don't know if I understood your question, do you want the systemd unit file for executing server.jar, right?
Try it by following sequences.
First create your blah service unit file.
# vim /etc/systemd/system/blah.service
[Unit]
Description=blah service
After=network.target
Requires=network.target
[Service]
Type=simple
EnvironmentFile=/etc/sysconfig/blah
ExecStart=/usr/bin/java -jar server.jar
Restart=always
User=blah_USERNAME
[Install]
WantedBy=multi-user.target
And reload the blah service unit file.
# systemctl daemon-reload
Test it!
# systemctl start blah
# systemctl status blah
# systemctl restart blah
# systemctl status blah
I hope this will help you.
Many system daemon can be started using start/stop command. I was just curious how start/stop works on Linux system. Say I wrote a daemon executable, how should I configure it so that it can be controlled by start/stop in Linux.
I make a daemon in linux (ArchLinux) few years ago, and it works every day perfectly.
There are 2 ways to do this. Short way and long way:
Short Way:
Create a file in /etc/systemd/system/ called for example mydaemon.service :
/etc/systemd/system/mydaemon.service
[Unit]
Description=This is my first daemon! - Fernando Pucci
After=network.target
[Service]
User=root
WorkingDirectory=/root
Type=oneshotmc
RemainAfterExit=yes
ExecStart=/bin/echo -e "Daemon started"
ExecStop=/bin/echo -e "Daemon Stopped"
[Install]
WantedBy=multi-user.target
This service does nothing but show Daemon Started or Stopped. You can change echoes by the sentences you need.
If you need to run some script, try the Long way:
Long way
Create a file in some directory, like root folder or /usr/lib/systemd/scripts called for example
/root/mydaemon.sh
start() {
<your start sentences here
and here>
}
stop() {
<your stop sentences here
and here>
}
case $1 in
start|stop) "$1" ;;
esac
You must to make it runnable (chmod x)
(And you can execute it with start or stop parameter to test it.)
And as second step, create another file in
/usr/lib/systemd/system/mydaemon.service
[Unit]
Description=Second daemon of Fernando Pucci
After=network.target
[Service]
User=root
WorkingDirectory=/root
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c '/root/mydaemon.sh start'
ExecStart=/bin/echo -e "MyDaemon Started"
ExecStop=/bin/bash -c '/root/mydaemon.sh stop'
ExecStop=/bin/echo -e "MyDaemon Stopped"
[Install]
WantedBy=multi-user.target
Starting and Stopping
systemctl start mydaemon
systemctl stop mydaemon
systemctl status mydaemon
systemctl enable mydaemon
systemctl disable mydaemon
You (and someone) can send me a private msg for help about that.
Curently, i want auditd service run forever and user can not stop this via any commands.
Current my auditd service:
~]# systemctl cat auditd
# /usr/lib/systemd/system/auditd.service
[Unit]
Description=Security Auditing Service
DefaultDependencies=no
After=local-fs.target systemd-tmpfiles-setup.service
Conflicts=shutdown.target
Before=sysinit.target shutdown.target
RefuseManualStop=yes
ConditionKernelCommandLine=!audit=0
[Service]
ExecStart=/sbin/auditd -n
## To not use augenrules, copy this file to /etc/systemd/system/auditd.service
## and comment/delete the next line and uncomment the auditctl line.
## NOTE: augenrules expect any rules to be added to /etc/audit/rules.d/
ExecStartPost=-/sbin/augenrules --load
#ExecStartPost=-/sbin/auditctl -R /etc/audit/audit.rules
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/auditd.service.d/override.conf
[Service]
ExecReload=
ExecReload=/bin/kill -HUP $MAINPID ; /sbin/augenrules --load
I can't stop this service from command:
# systemctl stop auditd.service
Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only.
But when i using service auditd stop command. I can stop this service normally.
# service auditd stop
Stopping logging: [ OK ]
How can i prevent it? Thanks
The administrator (root) will always be able to manually kill the auditd process (which is what the service command does). What systemd is doing here is only to prevent the administrator from doing it via the systemctl interface.
In both cases, unprivileged users can not kill the daemon.
If you want to restrict even what root can do, you will have to use SELinux and customize the policy.
Some actions of service command are not redirected to systemctl but run some specific scripts located in /usr/libexec/initscripts/legacy-actions.
In this case, stop command will call this script:
/usr/libexec/initscripts/legacy-actions/auditd/stop
If you want that, the audited service can't be stopped by service command, you can remove this script, the action "stop" will be redirected to systemctl, which will block it b/c of the parameter "RefuseManualStop=yes".
But this doesn't mean that you can't kill the process of course.
I am trying to redirect output of a systemd service to a file but it doesn't seem to work:
[Unit]
Description=customprocess
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=/var/log1.log
StandardError=/var/log2.log
Restart=always
[Install]
WantedBy=multi-user.target
Please correct my approach.
I think there's a more elegant way to solve the problem: send the stdout/stderr to syslog with an identifier and instruct your syslog manager to split its output by program name.
Use the following properties in your systemd service unit file:
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=<your program identifier> # without any quote
Then, assuming your distribution is using rsyslog to manage syslogs, create a file in /etc/rsyslog.d/<new_file>.conf with the following content:
if $programname == '<your program identifier>' then /path/to/log/file.log
& stop
Now make the log file writable by syslog:
# ls -alth /var/log/syslog
-rw-r----- 1 syslog adm 439K Mar 5 19:35 /var/log/syslog
# chown syslog:adm /path/to/log/file.log
Restart rsyslog (sudo systemctl restart rsyslog) and enjoy! Your program stdout/stderr will still be available through journalctl (sudo journalctl -u <your program identifier>) but they will also be available in your file of choice.
Source via archive.org
If you have a newer distro with a newer systemd (systemd version 236 or newer), you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME.
Long story:
In newer versions of systemd there is a relatively new option (the github request is from 2016 ish and the enhancement is merged/closed 2017 ish) where you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME. The file:path option is documented in the most recent systemd.exec man page.
This new feature is relatively new and so is not available for older distros like centos-7 (or any centos before that).
I would suggest adding stdout and stderr file in systemd service file itself.
Referring : https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
As you have configured it should not like:
StandardOutput=/home/user/log1.log
StandardError=/home/user/log2.log
It should be:
StandardOutput=file:/home/user/log1.log
StandardError=file:/home/user/log2.log
This works when you don't want to restart the service again and again.
This will create a new file and does not append to the existing file.
Use Instead:
StandardOutput=append:/home/user/log1.log
StandardError=append:/home/user/log2.log
NOTE: Make sure you create the directory already. I guess it does not support to create a directory.
You possibly get this error:
Failed to parse output specifier, ignoring: /var/log1.log
From the systemd.exec(5) man page:
StandardOutput=
Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket.
The systemd.exec(5) man page explains other options related to logging. See also the systemd.service(5) and systemd.unit(5) man pages.
Or maybe you can try things like this (all on one line):
ExecStart=/bin/sh -c '/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server 2>&1 > /var/log.log'
If for a some reason can't use rsyslog, this will do:
ExecStart=/bin/bash -ce "exec /usr/local/bin/binary1 agent -config-dir /etc/sample.d/server >> /var/log/agent.log 2>&1"
Short answer:
StandardOutput=file:/var/log1.log
StandardError=file:/var/log2.log
If you don't want the files to be cleared every time the service is run, use append instead:
StandardOutput=append:/var/log1.log
StandardError=append:/var/log2.log
We are using Centos7, spring boot application with systemd. I was running java as below. and setting StandardOutput to file was not working for me.
ExecStart=/bin/java -jar xxx.jar -Xmx512-Xms32M
Below workaround solution working without setting StandardOutput. running java through sh as below.
ExecStart=/bin/sh -c 'exec /bin/java -jar xxx.jar -Xmx512M -Xms32M >> /data/logs/xxx.log 2>&1'
Assume logs are already put to stdout/stderr, and have systemd unit's log in /var/log/syslog
journalctl -u unitxxx.service
Jun 30 13:51:46 host unitxxx[1437]: time="2018-06-30T11:51:46Z" level=info msg="127.0.0.1
Jun 30 15:02:15 host unitxxx[1437]: time="2018-06-30T13:02:15Z" level=info msg="127.0.0.1
Jun 30 15:33:02 host unitxxx[1437]: time="2018-06-30T13:33:02Z" level=info msg="127.0.0.1
Jun 30 15:56:31 host unitxxx[1437]: time="2018-06-30T13:56:31Z" level=info msg="127.0.0.1
Config rsyslog (System Logging Service)
# Create directory for log file
mkdir /var/log/unitxxx
# Then add config file /etc/rsyslog.d/unitxxx.conf
if $programname == 'unitxxx' then /var/log/unitxxx/unitxxx.log
& stop
Restart rsyslog
systemctl restart rsyslog.service
In my case 2>&1(stdout and stderr file descriptor symbol) had to be placed correctly,then log redirection worked as I expected
[Unit]
Description=events-server
[Service]
User=manjunath
Type=simple
ExecStart=/bin/bash -c '/opt/events-server/bin/start.sh my-conf 2>&1 >> /var/log/events-server/events.log'
[Install]
WantedBy=multi-user.target
Make your service file call a shell script instead of running the app directly. This way you have extra control. For example, you can make output files like those in /var/log/
Make a shell script like /opt/myapp/myapp.sh
#!/bin/sh
/usr/sbin/logrotate --force /opt/myapp/myapp.conf --state /opt/myapp/state.tmp
logger "[myapp] Run" # send a marker to syslog
myapp > /opt/myapp/myapp.log 2>&1 &
And your service file myapp.service contains:
...
[Service]
Type=forking
ExecStart=/bin/sh -c /opt/myapp/myapp.sh
...
A sample of log config file /opt/myapp/myapp.conf
/opt/myapp/myapp.log {
daily
rotate 20
missingok
compress
}
Then you will get myapp.log, and zipped myapp.log.1.gz ... for each time the service was started, and previous zipped.