How to redirect output of systemd service to a file - linux

I am trying to redirect output of a systemd service to a file but it doesn't seem to work:
[Unit]
Description=customprocess
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=/var/log1.log
StandardError=/var/log2.log
Restart=always
[Install]
WantedBy=multi-user.target
Please correct my approach.

I think there's a more elegant way to solve the problem: send the stdout/stderr to syslog with an identifier and instruct your syslog manager to split its output by program name.
Use the following properties in your systemd service unit file:
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=<your program identifier> # without any quote
Then, assuming your distribution is using rsyslog to manage syslogs, create a file in /etc/rsyslog.d/<new_file>.conf with the following content:
if $programname == '<your program identifier>' then /path/to/log/file.log
& stop
Now make the log file writable by syslog:
# ls -alth /var/log/syslog
-rw-r----- 1 syslog adm 439K Mar 5 19:35 /var/log/syslog
# chown syslog:adm /path/to/log/file.log
Restart rsyslog (sudo systemctl restart rsyslog) and enjoy! Your program stdout/stderr will still be available through journalctl (sudo journalctl -u <your program identifier>) but they will also be available in your file of choice.
Source via archive.org

If you have a newer distro with a newer systemd (systemd version 236 or newer), you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME.
Long story:
In newer versions of systemd there is a relatively new option (the github request is from 2016 ish and the enhancement is merged/closed 2017 ish) where you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME. The file:path option is documented in the most recent systemd.exec man page.
This new feature is relatively new and so is not available for older distros like centos-7 (or any centos before that).

I would suggest adding stdout and stderr file in systemd service file itself.
Referring : https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
As you have configured it should not like:
StandardOutput=/home/user/log1.log
StandardError=/home/user/log2.log
It should be:
StandardOutput=file:/home/user/log1.log
StandardError=file:/home/user/log2.log
This works when you don't want to restart the service again and again.
This will create a new file and does not append to the existing file.
Use Instead:
StandardOutput=append:/home/user/log1.log
StandardError=append:/home/user/log2.log
NOTE: Make sure you create the directory already. I guess it does not support to create a directory.

You possibly get this error:
Failed to parse output specifier, ignoring: /var/log1.log
From the systemd.exec(5) man page:
StandardOutput=
Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket.
The systemd.exec(5) man page explains other options related to logging. See also the systemd.service(5) and systemd.unit(5) man pages.
Or maybe you can try things like this (all on one line):
ExecStart=/bin/sh -c '/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server 2>&1 > /var/log.log'

If for a some reason can't use rsyslog, this will do:
ExecStart=/bin/bash -ce "exec /usr/local/bin/binary1 agent -config-dir /etc/sample.d/server >> /var/log/agent.log 2>&1"

Short answer:
StandardOutput=file:/var/log1.log
StandardError=file:/var/log2.log
If you don't want the files to be cleared every time the service is run, use append instead:
StandardOutput=append:/var/log1.log
StandardError=append:/var/log2.log

We are using Centos7, spring boot application with systemd. I was running java as below. and setting StandardOutput to file was not working for me.
ExecStart=/bin/java -jar xxx.jar -Xmx512-Xms32M
Below workaround solution working without setting StandardOutput. running java through sh as below.
ExecStart=/bin/sh -c 'exec /bin/java -jar xxx.jar -Xmx512M -Xms32M >> /data/logs/xxx.log 2>&1'

Assume logs are already put to stdout/stderr, and have systemd unit's log in /var/log/syslog
journalctl -u unitxxx.service
Jun 30 13:51:46 host unitxxx[1437]: time="2018-06-30T11:51:46Z" level=info msg="127.0.0.1
Jun 30 15:02:15 host unitxxx[1437]: time="2018-06-30T13:02:15Z" level=info msg="127.0.0.1
Jun 30 15:33:02 host unitxxx[1437]: time="2018-06-30T13:33:02Z" level=info msg="127.0.0.1
Jun 30 15:56:31 host unitxxx[1437]: time="2018-06-30T13:56:31Z" level=info msg="127.0.0.1
Config rsyslog (System Logging Service)
# Create directory for log file
mkdir /var/log/unitxxx
# Then add config file /etc/rsyslog.d/unitxxx.conf
if $programname == 'unitxxx' then /var/log/unitxxx/unitxxx.log
& stop
Restart rsyslog
systemctl restart rsyslog.service

In my case 2>&1(stdout and stderr file descriptor symbol) had to be placed correctly,then log redirection worked as I expected
[Unit]
Description=events-server
[Service]
User=manjunath
Type=simple
ExecStart=/bin/bash -c '/opt/events-server/bin/start.sh my-conf 2>&1 >> /var/log/events-server/events.log'
[Install]
WantedBy=multi-user.target

Make your service file call a shell script instead of running the app directly. This way you have extra control. For example, you can make output files like those in /var/log/
Make a shell script like /opt/myapp/myapp.sh
#!/bin/sh
/usr/sbin/logrotate --force /opt/myapp/myapp.conf --state /opt/myapp/state.tmp
logger "[myapp] Run" # send a marker to syslog
myapp > /opt/myapp/myapp.log 2>&1 &
And your service file myapp.service contains:
...
[Service]
Type=forking
ExecStart=/bin/sh -c /opt/myapp/myapp.sh
...
A sample of log config file /opt/myapp/myapp.conf
/opt/myapp/myapp.log {
daily
rotate 20
missingok
compress
}
Then you will get myapp.log, and zipped myapp.log.1.gz ... for each time the service was started, and previous zipped.

Related

systemctl Exec format error when trying to run service

Currently I wanted to run my dedicated server on my vps. When I run system systemctl start csgo.service it gives me error Load: error (Reason: Exec format error) when I run systemctl status csgo.service it gives me /lib/systemd/system/csgo.service:12: Executable path is not absolute: killall -TERM srcds_linux. Below are the service file that I am trying to run, am I making any mistake since it says format error?
[Unit]
Description=CSGO Server
[Service]
Type=simple
User=steam
Group=steam
Restart=on-failure
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
ExecStart=/home/steam/steamcmd/csgo/srcds_run -game csgo -console -usercon +game_type 0 +game_mode 1 -tickrate 128 +mapgroup mg_active +map de_dust2 +sv_setsteamaccount GsltKeyHere -net_port_try 1
ExecStop=killall -TERM srcds_linux
[Install]
WantedBy=multi-user.target
My dedicated server files are inside /home/steam/steamcmd/csgo
Quoting the manual on unit files:
Note that shell command lines are not directly supported. If shell command lines are to be used, they need to be passed explicitly to a shell implementation of some kind.
Example: ExecStart=sh -c 'dmesg | tac'
You'll need to either use sh like that or figure out the actual path to your killall executable, e.g.
[Unit]
ExecStop=sh -c 'killall -TERM srcds_linux'
or
[Unit]
ExecStop=/sbin/killall -TERM srcds_linux
As an aside, that's not the best of ExecStop commands; it'll ruthlessly kill all srcds_linux executables, no matter if they're related to this service or not. Having no ExecStop command will have systemd terminate the service by itself:
Note that it is usually not sufficient to specify a command for this setting that only asks the service to terminate (for example, by queuing some form of termination signal for it), but does not wait for it to do so. Since the remaining processes of the services are killed according to KillMode= and KillSignal= as described above immediately after the command exited, this may not result in a clean stop. The specified command should hence be a synchronous operation, not an asynchronous one.

While loop in bash script breaks systemd service

I'm on Debian and I have a systemd service that calls a bash script.
The script contains an infinite while loop, as I need it to check something every X seconds infinitely.
The systemd service crashes once it hits the "while true; do" line.
The script runs fine if I execute it manually.
Why doesn't systemd like it? What do I do?
Here are the service and the script. As I've indicated, an echo statement before the "while true; do" prints. The echo statement after the "while true; do" line does not print.
/etc/systemd/system/stream.service:
[Service]
WorkingDirectory=/home/pi/
ExecStart=/home/pi/joi_main.sh
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=stream_service
User=pi
Group=pi
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target
/home/pi/joi_main.sh:
#!/bin/bash -e
today=`/bin/date '+%Y_%m_%d__%H_%M_%S'`
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
#Wait 120s for system to finish booting
sleep 120
#Initial config
export AUDIODEV=mic_mono
export AUDIODRIVER=alsa
sudo sysctl fs.pipe-max-size=1048576
echo "This line prints"
# Check if video buffer is full every minute. if full, the stream needs to restart
while true; do
echo "This line doesn't"
if grep "100% full" /home/pi/video_buffer_usage.txt; then
echo "Buffer is full!"
# Kill existing processes
pkill -f “raspivid|rec|buffer|ffmpeg”
# Wait 10s
sleep 10
./joi_stream.sh &
fi
sleep 60
done
Journalctl seems completely unhelpful, but here it is. No errors. Why is "session closed"?
Mar 31 02:13:41 raspberrypi sudo[1369]: pi : TTY=unknown ; PWD=/home/pi ; USER=root ; COMMAND=/sbin/sysctl fs.pipe-max-size=1048576
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session closed for user root
(Please don't tell me to start yet another systemd service for just this while loop. I want it to be part of this main script because it needs to run after everything else, and if I turn off the main service I don't want the while loop running either, so maintaining two systemd services would only add troube.)
The contents of ./joi_stream.sh were not shared, but here's a problem I see with your systemd solution. It doesn't directly explain your behavior, but may be related:
In your systemd configuration, you redirect both STDOUT and STDERR to syslog, but in your script, you redirect STDERR (file descriptor "2") to a file, and redirect STDOUT (file descriptor "1') to STDERR.
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
If your ./joi_stream.sh expected your redirection of these file descriptors to another file to work, it may not. If the file is just for logging, I would get rid of these lines and let the systemd journal handle that-- it will tag the logs with your unit you can review your logs specifically:
journalctl -u your-unit-name.service
Also, in systemd, you wouldn't normally put in a sleep to wait until the systemd has booted. Instead, you would use a .timer unit.
The .timer file would instruct to run the main logic every minute, so the "while" loop would not be required. The timer unit would contain directives like:
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
It would be timer unit which is enabled to start on boot. Timer files can be super-simple. Just create a .timer file in /etc/systemd/system and give it the same name as the service file you want it to activate:
[Unit]
Description=Runs my service every minute
[Timer]
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
[Install]
WantedBy=timers.target
To start and test your timer immediately, run:
sudo systemctl start my-service.timer
You can review the status of timers with:
sudo systemctl list-timers
The systemd solution is more robust than the rc.local solution. If your rc.local solution dies for any reason, it will not restart. However, if your script dies will run under systemd, the timer will still run it again a minute later.
FYI, everything works if I call /home/pi/joi_main.sh from /etc/rc.local instead of using a systemd service. I'll use rc.local and kill the service.

Fedora 20 how to run script at the end of startup

I am using Fedora 20. I have a two lines bash script needs to be run at the end of the startup. I want it to be run automatically each time when machine is startup. How can I do this?
I tried "sudo crontab -e" to insert my executable script but it always gave me error teling me the the time is not right and cannot modify the file.
You can create a Systemd unit file in /usr/lib/systemd/system/<service_name>.service. Here is a template:
[Unit]
Description=<description_string>
[Service]
WorkingDirectory=<working_directory>
Type=forking
ExecStart=/bin/bash <absolute_path_to_script>
KillMode=process
[Install]
WantedBy=multi-user.target
Replace anything in the angle brackets with your specific information. The 'WantedBy=multi-user.target' is the magic that tells Systemd to run your script on each start.
On the command line, tell Systemd to enable your service:
systemctl enable <service_name>.service
The next time you reboot your script should be run. Logs will be written to /var/log/messages.
Fedora has some basic documentation on unit files: Systemd unit files
You can append /etc/rc.local it runs just after the system starts up.
You may have to create it if doesn't exist:
Check this answer
Charlie's answer is better but you can still use Tiago's answer.
Just don't forget if you want to use /etc/rc.local way, grant execution permission to this file after editing:
chmod +x /etc/rc.local

How to run last and print my script output during boot with systemd?

I’m trying to configure my host during deployment process and to give an output to the screen of what my configuration script is doing.
In RHEL6 it was easy i was echoing what I want to screen or used dialog to display the output, and only when my script was done i got the login prompt.
( I used rc3.d or rc5.d folder with script name S99.myscript.sh)
In RHEL7 i can’t mimic this process.
rc.local does not display my output during booting and also its not guaranteed it will run last.
I guess I need to create a systemd service file that will run my script.
But how do I output the result to the screen while booting?
And how do I make sure I will not get the log-in prompt before my script ends?
below service example works like a charm :)
[Unit]
Description=ldt_bootscript1.service
After=network.target
Before=getty#tty1.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c "/bin/bash /tmp/ldt_scripts/postinstall/rc.firstboot.qas | /usr/bin/dialog --clear --backtitle \"Linux Deployment\" --title \"tests\" --progressbox 20 70 > /dev/console 2>&1"
ExecStartPre=/usr/bin/echo -e \033%G
ExecReload=/bin/kill -HUP $MAINPID
RemainAfterExit=no
WorkingDirectory=/
Environment=TERM=xterm
[Install]
WantedBy=multi-user.target

How to configure ElasticSearch to restart after crash

Using the apt-get package for ElasticSearh, how can I configure the service to restart itself automatically after crashing on Ubuntu?
Restart on failure option is missing in the default service of elasticsearch.
So, We can add Restart=always option in the service.
Steps to add - Restart=always
Edit elasticsearch service unit file using the command sudo systemctl edit elasticsearch.service. This command will create a file /etc/systemd/system/elasticsearch.service.d/override.conf.
Now, add the following lines in the unit file.
[Service]
Restart=always
Save the file and refresh the unit file using command sudo systemctl daemon-reload
Can check the changes using command sudo systemctl cat elasticsearch.service.
Note:
We can use Restart= always, on-abnormal, on-success, on-failure, etc based on the requirement. Reference.
Editing unit file - Reference
Write a #!/bin/sh script as follows:
if ps -ef | grep -v grep | grep elastic ; then
exit 0
else
/etc/init.d/elasticsearch start >> /var/run/elasticsearch.pid &
exit 0
fi
Auto restart elasticsearch services in 7.14.1:
Go to:
nano /usr/lib/systemd/system/elasticsearch.service
The location of the service file is changed in 7.14.1
Then add this line to the service file:
[Service]
Restart=always
After that save the file and restart the elasticsearch service.
Now you are good to go. (After this there won't be any crash)

Resources