Why does systemd keep killing my service - linux

I would like to start an interactive script from systemd after the getty.target has been reached. This works so far, however, systemd kills the script after a couple of seconds. The systemd unit looks like the following:
[Unit]
Description = Some interactive script
Requires = getty#tty1.service
After = getty#tty1.service
[Service]
Type = oneshot
ExecStart = /usr/local/bin/my-script
StandardInput = tty
StandardOutput = tty
TTYPath = /dev/tty1
TTYReset = yes
TTYVHangup = yes
[Install]
WantedBy = multi-user.target
Within the script there are calls to dialog, mount etc. Nothing very special but it is an interactive script. Systemd keeps killing the script and I don't understand why. The output of systemctl status interactive-script.service looks like:
● interactive-script.service - Some interactive script
Loaded: loaded (/etc/systemd/system/interactive-script.service; enabled)
Active: inactive (dead) since Tue 2016-06-28 10:18:07 UTC; 14min ago
Main PID: 364 (code=killed, signal=HUP)
And the log output gotten with journalctl -b -u interactive-script.service is empty:
-- Logs begin at Mon 2015-11-09 11:49:52 UTC, end at Tue 2016-06-28 10:30:28 UTC. --
I already tried to add KillMode=none, no luck. Then I tried TimeoutStartSec=infinity - systemd complains that it doesn't understand it therefore I tried to set it to 10000 but the script gets killed after just some seconds. I tried to run it as Type=simple and Type=forking, all to no avail.
The point is that starting the script seems to work fine (the dialogs appear) but systemd keeps killing the script. How can I achieve that systemd does not kill this interactive script?

Related

Raspberry pi: Associate "screen" and Systemd to auto-start a minecraft server at pi's start

I'm new in raspberry pi programming, and i want to be able to launch a minecraft server at the start of the pi.
For that, I've already loocked at Systemd files and screen command.
I manage to make them work separately, but not together, it's why I'm looking for help there.
Firstly, I'm using a Raspberry pi 4 4Go with raspbian v10, and forge 1.12.2 with java 8.
I did a .sh file to launch easier the server:
#!/bin/bash
screen -S mcserver -dm java -Xms1024M -Xmx2048M -jar /home/pi/MinecraftServer/server/forge-1.12.2-14.23.5.2854.jar nogui
When I run the file, the server start perfectly in a socket as I want.
Secondly, I have a systemd file (auto-run-server.service):
[Unit]
Description=Auto run mc server
[Service]
ExecStart=/home/pi/MinecraftServer/server/minecraft.sh
[Install]
WantedBy=multi-user.target
But when I execute the service, nothing is happening, the status of the service shows a sucess, but there is nothing in screens (screen -list)
And when i replace the ExecStart value by
ExecStart=java -Xms1024M -Xmx2048M -jar /home/pi/MinecraftServer/server/forge-1.12.2-14.23.5.2854.jar nogui
The server starts, but the problem is that I want to access to a terminal to run commands in minecraft server, and i didn't find solution to access from there.
( It's why I want to create a "screen" )
I'm fully open to your answers, even if they don't use "screen", as long as I can access to a server terminal.
Thanks in advance.
I'm using the follow systemd unit for testing:
[Service]
ExecStart=/tmp/screentest.sh
And this screentest.sh shell script:
#!/bin/sh
screen -S mcserver -dm sh -c 'while :; do date; sleep 5; done'
If I start the service (systemctl start screentest) and then run systemctl status screentest, I see:
● screentest.service
Loaded: loaded (/etc/systemd/system/screentest.service; static; vendor preset: enabled)
Active: inactive (dead)
The problem here is that the screen command exits immediately when running with -d, so systemd believes the command has completed and cleans everything up by removing any additional processes spawned by the service.
We can tell systemd that the service spawns a child and exits by setting the service type to forking:
[Service]
Type=forking
ExecStart=/tmp/screentest.sh
With this change in place, after starting the service we see:
● screentest.service
Loaded: loaded (/etc/systemd/system/screentest.service; static; vendor preset: enabled)
Active: active (running) since Sun 2021-01-10 09:58:11 EST; 4s ago
Process: 14461 ExecStart=/tmp/screentest.sh (code=exited, status=0/SUCCESS)
Main PID: 14463 (screen)
Tasks: 3 (limit: 4915)
CGroup: /system.slice/screentest.service
├─14463 SCREEN -S mcserver -dm sh -c while :; do date; sleep 5; done
├─14464 sh -c while :; do date; sleep 5; done
└─14466 sleep 5
And screen -list shows:
root#raspberrypi:/etc/systemd/system# screen -list
There is a screen on:
14612.mcserver (01/10/2021 10:01:55 AM) (Detached)
1 Socket in /run/screen/S-root.

While loop in bash script breaks systemd service

I'm on Debian and I have a systemd service that calls a bash script.
The script contains an infinite while loop, as I need it to check something every X seconds infinitely.
The systemd service crashes once it hits the "while true; do" line.
The script runs fine if I execute it manually.
Why doesn't systemd like it? What do I do?
Here are the service and the script. As I've indicated, an echo statement before the "while true; do" prints. The echo statement after the "while true; do" line does not print.
/etc/systemd/system/stream.service:
[Service]
WorkingDirectory=/home/pi/
ExecStart=/home/pi/joi_main.sh
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=stream_service
User=pi
Group=pi
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target
/home/pi/joi_main.sh:
#!/bin/bash -e
today=`/bin/date '+%Y_%m_%d__%H_%M_%S'`
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
#Wait 120s for system to finish booting
sleep 120
#Initial config
export AUDIODEV=mic_mono
export AUDIODRIVER=alsa
sudo sysctl fs.pipe-max-size=1048576
echo "This line prints"
# Check if video buffer is full every minute. if full, the stream needs to restart
while true; do
echo "This line doesn't"
if grep "100% full" /home/pi/video_buffer_usage.txt; then
echo "Buffer is full!"
# Kill existing processes
pkill -f “raspivid|rec|buffer|ffmpeg”
# Wait 10s
sleep 10
./joi_stream.sh &
fi
sleep 60
done
Journalctl seems completely unhelpful, but here it is. No errors. Why is "session closed"?
Mar 31 02:13:41 raspberrypi sudo[1369]: pi : TTY=unknown ; PWD=/home/pi ; USER=root ; COMMAND=/sbin/sysctl fs.pipe-max-size=1048576
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session closed for user root
(Please don't tell me to start yet another systemd service for just this while loop. I want it to be part of this main script because it needs to run after everything else, and if I turn off the main service I don't want the while loop running either, so maintaining two systemd services would only add troube.)
The contents of ./joi_stream.sh were not shared, but here's a problem I see with your systemd solution. It doesn't directly explain your behavior, but may be related:
In your systemd configuration, you redirect both STDOUT and STDERR to syslog, but in your script, you redirect STDERR (file descriptor "2") to a file, and redirect STDOUT (file descriptor "1') to STDERR.
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
If your ./joi_stream.sh expected your redirection of these file descriptors to another file to work, it may not. If the file is just for logging, I would get rid of these lines and let the systemd journal handle that-- it will tag the logs with your unit you can review your logs specifically:
journalctl -u your-unit-name.service
Also, in systemd, you wouldn't normally put in a sleep to wait until the systemd has booted. Instead, you would use a .timer unit.
The .timer file would instruct to run the main logic every minute, so the "while" loop would not be required. The timer unit would contain directives like:
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
It would be timer unit which is enabled to start on boot. Timer files can be super-simple. Just create a .timer file in /etc/systemd/system and give it the same name as the service file you want it to activate:
[Unit]
Description=Runs my service every minute
[Timer]
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
[Install]
WantedBy=timers.target
To start and test your timer immediately, run:
sudo systemctl start my-service.timer
You can review the status of timers with:
sudo systemctl list-timers
The systemd solution is more robust than the rc.local solution. If your rc.local solution dies for any reason, it will not restart. However, if your script dies will run under systemd, the timer will still run it again a minute later.
FYI, everything works if I call /home/pi/joi_main.sh from /etc/rc.local instead of using a systemd service. I'll use rc.local and kill the service.

systemctl status shows inactive dead

I am trying to write my own (simple) systemd service which does something simple.( Like writing numbers 1 to 10 to a file, using the shell script).
My service file looks like below.
[Unit]
Description=NandaGopal
Documentation=https://google.com
After=multi-user.target
[Service]
Type=forking
RemainAfterExit=yes
ExecStart=/usr/bin/hello.sh &
[Install]
RequiredBy = multi-user.target
This is my shell script.
#!/usr/bin/env bash
source /etc/profile
a=0
while [ $a -lt 10 ]
do
echo $a >> /var/log//t.txt
a=`expr $a + 1`
done
For some reason, the service doesn't come up and systemctl is showing the below output.
root#TARGET:~ >systemctl status -l hello
* hello.service - NandaGopal
Loaded: loaded (/usr/lib/systemd/system/hello.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://google.com
Been trying to figure out what went wrong for the last 2 days.
You have set Type=Forking, but your service doesn't work. Try
Type=oneshot
You have a "&" your ExecStart line, which is not necessary.
The service is disabled, which means it was not enabled to start at boot. You should run systemctl enable hello to set it to start at boot.
You can check man systemd.directives to find an index of all the directives that you can use in your unit files.
Few points:
If you use Type=forking, it is recommended to specify PidFile.
In your case, Type=simple, and ExecStart without & will work.
use systemctl start service-name to start a service
Then use systemctl status service-name to check its status.
status will be inactive/dead if service is not started.

systemd: SIGTERM immediately after start

I am trying systemd for the first time. I want to start a process at system bootup. And I have a problem in getting it up and running.
systemd should run a script (start.sh). This script starts a processes (lets call it P) in the background and exits with code 0.
P keeps running forever till a signal happends.
If I run start.sh manually all is ok.
If I let it start by systemd P gets immediately after the start a SIGTERM and terminates.
So it get started but what about the signal??
It terminates P and I am not sure whats its origin and the reason for it.
Maybe my unit is wrong but I have no idea how to set it for my needs.
I tried service-type simple, idle and oneshot.
Thanks for help!
Chris
Here is my unit.
[Unit]
Description=Test
After=sshd.service
[Service]
Type=oneshot
ExecStart=/home/max/start.sh start
Restart=no
User=root
SuccessExitStatus=0
[Install]
WantedBy=multi-user.target
Thats the status.
Loaded: loaded (/etc/systemd/system/test.service; enabled)
Active: inactive (dead) since Die 2016-02-23 20:56:59 CET; 20min ago
Process: 1046 ExecStart=/home/max/test.sh start (code=exited, status=0/SUCCESS)
When start.sh finishes, systemd kills everything in the same cgroup as start.sh
Your options are:
setting KillMode in the Unit section to process (the default is control-group). That will cause systemd to only kill the process which it directly fired.
to not make start.sh start something in the background and exit but to execute it right there in the foreground
I think in your situation option 2 is viable and more straightforward.
Source: https://unix.stackexchange.com/a/231201/45329
Although changing the KillMode to process like below will work in your situation, it is not the recommended solution.
[Service]
KillMode=process
...
The problem with KillMode set to process is that systemd loses control over all the children of the process it started. That means, if anything happens and one of your processes does not die for some reason, it will continue to linger around.
A better solution in your situation would be to create all the processes, keep their pid and then wait on them.
The wait command that you use in your shell script may vary depending on which shell you are using (the link I proposed is for bash). Having the shell script wait for all the children is in effect the same as starting one child, which does not get detached, in the foreground.
So something like this, more or less:
#!/bin/bash
# Start your various processes
process1 &
PROCESS1_PID=$!
process2 &
PROCESS2_PID=$!
process3 &
PROCESS3_PID=$!
# Wait on your processes
wait $PROCESS1_PID $PROCESS2_PID $PROCESS3_PID
# OR, if I'm correct, bash also allows you to wait on all children
# with just a plain wait like so:
wait
# reach here only after children 1, 2, and 3 died

Redirect systemd script output to console

How can I redirect output of script which is run by systemd ExecStart script to boot console?
I need to debug what is wrong with script until boot but I can't use journalctl because it's embedded linux with ROM rootfs.
Now my .service file looks like:
[Unit]
Description=Init script
After=network.target
Before=getty#tty1.service
[Service]
Type=oneshot
ExecStart=/usr/lib/systemd/test_init_script
ExecStartPre=/usr/bin/echo -e \033%G
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/
Enviroment=TERM=xterm
[Install]
WantedBy=multi-user.target
test_init_script:
#!/bin/sh -
echo "Test!"
And it didn't work, after boot I receive message:
#systemctl status test_init_script.service
test_init_script.service - Init script
Loaded: loaded (/usr/lib/systemd/test_init_script.service)
Active: failed (Result: exit-code) Since Thu 1970-01-01 08:26:03 CST; 19s ago
Process: 170 ExecStartOre=/usr/bin/echo -e -G (code=exited, status=203/EXEC)
Did anyone know how to redirect script output to terminal?
Ok, so I resolve this problem by building image with changed file:
/etc/journald.conf
Where I change storage option to volatile:
Storage=volatile
This option means that all journald data will be store in RAM so this is a workaround for read only file system.
Please refer to journald manual page to see more options.

Resources