Can't detach child process when main process is started from systemd - linux

Question

I want to spawn long-running child processes that survive when the main process restarts/dies. This works fine when running from the terminal:
$ cat exectest.go
package main
import (
"log"
"os"
"os/exec"
"syscall"
"time"
)
func main() {
if len(os.Args) == 2 && os.Args[1] == "child" {
for {
time.Sleep(time.Second)
}
} else {
cmd := exec.Command(os.Args[0], "child")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
log.Printf("child exited: %v", cmd.Run())
}
}
$ go build
$ ./exectest
^Z
[1]+ Stopped ./exectest
$ bg
[1]+ ./exectest &
$ ps -ef | grep exectest | grep -v grep | grep -v vim
snowm 7914 5650 0 23:44 pts/7 00:00:00 ./exectest
snowm 7916 7914 0 23:44 ? 00:00:00 ./exectest child
$ kill -INT 7914 # kill parent process
[1]+ Exit 2 ./exectest
$ ps -ef | grep exectest | grep -v grep | grep -v vim
snowm 7916 1 0 23:44 ? 00:00:00 ./exectest child
Note that the child process is still alive after parent process was killed. However, if I start the main process from systemd like this...
[snowm#localhost exectest]$ cat /etc/systemd/system/exectest.service
[Unit]
Description=ExecTest
[Service]
Type=simple
ExecStart=/home/snowm/src/exectest/exectest
User=snowm
[Install]
WantedBy=multi-user.target
$ sudo systemctl enable exectest
ln -s '/etc/systemd/system/exectest.service' '/etc/systemd/system/multi-user.target.wants/exectest.service'
$ sudo systemctl start exectest
... then the child also dies when I kill the main process:
$ ps -ef | grep exectest | grep -v grep | grep -v vim
snowm 8132 1 0 23:55 ? 00:00:00 /home/snowm/src/exectest/exectest
snowm 8134 8132 0 23:55 ? 00:00:00 /home/snowm/src/exectest/exectest child
$ kill -INT 8132
$ ps -ef | grep exectest | grep -v grep | grep -v vim
$
How can I make the child survive?
Running go version go1.4.2 linux/amd64 under CentOS Linux release 7.1.1503 (Core).

Answer 1

Solution is to add
KillMode=process
to the service block. Default value is control-group which means systemd cleans up any child processes.
From man systemd.kill
KillMode= Specifies how processes of this unit shall be killed. One of
control-group, process, mixed, none.
If set to control-group, all remaining processes in the control group
of this unit will be killed on unit stop (for services: after the stop
command is executed, as configured with ExecStop=). If set to process,
only the main process itself is killed. If set to mixed, the SIGTERM
signal (see below) is sent to the main process while the subsequent
SIGKILL signal (see below) is sent to all remaining processes of the
unit's control group. If set to none, no process is killed. In this
case, only the stop command will be executed on unit stop, but no
process be killed otherwise. Processes remaining alive after stop are
left in their control group and the control group continues to exist
after stop unless it is empty.

Answer 2

The only way I know to solve this is to launch the child process with the --scope argument.
systemd-run --user --scope firefox
KillMode has been mentioned here also, but changing the KillMode also means that if your main process crashes, systemd won't restart it if any child process is still running.

Answer 3

If you cannot (like me) to change the KillMode of the service for some reason, you could try the at command (see man).
You can schedule your command to run 1 minute ahead. See an example:
# this will remove all .tmp files from "/path/" in 1 minute ahead (this task will run once)
echo rm /path/*.tmp | at now + 1 minute