Start a process in Go and detach from it - linux

I need to start a new process in Go with the following requirements:
The starting process should run even after the Go process is terminated
I need to be able to set the Unix user/group that's running it
I need to be able to set the environment variables inherited
I need control over std in/out/err
Here is an attempt:
var attr = os.ProcAttr {
Dir: "/bin",
Env: os.Environ(),
Files: []*os.File{
os.Stdin,
"stdout.log",
"stderr.log",
},
}
process, err := os.StartProcess("sleep", []string{"1"}, &attr)
This works fine but has the following shortcomings from the requirements:
No way to set Unix user/group
The started process ends when the Go process (parent) stops
This needs to run on Linux only if that simplifies things.

You can use process.Release to detach the child process from the parent one and make it survive after parent death
Look at the definition of *os.ProcAttr.Sys.Credentials attribute : it looks like using the attribute you can set process user and group ID.
Here is a working version of your example (I did not check if process ID's where actually the one set )
package main
import "fmt"
import "os"
import "syscall"
const (
UID = 501
GUID = 100
)
func main() {
// The Credential fields are used to set UID, GID and attitional GIDS of the process
// You need to run the program as root to do this
var cred = &syscall.Credential{ UID, GUID, []uint32{} }
// the Noctty flag is used to detach the process from parent tty
var sysproc = &syscall.SysProcAttr{ Credential:cred, Noctty:true }
var attr = os.ProcAttr{
Dir: ".",
Env: os.Environ(),
Files: []*os.File{
os.Stdin,
nil,
nil,
},
Sys:sysproc,
}
process, err := os.StartProcess("/bin/sleep", []string{"/bin/sleep", "100"}, &attr)
if err == nil {
// It is not clear from docs, but Realease actually detaches the process
err = process.Release();
if err != nil {
fmt.Println(err.Error())
}
} else {
fmt.Println(err.Error())
}
}

What I have found that seems to work cross-platform is to re-run the program with a special flag. In your main program, check for this flag. If present on startup, you're in the "fork". If not present, re-run the command with the flag.
func rerunDetached() error {
cwd, err := os.Getwd()
if err != nil {
return err
}
args := append(os.Args, "--detached")
cmd := exec.Command(args[0], args[1:]...)
cmd.Dir = cwd
err = cmd.Start()
if err != nil {
return err
}
cmd.Process.Release()
return nil
}
This will simply re-run your process with the exact parameters and append --detached to the arguments. When your program starts, check for the --detached flag to know if you need to call rerunDetached or not. This is sort of like a poor mans fork() which will work across different OS.

Related

Go exec.CommandContext is not being terminated after context timeout

In golang, I can usually use context.WithTimeout() in combination with exec.CommandContext() to get a command to automatically be killed (with SIGKILL) after the timeout.
But I'm running into a strange issue that if I wrap the command with sh -c AND buffer the command's outputs by setting cmd.Stdout = &bytes.Buffer{}, the timeout no longer works, and the command runs forever.
Why does this happen?
Here is a minimal reproducible example:
package main
import (
"bytes"
"context"
"os/exec"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
cmdArgs := []string{"sh", "-c", "sleep infinity"}
bufferOutputs := true
// Uncommenting *either* of the next two lines will make the issue go away:
// cmdArgs = []string{"sleep", "infinity"}
// bufferOutputs = false
cmd := exec.CommandContext(ctx, cmdArgs[0], cmdArgs[1:]...)
if bufferOutputs {
cmd.Stdout = &bytes.Buffer{}
}
_ = cmd.Run()
}
I've tagged this question with Linux because I've only verified that this happens on Ubuntu 20.04 and I'm not sure whether it would reproduce on other platforms.
My issue was that the child sleep process was not being killed when the context timed out. The sh parent process was being killed, but the child sleep was being left around.
This would normally still allow the cmd.Wait() call to succeed, but the problem is that cmd.Wait() waits for both the process to exit and for outputs to be copied. Because we've assigned cmd.Stdout, we have to wait for the read-end of the sleep process' stdout pipe to close, but it never closes because the process is still running.
In order to kill child processes, we can instead start the process as its own process group leader by setting the Setpgid bit, which will then allow us to kill the process using its negative PID to kill the process as well as any subprocesses.
Here is a drop-in replacement for exec.CommandContext I came up with that does exactly this:
type Cmd struct {
ctx context.Context
*exec.Cmd
}
// NewCommand is like exec.CommandContext but ensures that subprocesses
// are killed when the context times out, not just the top level process.
func NewCommand(ctx context.Context, command string, args ...string) *Cmd {
return &Cmd{ctx, exec.Command(command, args...)}
}
func (c *Cmd) Start() error {
// Force-enable setpgid bit so that we can kill child processes when the
// context times out or is canceled.
if c.Cmd.SysProcAttr == nil {
c.Cmd.SysProcAttr = &syscall.SysProcAttr{}
}
c.Cmd.SysProcAttr.Setpgid = true
err := c.Cmd.Start()
if err != nil {
return err
}
go func() {
<-c.ctx.Done()
p := c.Cmd.Process
if p == nil {
return
}
// Kill by negative PID to kill the process group, which includes
// the top-level process we spawned as well as any subprocesses
// it spawned.
_ = syscall.Kill(-p.Pid, syscall.SIGKILL)
}()
return nil
}
func (c *Cmd) Run() error {
if err := c.Start(); err != nil {
return err
}
return c.Wait()
}

How to ptrace multi-process or mutli-thread application from Go

I am attempting to write a program that would print all of the syscalls a program makes. I am having trouble extending this code to work with multi-processing scripts. I started off with code from https://github.com/lizrice/strace-from-scratch, and now I would like to trace offspring processes as well.
I tried adding the options PTRACE_O_TRACEVFORK | PTRACE_O_TRACEFORK | PTRACE_O_TRACECLONE, but this causes the process to hang for some reason. I do not know why. If I do not specify these options, then the process runs to completion, but of course children are not traced.
package main
import (
"fmt"
seccomp "github.com/seccomp/libseccomp-golang"
"golang.org/x/sys/unix"
"log"
"os"
"os/exec"
"runtime"
)
func init() {
runtime.LockOSThread()
}
func main() {
var err error
if len(os.Args) < 2 {
log.Fatalf("usage: ./trace-files program [arg]...")
}
cmd := exec.Command(os.Args[1], os.Args[2:]...)
cmd.Stderr = os.Stderr
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.SysProcAttr = &unix.SysProcAttr{
Ptrace: true,
}
if err = cmd.Start(); err != nil {
log.Fatalf("error starting command: %s\n", err)
}
if err = cmd.Wait(); err != nil {
// We expect "trace/breakpoint trap" here.
fmt.Printf("Wait returned: %s\n", err)
}
pid := cmd.Process.Pid
exit := true
var regs unix.PtraceRegs
var status unix.WaitStatus
// TODO: Setting these options causes the multiprocessing Python script to hang.
ptraceOptions := unix.PTRACE_O_TRACEVFORK | unix.PTRACE_O_TRACEFORK | unix.PTRACE_O_TRACECLONE
if err = unix.PtraceSetOptions(pid, ptraceOptions); err != nil {
log.Fatalf("error setting ptrace options: %s", err)
}
fmt.Println("pid\tsyscall")
for {
if exit {
err = unix.PtraceGetRegs(pid, &regs)
if err != nil {
break
}
name, err := seccomp.ScmpSyscall(regs.Orig_rax).GetName()
if err != nil {
fmt.Printf("error getting syscall name for orig_rax %d\n", regs.Orig_rax)
}
fmt.Printf("%d\t%s\n", pid, name)
}
if err = unix.PtraceSyscall(pid, 0); err != nil {
log.Fatalf("error calling ptrace syscall: %s\n", err)
}
// TODO: is it OK to overwrite pid here?
pid, err = unix.Wait4(pid, &status, 0, nil)
if err != nil {
log.Fatalf("error calling wait")
}
exit = !exit
}
}
For testing purposes, I have written a Python script that uses multiprocessing and prints the process IDs that it spawned.
import multiprocessing
import os
def fun(x):
return os.getpid()
if __name__ == "__main__":
print("PYTHON: starting multiprocessing pool")
with multiprocessing.Pool() as pool:
processes = pool.map(fun, range(1000000))
print("PYTHON: ended multiprocessing pool")
processes = map(str, set(processes))
print("PYTHON: process IDs: ", ", ".join(processes))
When I run the Go code above on a single process program, like ls, then things seems to work fine.
go run . ls
But when I run the Go code on the Python script, then the output hangs (but only if I supply the ptrace options I mentioned above).
go run . python script.py
My end goal for this program is to get a list of all of the files a program uses. I will inspect /proc/PID/maps for each syscall for that part, but first I would like to know how to trace multi-process programs. I tried looking through the documentation and code for strace, but that confused me further...

does /proc/[pid]/stat in all linux distributions always available?

I want to find the best universal way to check if a process exist and is running on any linux.
In Unix / BSD I am available to do this via kqueue thanks to the syscall.Kqueue() using EVFILT_PROC / NOTE_EXIT it doesn't' matter if is a mac os X, netbsd, freebsd, etc the code will just work and help to monitor the status of a PID.
Trying to achieve the same on linux, I came with the idea to check periodically for the existence of the /proc/[pid]/stat file, instead of sending a signal 0, kill -s 0 like suggested here: https://stackoverflow.com/a/15210305/1135424 mainly to simplify the logic due that non-nil error could be returned for existing processes.
Probably using something like:
initialStat, err := os.Stat(fmt.Sprintf("/proc/%d/stat", self.pid)
if err != nil {
return
}
for {
stat, err := os.Stat(fmt.Sprintf("/proc/%d/stat", self.pid)
if err != nil {
return err
}
if stat.Size() != initialStat.Size() || stat.ModTime() != initialStat.ModTime() {
return nil
}
// wondering how to avoid sleeping here
time.Sleep(time.Second)
}
But wondering if in all linux /proc/[pid]/stat is always available or if by sending signal 0 kill -0 $PID is basically doing exactly the same thing.
At the end I can always fallback to kill -O $PID but just wondering what posible solutions could be used (probably inotify) with the intention to avoid having to sleep on the loop mainly for not consuming to much CPU resources.
For child processes, you should use waitpid.
There's a netlink API for process events (http://netsplit.com/the-proc-connector-and-socket-filters) which might be usable for your case:
http://godoc.org/github.com/cloudfoundry/gosigar/psnotify#PROC_EVENT_EXIT
import "github.com/cloudfoundry/gosigar/psnotify"
func waitpid(pid int) error {
watcher, err := psnotify.NewWatcher()
if err != nil {
return err
}
if err := watcher.Watch(pid, psnotify.PROC_EVENT_EXIT); err != nil {
return err
}
defer watcher.Close()
// At this point, you should probably syscall.Kill(pid, 0) to check that the process is still running.
for {
select {
case ev := <-watcher.Error:
// TODO..
return ev
case <-watcher.Exit:
return nil
}
}
}

os.Process.Wait() after os.FindProcess(pid) works on windows not on linux

I have an issue when trying to recover a process in go. My go app launch a bunch of processes and when it crashes the processes are out there in the open and when I rerun my app I want to recover my processes. On windows everything works as expected I can wait() on the process kill() it etc.. but in linux it just goes trough my wait() without any error.
Here is the code
func (proc *process) Recover() {
pr, err := os.FindProcess(proc.Cmd.Process.Pid)
if err != nil {
return
}
log.Info("Recovering " + proc.Name + proc.Service.Version)
Processes.Lock()
Processes.Map[proc.Name] = proc
Processes.Unlock()
proc.Cmd.Process = pr
if proc.Service.Reload > 0 {
proc.End = make(chan bool)
go proc.KillRoutine()
}
proc.Cmd.Wait()
if proc.Status != "killed" {
proc.Status = "finished"
}
proc.Time = time.Now()
channelProcess <- proc
//confirmation that process was killed
if proc.End != nil {
proc.End <- true
}
}
process is my own struct to handle processes the important part is cmd which is from the package "os/exec" I have also tried to directly call pr.wait() with the same issue
You're not handing the error message from Wait. Try:
ps, err := proc.Cmd.Wait()
if err != nil {
/* handle it */
}
Also the documentation says:
Wait waits for the Process to exit, and then returns a ProcessState
describing its status and an error, if any. Wait releases any
resources associated with the Process. On most operating systems, the
Process must be a child of the current process or an error will be
returned.
In your case since you're "recovering", your process is not the parent of the processes you found using os.FindProcess.
So why does it work on windows? I suspect it is because on windows it boils down to WaitForSingleObject which doesn't have that requirement.

What's the optimal way to execute a nodejs script from golang, that returns a string, and then communicate that string back to a golang variable?

I'm currently doing it with os/exec and Stdout on golang's side, and console.log("string") on nodejs's side.
Basically I need to generate a string but can only do so within nodejs but the majority of my code is in golang, so I'm trying to make this little blip in my code as seamless, secure, and reliable as possible and I'm a little uneasy about resting such an important part of my program on "console.log" and reading from shell output.
In short: I'm wondering if there exists a better and more standard communication line between my node and go code then console.log + shell output, or is that perhaps optimal enough?
Oh and the function of this particular part of my program is to take a markdown text file and convert it to HTML using markdown-it.
Some ideas:
Communicate through HTTP (send the data/string to a golang http listener)
Communicate through the filesystem (write the string to a temporary file and read it with golang)
Communicate through "something similiar to HTTP but specific to local application data sharing"
P.S.
I can't use otto, since markdown-it doesn't run there.
Actual code:
parser.go
package main
import (
"os"
"os/exec"
"fmt"
"bytes"
)
func main() {
cmd := "node"
args := []string{"parser.js", "/home/user1/dev/current/wikis/Bob's Pain/markup/index.md"}
process := exec.Command(cmd, args...)
stdin, err := process.StdinPipe()
if err != nil {
fmt.Println(err)
}
defer stdin.Close()
buf := new(bytes.Buffer) // THIS STORES THE NODEJS OUTPUT
process.Stdout = buf
process.Stderr = os.Stderr
if err = process.Start(); err != nil {
fmt.Println("An error occured: ", err)
}
process.Wait()
fmt.Println("Generated string:", buf)
}
parser.js
var md = require('markdown-it')();
var yaml = require('js-yaml');
var fs = require('fs');
if (process.argv.length < 3) {
console.log('Usage: node ' + process.argv[1] + ' FILENAME');
process.exit(1);
}
var filename = process.argv[2];
fs.readFile(filename, 'utf8', function(err, data) {
if (err) {
throw err;
}
parse(data)
});
function parse(data) {
data = data.split("---")
yamlData = data[1];
markData = data[2];
y = yamlProcess(yamlData);
markData = "# "+y.title+"\n\n"+markData
html = markdownToHTML(markData);
console.log(html) // SEND THE DATA BACK TO GOLANG
}
function yamlProcess(data) {
try {
var doc = yaml.safeLoad(data);
return doc;
} catch (e) {
console.log(e);
return {};
}
}
function markdownToHTML(data) {
return md.render(data);
}
The easiest way to do this with os/exec:
command := "node parser.js /path/to/some/file.md"
parts := strings.Fields(command)
data, err := exec.Command(parts[0], parts[1:]...).Output()
if err != nil {
panic(err)
}
output := string(data)
"output" is the data that is printed from your NodeJS script. "command" is any command as a string.
I've approached similar requirement both ways.
For a build pipeline extension, I'd write a Python script that takes arguments from command line and outputs results to stdout. It's a simple interface, for a "run once", "everything succeeds otherwise fail fast" usage. If that's the same in your case, I'd keep the implementation as-is.
For a web application, I had Java service for just a specific function (in my case, Natty date recognition from a natural language string). This has the benefit that the application is already "warmed up" at the time of the call, and will definitely respond faster rather than booting up each time the request comes in. Going with a rest interface will probably reveal more benefits over time - e.g. simpler client implementation, monitoring, deployment options, switch implementation without changing clients, etc. It's just more conventional this way.

Resources