I'm developing code for Linux, and cannot seem to kill processes when running in a Jenkins environment.
I have test script that spawns processes and cleans them up as it goes through the tests. One of the processes also spawns and cleans up one of its own subprocesses. All of the "cleanup" is done by sending a SIGINT, followed by a wait. Everything works fine with this when run from a terminal, except when running through Jenkins.
When the same exact thing is run in Jenkins, processes killed with SIGINT do not die, and the call to wait blocks forever. This wreaks havoc on my test. I could update the logic to not do a blocking wait, but I don't feel I should have to change my production code to accommodate Jenkins.
Any ideas?
Process tree killer may be your answer - https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller
In testing, this would usually work when I ran the tests from the command line, but would almost always fail when that unit test script was called from another script. Frankly, it was bizarre....
Then I realized that when I had stray processes, they would indeed go away when I killed them with SIGTERM. But WHY?????
I didn't find a 100%-definitive answer. But thinking about it logically, if the process is not attached to a terminal, then maybe the "terminal interrupt" signal (SIGINT), wouldn't work...?
In doing some reading, what I learned is that, basically, when it's a shell that executes a process, the SIGINT action may be set to 'ignore'. That make sense (to me, anyway) because you wouldn't want CTRL-C at the command line to kill all of your background processes:
When the shell executes a process “in the background” (or when another background process executes another process), the newly executed process should ignore the interrupt and quit characters. Thus, before a shell executes a background process, it should set SIGINT and SIGQUIT to SIG_IGN.
Our production code isn't a shell, but it is started from a shell, and Jenkins uses /bin/sh to run stuff. So, this would add up.
So, since there is an implied association between SIGINT and the existence of a TTY, SIGTERM is a better option for killing your own background processes:
It should be noted that SIGINT is nearly identical to SIGTERM. (1)
I've changed the code that kills the proxyserver processes, and the Python unit test code, to use the SIGTERM signal. Now everything runs at the terminal and in Jenkins.
Related
I noticed that CTRL+C doesn't work well with podman build when I'm running it from within tmux. When I hit CTRL+C I see a ^C appearing in the terminal, but podman doesn't stop building the image. I think it stops later at some point, but it does not do that immediately.
Why is that?
//EDIT
I noticed that building with docker build allows me to stop with CSTL+C without any issue.
the process with pid 1 that runs inside your pod intercepts signals such as SIGTERM and SIGINT (from pressing CTRL+C).
It then sends a SIGTERM to all the processes under its management (anything running in the pod), so those processess have some time to terminate cleanly/gracefully.
If there are still processes around after a certain amount of time (10 seconds i believe), the pid 1 process will send a SIGKILL to all remaining processes, which kills them instantly; after that the pid 1 process itself can exit and let the entire pid namespace expire.
Basically, the short version is that the podman init process is trying to cleanly power things down instead of just ripping out the powercord.
If it bothers you, i imagine the 10 second delay is configurable somewhere (i've never looked). Another alternative may be to send a stronger signal.
Background processes don't belong to a user and a terminal, nor do daemon processes. What is the main difference between the two? If I were to write a server program, should I run it as a background process or a daemon?
When one says 'background process', it's usually in the context of a shell (like bash), which implements job control.
When a process (or a process group) is put into the background, it's still part of the session created by the shell and will still have an association with the shell's controlling terminal. The standard input/output of a background process will still be linked to the terminal (unless explicitly changed). Also, depending on how the shell exits, it may send a SIGHUP signal to all the background processes (See this answer to know exactly when). Until the shell terminates, it remains the parent of the background process.
A daemon on the other hand does not have a controlling terminal and is usually explicitly made to be a child of the init process. The standard input/output of a dare usually redirected to /dev/null
A Background process usually refers to a process which:
Another process is its parent; eg, a shell;
It has standard streams (input, output, error) connected to that parent
The most common type is when you run a shell program with a trailing &. It generally shares the shell’s output streams, but will get a signal and stop if it tries to read from its input stream.
More significantly (usually), a background process like this is still parented, so signals to that process group will continue to it. If the parent process terminates, the children will receive signals that will most likely terminate them, as well. (This is probably the biggest difference between the two for most users.)
A Daemon process is one that:
Has no parent, ie, its parent process is the system (or container) initial thread, commonly systemd (Linux), init (other Unix), or launchd? (MacOS);
Typically has its output disconnected, or connected to a log file;
Typically has its input disconnected.
Daemons are usually also written to accept the “user hung up” signal (SIGHUP), which would terminate a program if not handled, as a special instruction to re-read their configuration files and continue working.
Most often, these are processes created by some system-level facility that continue to operate completely independently of user activity (logins, logouts, and the like). Things that, themselves, handle logins (getty or gdm and the like), as well as other network-facing services (web servers, mail servers, etc) may be daemons, as well as self-monitoring services like cron, or smartd.
I'm working with parallel processing and rather than dealing with cvars and locks I've found it's much easier to run a few commands in a shell script in sequence to avoid race conditions in one place. The new problem is that one of these commands calls another program, which the OS has decided to put into a new process. I need to kill this process from the parent program, but the parent program only knows the pid of the parent (shell script), so this process keeps executing on its own.
Is there a way in bash to set a subprocess to die when the parent dies? I've tried to figure out how to execute it as a daemon because I read daemons exit when the parent dies, but it's tricky and I can't quite get it right. Thanks!
Found the problem, and this fixed it (except for some pesky messages that somehow cannot be redirected to /dev/null).
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
I've been studying this link to finish my shell assignment: http://www.gnu.org/software/libc/manual/html_node/Launching-Jobs.html#Launching-Jobs and it's been particularly helpful. My confusion is that, to give the shell control of the stdin file descriptor again, I need to call tcsetpgrp from the shell after the child is terminated.
How do I get tcsetpgrp() to work in C?
I've searched different Stack Overflow questions, but none properly tell me why GNU promotes this approach. Because the shell currently is in the "background", tcsetpgrp() will send SIGTTOU to my process group. The current solution is to ignore it before calling the method and maybe reset it to default afterwards. What should I do?
EDIT: I would like to note that the child is first set in another process group before the shell passes control of stdin to it with tcsetpgrp(). Once the child dies, the shell calls tcsetpgrp() to reclaim stdin. GNU suggests this as a possible implementation, but says it uses a slightly different implementation for simplicity here.
If tcsetpgrp() is called by a member of a background process group in its session, and the calling process is not blocking or ignoring SIGTTOU, a SIGTTOU signal is sent to all members of this background process group.
I'm also towards the end of implementing a shell program, and can take a stab at this question:
TL;DR: even though the shell will be a background process at that point, it needs to reclaim the terminal foreground for itself once the most recent foreground process group exits; otherwise the terminal will hang and no process will consume the user's input.
Here's what the session looks like when a shell launches a job from a line of command:
before the child processes call execve(), they are assigned into a process group, so that it's simpler to manage them as a job;
now that the shell and the child processes are in different process group, and that the terminal can only serve one process group at a time, the shell has to donate the terminal foreground to this child process group, so that the child job could read inputs and receive signals directly from the user;
after the foreground donation, the shell becomes a background process, but since there's no point in running the shell in the background (its main function is to read/write to the terminal and launch jobs), it should not proceed until it reclaims the foreground;
another reason why the shell must immediately reclaim the foreground after the exit of the most recent child foreground process group, is that there will be no running process consuming the terminal input/signal between the time a) when the child foreground process group exit, and b) when the shell reclaims the foreground. The terminal will essentially hang and become unresponsive;
as for why the shell must reclaim the foreground itself, it's because the child processes will be calling execve() and loading entirely new process images. It is extremely difficult to enforce and advocate for a contract where every child process in the foreground returns the foreground back to the shell process;
lastly, since tcsetpgrp() sends the caller background process a SIGTTOU, the shell must register an ignorer handler for SIGTTOU, at least during the time it's reclaiming the terminal foreground;
The GNU C Library manual also has another section (28.5.4 Foreground and Background) that briefly explains why the shell must reclaim the terminal forg
I've read about fork and from what I understand, the process is cloned but which process? The script itself or the process that launched the script?
For example:
I'm running rTorrent on my machine and when a torrent completes, I have a script run against it. This script fetches data from the web so it takes a few seconds to complete. During this time, my rtorrent process is frozen. So I made the script fork using the following
my $pid = fork();
if ($pid == 0) { blah blah blah; exit 0; }
If I run this script from the CLI, it comes back to the shell within a second while it runs in the background, exactly as I intended. However, when I run it from rTorrent, it seems to be even slower than before. So what exactly was forked? Did the rtorrent process clone itself and my script ran in that, or did my script clone itself? I hope this makes sense.
The fork() function returns TWICE! Once in the parent process, and once in the child process. In general, both processes are IDENTICAL in every way, as if EACH one had just returned from fork(). The only difference is that in one, the return value from fork() is 0, and in the other it is non-zero (the PID of the child process).
So whatever process was running your Perl script (if it is an embedded Perl interpreter inside rTorrent then rTorrent would be the process) would be duplicated at exactly the point that the fork() happened.
I believe I found the problem by looking through rTorrent's source. For some processes, it will read all of the output sent to stdout before continuing. If this is happening to your process, rTorrent will block until you close the stdout process. Because you're forking, your child process shares the same stdout as the parent. Your parent process will exit, but the pipe remains open (because your child process is still running). If you did an strace of rTorrent, I'd bet that it'd be blocked on this read() call while executing your command.
Try closing/redirecting stdout in your perl script before the fork().
The entire process containing the interpreter forks. Fortunately memory is copy-on-write so it doesn't need to copy all the process memory in order to fork. However, things such as file descriptors remain open. This allows child processes to handle them, but may cause issues if they aren't closed appropriately. In general, fork() should not be used in an embedded interpreter except under extreme duress.
To answer the nominal question, since you commented that the accepted answer fails to do so, fork affects the process in which it is called. In your example of rTorrent spawning a Perl process which then calls fork, it is the Perl process which is duplicated, since it was the Perl process which called fork.
In the general case, there is no way for a process to fork any process other than itself. If it were possible to tell another arbitrary process to go fork itself, that would open up no end of security and performance issues.
My advice would be "don't do that".
If the Perl interpreter is embedded within the rtorrent process, you've almost certainly forked an entire rtorrent process, the effects of which are probably ill-defined at best. It's generally a bad idea to play with process-level stuff in an embedded interpreter regardless of language.
There's an excellent chance that some sort of lock is not being properly released, or that threads within the processes are proceeding in unintended and possibly competing ways.
When we create a process using fork the child process will have the copy of the address space.So the child also can use the address space.And it also can access the files which is opened by the parent.We can have the control over the child.To get the complete status of the child we can use wait.