As far as I understood from the books and bash manuals is that. When a user logs out from bash all the background jobs that is started by the user will automatically terminate, if he is not using nohup or disown. But today I tested it :
Logged in to my gnome desktop and accessed gnome-terminal.
There are two tabs in the terminal and in one I created a new user called test and logged in as test
su - test
started a script.
cat test.sh
#!/bin/bash
sleep 60
printf "hello world!!"
exit 0
./test.sh &
After that I logged out of test and closed the tab
In the next tab I exected ps aux as root and found that job is still running.
How this is happening ?
Whether running background jobs are terminated on exit depends on the shell. Bash normally does not do this, but can be configured to for login shells (shopt -s huponexit). In any case, access to the tty is impossible after the controlling process (such as a login shell) has terminated.
Situations that do always cause SIGHUP include:
Anything in foreground when the tty is closed down.
Any background job that includes stopped processes when their shell terminates (SIGCONT and SIGHUP). Shells typically warn you before letting this happen.
huponexit summary:
On: Background jobs will be terminated with SIGHUP when shell exits
$ shopt -s huponexit
$ shopt huponexit
huponexit on
Off: Background jobs will NOT be terminated with SIGHUP when shell exits.
$ shopt -u huponexit
$ shopt huponexit
huponexit off
Only interactive shells kill jobs when you close them. Other shells (for example those you get by using su - username) don't do that. And interactive shells only kill direct subprocesses.
Related
The problem i encountered is pretty simple. I tried spawning bash via runuser, after using root user for some configuration. But when I run runuser -l user -c 'bash', and then press CRTL+C, instead of interrupting whatever command I ran inside of the bash, it kills the rununser shell Session terminated, killing shell....
How can I create interactive shell without (ideally using runuser) without it being killed by CTRL+C?
I am trying to use the "nohup" command to avoid killing a background process when exiting the terminal on linux MATE.
The process I want to run is a MPIrun process and I use the following command:
nohup mpirun -np 8 solverName -parallel >log 2>&1
when I leave the terminal, the processes running on the different cores are killed.
Also another thing I remarked in the log file, is that if I try to just run the following command
mpirun -np 8 solverName -parallel >log 2>&1
and then to CTRL+Z (stopping the process) the log file indicates :
Forwarding signal 20 to job
and I am unable to actually stop the mpirun command. So I guess there is something I don't understand in what I am doing
The job run in the background is still owned by your login shell (the nohup command doesn't exit until the mpirun command terminates), so it gets signalled when you disconnect. This script (I call it bk) is what I use:
#!/bin/sh
#
# #(#)$Id: bk.sh,v 1.9 2008/06/25 16:43:25 jleffler Exp $"
#
# Run process in background
# Immune from logoffs -- output to file log
(
echo "Date: `date`"
echo "Command: $*"
nice nohup "$#"
echo "Completed: `date`"
echo
) >>${LOGFILE:=log} 2>&1 &
(If you're into curiosities, note the careful use of $* and "$#". The nice runs the job at a lower priority when I'm not there. And version 1.1 was checked into version control — SCCS at the time — on 1987-08-10.)
For your process, you'd run:
$ bk mpirun -np 8 solverName -parallel
$
The prompt returns almost immediately. The key differences between what is in that code and what you do direct from the command line are:
There's a sub-process for the shell script, which terminates promptly.
The script itself runs the command in a sub-shell in background.
Between them, these mean that the process is not interfered with by your login shell; it doesn't know about the grandchild process.
Running direct on the command line, you'd write:
(nohup mpirun -np 8 solverName -parallel >log 2>&1 &)
The parentheses start a subshell; the sub-shell runs nohup in the background with I/O redirection and terminates. The continuing command is a grandchild of your login shell and is not interfered with by your login shell.
I'm not an expert in mpirun, never having used it, so there's a chance it does something I'm not expecting. My impression from the manual page is that it acts more or less like a regular process even though it can run multiple other processes, possibly on multiple nodes. That is, it runs the other processes but monitors and coordinates them and only exits when its children are complete. If that's correct, then what I've outlined is accurate enough.
To kill the process you need the following command.
first:
$ jobs -l
this gives you the PID of the process like this
[1]+ 47274 Running nohup mpirun -np 8 solverName -parallel >log 2>&1
then execute the following command to kill the process.
kill -9 {program PID i.e 47274 }
this will help you with killing the process.
note that ctrl+Z does not kill the process but it suspends it.
for the first part of the question, I recommend to try this command and see if it works or not.
nohup nohup mpirun -n 8 --your_flags ./compited_solver_name > Output.txt &
it worked for me.
tell us if it doesn't work for you.
I have an embedded system, on which I do telnet and then I run an application in background:
./app_name &
Now if I close my terminal and do telnet from other terminal and if I check then I can see this process is still running.
To check this I have written a small program:
#include<stdio.h>
main()
{
while(1);
}
I ran this program in my local linux pc in background and I closed the terminal.
Now, when I checked for this process from other terminal then I found that this process was also killed.
My question is:
Why undefined behavior for same type of process?
On which it is dependent?
Is it dependent on version of Linux?
Who should kill jobs?
Normally, foreground and background jobs are killed by SIGHUP sent by kernel or shell in different circumstances.
When does kernel send SIGHUP?
Kernel sends SIGHUP to controlling process:
for real (hardware) terminal: when disconnect is detected in a terminal driver, e.g. on hang-up on modem line;
for pseudoterminal (pty): when last descriptor referencing master side of pty is closed, e.g. when you close terminal window.
Kernel sends SIGHUP to other process groups:
to foreground process group, when controlling process terminates;
to orphaned process group, when it becomes orphaned and it has stopped members.
Controlling process is the session leader that established the connection to the controlling terminal.
Typically, the controlling process is your shell. So, to sum up:
kernel sends SIGHUP to the shell when real or pseudoterminal is disconnected/closed;
kernel sends SIGHUP to foreground process group when the shell terminates;
kernel sends SIGHUP to orphaned process group if it contains stopped processes.
Note that kernel does not send SIGHUP to background process group if it contains no stopped processes.
When does bash send SIGHUP?
Bash sends SIGHUP to all jobs (foreground and background):
when it receives SIGHUP, and it is an interactive shell (and job control support is enabled at compile-time);
when it exits, it is an interactive login shell, and huponexit option is set (and job control support is enabled at compile-time).
See more details here.
Notes:
bash does not send SIGHUP to jobs removed from job list using disown;
processes started using nohup ignore SIGHUP.
More details here.
What about other shells?
Usually, shells propagate SIGHUP. Generating SIGHUP at normal exit is less common.
Telnet or SSH
Under telnet or SSH, the following should happen when connection is closed (e.g. when you close telnet window on PC):
client is killed;
server detects that client connection is closed;
server closes master side of pty;
kernel detects that master pty is closed and sends SIGHUP to bash;
bash receives SIGHUP, sends SIGHUP to all jobs and terminates;
each job receives SIGHUP and terminates.
Problem
I can reproduce your issue using bash and telnetd from busybox or dropbear SSH server: sometimes, background job doesn't receive SIGHUP (and doesn't terminate) when client connection is closed.
It seems that a race condition occurs when server (telnetd or dropbear) closes master side of pty:
normally, bash receives SIGHUP and immediately kills background jobs (as expected) and terminates;
but sometimes, bash detects EOF on slave side of pty before handling SIGHUP.
When bash detects EOF, it by default terminates immediately without sending SIGHUP. And background job remains running!
Solution
It is possible to configure bash to send SIGHUP on normal exit (including EOF) too:
Ensure that bash is started as login shell. The huponexit works only for login shells, AFAIK.
Login shell is enabled by -l option or leading hyphen in argv[0]. You can configure telnetd to run /bin/bash -l or better /bin/login which invokes /bin/sh in login shell mode.
E.g.:
telnetd -l /bin/login
Enable huponexit option.
E.g.:
shopt -s huponexit
Type this in bash session every time or add it to .bashrc or /etc/profile.
Why does the race occur?
bash unblocks signals only when it's safe, and blocks them when some code section can't be safely interrupted by a signal handler.
Such critical sections invoke interruption points from time to time, and if signal is received when a critical section is executed, it's handler is delayed until next interruption point happens or critical section is exited.
You can start digging from quit.h in the source code.
Thus, it seems that in our case bash sometimes receives SIGHUP when it's in a critical section. SIGHUP handler execution is delayed, and bash reads EOF and terminates before exiting critical section or calling next interruption point.
Reference
"Job Control" section in official Glibc manual.
Chapter 34 "Process Groups, Sessions, and Job Control" of "The Linux Programming Interface" book.
When you close the terminal, shell sends SIGHUP to all background processes – and that kills them. This can be suppressed in several ways, most notably:
nohup
When you run program with nohup it catches SIGHUP and redirect program output.
$ nohup app &
disown
disown tells shell not to send SIGHUP
$ app &
$ disown
Is it dependent on version of linux?
It is dependent on your shell. Above applies at least for bash.
AFAIK in both cases the process should be killed. In order to avoid this you have to issue a nohup like the following:
> nohup ./my_app &
This way your process will continue executing. Probably the telnet part is due to a BUG similar to this one:
https://bugzilla.redhat.com/show_bug.cgi?id=89653
In order completely understand whats happening you need to get into unix internals a little bit.
When you are running a command like this
./app_name &
The app_name is sent to background process group. You can check about unix process groups here
When you close bash with normal exit it triggers SIGHUP hangup signal to all its jobs. Some information on unix job control is here.
In order to keep your app running when you exit bash you need to make your app immune to hangup signal with nohup utility.
nohup - run a command immune to hangups, with output to a non-tty
And finally this is how you need to do it.
nohup app_name & 2> /dev/null;
In modern Linux--that is, Linux with systemd--there is an additional reason this might happen which you should be aware of: "linger".
systemd kills processes left running from a login shell, even if the process is properly daemonized and protected from HUP. This is the default behavior in modern configurations of systemd.
If you run
loginctl enable-linger $USER
you can disable this behavior, allowing background processes to keep running. The mechanisms covered by the other answers still apply, however, and you should also protect your process against them.
enable-linger is permanent until it is re-disabled. You can check it with
ls /var/lib/systemd/linger
This may have files, one per username, for users who have enable-linger. Any user listed in the directory has the ability to leave background processes running at logout.
I use this command in linux terminal to connect to a server and use it as proxy :
ssh -N -D 7070 root#ip_address
it's get the password and connect and everything is Ok but how can I put this process in background ?
I used CTRL+Z but it stop not put this process in background ...
CTRL-Z is doing exactly what it should, which is stop the process. If you then want to put it in the background, the shell command for doing that is bg:
$ ssh -N -D 7070 -l user 192.168.1.51
user#192.168.1.51's password:
^Z
[1]+ Stopped ssh -N -D 7070 -l mjfraioli 192.168.1.51
$ bg
[1]+ ssh -N -D 7070 -l user 192.168.1.51 &
That way you can enter the password interactively, and only once that is complete, stop it and put it into the background.
Try adding an ampersand to the end of your command:
ssh -N -D 7070 root#ip_address &
Explanation:
This trailing ampersand directs the shell to run the command in the background, that is, it is forked and run in a separate sub-shell, as a job, asynchronously. The shell will immediately return the return status of 0 for true and continue as normal, either processing further commands in a script or returning the cursor focus back to the user in a Linux terminal.
The shell will print out the forked process’s job number and process ID (PID) like so:
$ ./myscript.py &
[1] 1337
The stdout of the forked process will still be attached to the parent, so any output will still appear in your terminal.
After a process is forked using a single trailing ampersand &, its process ID (PID) is stored in a special variable $!. This can be used later to refer to the process:
$ echo $!
1337
Once a process is forked, it can be seen in the jobs list:
$ jobs
[1]+ Running ./myscript.py &
And it can be brought back to the command line before it finishes with the foreground command:
fg
The foreground command takes an optional argument of the job number, if you have forked multiple processes.
A single ampersand & can also delimit a list of commands to be run asynchronously.
./script.py & ./script2.py & ./script3.py &
In this example, all 3 python scripts are run at the same time, in separate sub-shells. Their stdout will still be attached to the parent shell, so if running this from a Linux terminal, you will still see the outputs.
This can also be used as a quick hack to take advantage of multiple cores with shell scripts, but be warned, it is a hack!
To detach a process completely from the shell, you may want to pipe the stdout and stderr to a file or to /dev/null. A nice way of doing this is with the nohup command.
source for above explanation: http://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html
You can add option -f to make the ssh command run in background.
So the answer is ssh -f -D port username#hostname -N.
a strange thing to me
a script while.sh,it's content is:
while [ 1 ];do
sleep 1
echo `date`
done
run as $while.sh >& while.log & (without nohup or disown or setsid or double fork())
exit and login again can see this process is still exist，it's ppid is 1 and it's tty is ?
my system is rhel6(rhel5 is the same, bash
in centos5.x it must use nohup or disown or do double fork() in code
what happen in rhel6
Is the huponexit shell option set?
$ shopt
...
huponexit off
Bash will send a SIGHUP signal to its jobs if it receives a SIGHUP itself, but it won't signal them when it exits normally unless you enable this option.
For what it's worth this is disabled on both RHEL6 and RHEL5, at least on the systems I just tested. I tried this command:
$ sleep 1000 &
It was not killed when I logged out and logged back in unless I deliberately enabled shopt -s huponexit.