Is there a simple way I can completely pause an unprivileged docker container from the inside while retaining the ability to unpause/exec it from the outside?
TL;DR;
On a linux container, the answer is definitely, yes, because those two are equivalent:
From host:
docker pause [container-id]
From the container:
kill -SIGSTOP [process(es)-id]
or, even shorter
kill -SIGSTOP -1
Mind that:
If your process ID, or PID is 1, then you fall under an edge case, because PID 1, the init process, do have a specific meaning and behaviour in Linux.
Some processes might spawn child worker, as the NGINX example below.
And those two are also equivalent:
From host:
docker unpause [container-id]
From the container:
kill -SIGCONT [process(es)-id]
or, even shorter
kill -SIGCONT -1
Also mind that, in some edges cases, this won't work. The edge cases being that your process is meant to catch those two signals, SIGSTOP and SIGCONT, and ignore them.
In those cases, you will have to
either, be a privileged user, because the use of the cgroup freezer is under a folder, that is per default, read only in Docker, and probably this will end you in a dead end, because you will not be able to jump in the container anymore.
or, run your container with the flag --init so the PID 1 will just be a wrapper process initialised by Docker and you won't need to pause it anymore in order to pause the processes running inside your container.
You can use the --init flag to indicate that an init process should be used as the PID 1 in the container. Specifying an init process ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container.
The default init process used is the first docker-init executable found in the system path of the Docker daemon process. This docker-init binary, included in the default installation, is backed by tini.
This is definitely possible for Linux containers, and is explained, somehow, in the documentation, where they point out that running docker pause [container-id] just means that Docker will use an equivalent mechanism to sending the SIGSTOP signal to the process run in your container.
The docker pause command suspends all processes in the specified containers. On Linux, this uses the freezer cgroup. Traditionally, when suspending a process the SIGSTOP signal is used, which is observable by the process being suspended. With the freezer cgroup the process is unaware, and unable to capture, that it is being suspended, and subsequently resumed. On Windows, only Hyper-V containers can be paused.
See the freezer cgroup documentation for further details.
Source: https://docs.docker.com/engine/reference/commandline/pause/
Here would be an example on an NGINX Alpine container:
### For now, we are on the host machine
$ docker run -p 8080:80 -d nginx:alpine
f444eaf8464e30c18f7f83bb0d1bd07b48d0d99f9d9e588b2bd77659db520524
### Testing if NGINX answers, successful
$ curl -I -m 1 http://localhost:8080/
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Sun, 28 Jun 2020 11:49:33 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 26 May 2020 15:37:18 GMT
Connection: keep-alive
ETag: "5ecd37ae-264"
Accept-Ranges: bytes
### Jumping into the container
$ docker exec -ti f7a2be0e230b9f7937d90954ef03502993857c5081ab20ed9a943a35687fbca4 ash
### This is the container, now, let's see the processes running
/ # ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID VSZ RSS TT STAT TIME RUSER COMMAND
1 6000 4536 ? S 0:00 root nginx: master process nginx -g daemon off;
29 6440 1828 ? S 0:00 nginx nginx: worker process
30 6440 1828 ? S 0:00 nginx nginx: worker process
31 6440 1828 ? S 0:00 nginx nginx: worker process
32 6440 1828 ? S 0:00 nginx nginx: worker process
49 1648 1052 136,0 S 0:00 root ash
55 1576 4 136,0 R 0:00 root ps -o pid,vsz,rss,tty,stat,time,ruser,args
### Now let's send the SIGSTOP signal to the workers of NGINX, as docker pause would do
/ # kill -SIGSTOP 29 30 31 32
### Running ps again just to observer the T (stopped) state of the processes
/ # ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID VSZ RSS TT STAT TIME RUSER COMMAND
1 6000 4536 ? S 0:00 root nginx: master process nginx -g daemon off;
29 6440 1828 ? T 0:00 nginx nginx: worker process
30 6440 1828 ? T 0:00 nginx nginx: worker process
31 6440 1828 ? T 0:00 nginx nginx: worker process
32 6440 1828 ? T 0:00 nginx nginx: worker process
57 1648 1052 136,0 S 0:00 root ash
63 1576 4 136,0 R 0:00 root ps -o pid,vsz,rss,tty,stat,time,ruser,args
/ # exit
### Back on the host to confirm NGINX doesn't answer anymore
$ curl -I -m 1 http://localhost:8080/
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
$ docker exec -ti f7a2be0e230b9f7937d90954ef03502993857c5081ab20ed9a943a35687fbca4 ash
### Sending the SIGCONT signal as docker unpause would do
/ # kill -SIGCONT 29 30 31 32
/ # ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID VSZ RSS TT STAT TIME RUSER COMMAND
1 6000 4536 ? S 0:00 root nginx: master process nginx -g daemon off;
29 6440 1828 ? S 0:00 nginx nginx: worker process
30 6440 1828 ? S 0:00 nginx nginx: worker process
31 6440 1828 ? S 0:00 nginx nginx: worker process
32 6440 1828 ? S 0:00 nginx nginx: worker process
57 1648 1052 136,0 S 0:00 root ash
62 1576 4 136,0 R 0:00 root ps -o pid,vsz,rss,tty,stat,time,ruser,args 29 30 31 32
/ # exit
### Back on the host to confirm NGINX is back
$ curl -I http://localhost:8080/
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Sun, 28 Jun 2020 11:56:23 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 26 May 2020 15:37:18 GMT
Connection: keep-alive
ETag: "5ecd37ae-264"
Accept-Ranges: bytes
For the cases where the meaningful process is the PID 1 and so, is protected by the Linux kernel, you might want to try the --init flag at the run of your container so Docker will create a wrapper process that will be able to pass the signal to your application.
$ docker run -p 8080:80 -d --init nginx:alpine
e61e9158b2aab95007b97aa50bc77fff6b5c15cf3b16aa20a486891724bec6e9
$ docker exec -ti e61e9158b2aab95007b97aa50bc77fff6b5c15cf3b16aa20a486891724bec6e9 ash
/ # ps -o pid,vsz,rss,tty,stat,time,ruser,args
PID VSZ RSS TT STAT TIME RUSER COMMAND
1 1052 4 ? S 0:00 root /sbin/docker-init -- /docker-entrypoint.sh nginx -g daemon off;
7 6000 4320 ? S 0:00 root nginx: master process nginx -g daemon off;
31 6440 1820 ? S 0:00 nginx nginx: worker process
32 6440 1820 ? S 0:00 nginx nginx: worker process
33 6440 1820 ? S 0:00 nginx nginx: worker process
34 6440 1820 ? S 0:00 nginx nginx: worker process
35 1648 4 136,0 S 0:00 root ash
40 1576 4 136,0 R 0:00 root ps -o pid,vsz,rss,tty,stat,time,ruser,args
See how nginx: master process nginx -g daemon off; that was PID 1 in the previous use case became PID 7, now?
This ables us to kill -SIGSTOP -1 and be sure all meaningful processes are stopped, still we won't be locked out of the container.
While digging on this, I found this blog post that seems like a good read on the topic: https://major.io/2009/06/15/two-great-signals-sigstop-and-sigcont/
Also related it the ps manual page extract about process state code:
Here are the different values that the s, stat and state output
specifiers (header "STAT" or "S") will display to describe the state
of a process:
D uninterruptible sleep (usually IO)
I Idle kernel thread
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped by job control signal
t stopped by debugger during the tracing
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by
its parent
For BSD formats and when the stat keyword is used, additional
characters may be displayed:
< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom
IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL
pthreads do)
+ is in the foreground process group
Source https://man7.org/linux/man-pages/man1/ps.1.html#PROCESS_STATE_CODES
docker pause command from inside is not possible for an unprivileged container. It would need access to the docker daemon by mounting the socket.
You would need to build a custom solution. Just the basic idea: You could bindmount a folder from the host. Inside this folder you create a file which acts as a lock. So when you pause inside the container you would create the file. While the file exists you activly wait/sleep. As soon as the host would delete the file at path which was mounted, your code would resume. That is a rather naive approach because you actively wait, but it should do the trick.
You can also look into inotify to overcome activ waiting.
https://lwn.net/Articles/604686/
On my Windows machine, I started a Docker container from docker compose. My entrypoint is a Go filewatcher that runs a task of a taskmanager on every filechange. The executed task builds and runs the Go program.
But before I can build and run the program again after filechanges I have to kill the previous running version. But every time I kill the app process, the container is also gone.
The goal is to kill only the svc1 process with PID 74 in this example. I tried pkill -9 svc1 and kill $(pgrep svc1). But every time the parent processes are killed too.
The commandline output from inside the container:
root#bf073c39e6a2:/app/cmd/svc1# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 2.5 0.0 104812 2940 ? Ssl 13:38 0:00 /go/bin/watcher
root 13 0.0 0.0 294316 7576 ? Sl 13:38 0:00 /go/bin/task de
root 74 0.0 0.0 219284 4908 ? Sl 13:38 0:00 /svc1
root 82 0.2 0.0 18184 3160 pts/0 Ss 13:38 0:00 /bin/bash
root 87 0.0 0.0 36632 2824 pts/0 R+ 13:38 0:00 ps -aux
root#bf073c39e6a2:/app/cmd/svc1# ps -afx
PID TTY STAT TIME COMMAND
82 pts/0 Ss 0:00 /bin/bash
88 pts/0 R+ 0:00 \_ ps -afx
1 ? Ssl 0:01 /go/bin/watcher -cmd /go/bin/task dev -startcmd
13 ? Sl 0:00 /go/bin/task dev
74 ? Sl 0:00 \_ /svc1
root#bf073c39e6a2:/app/cmd/svc1# pkill -9 svc1
root#bf073c39e6a2:/app/cmd/svc1
Switching to the containerlog:
task: Failed to run task "dev": exit status 255
2019/08/16 14:20:21 exit status 1
"dev" is the name of the task in the taskmanger.
The Dockerfile:
FROM golang:stretch
RUN go get -u -v github.com/radovskyb/watcher/... \
&& go get -u -v github.com/go-task/task/cmd/task
WORKDIR /app
COPY ./Taskfile.yml ./Taskfile.yml
ENTRYPOINT ["/go/bin/watcher", "-cmd", "/go/bin/task dev", "-startcmd"]
I expect only the process with the target PID is killed and not the parent process that spawned it it.
You can use process manager like "supervisord" and configure it to re-execute your script or the command even if you killed it's process which will keep your container up and running.
Using Apache2 on Linux, I get this error message when trying to start it.
$ sudo /usr/local/apache2/bin/apachectl start
httpd not running, trying to start
(98)Address already in use: make_sock: unable to listen for connections on address 127.0.0.1:80
no listening sockets available, shutting down
Unable to open logs
$ sudo /usr/local/apache2/bin/apachectl stop
httpd (no pid file) not running
Some facts:
This is one of the last lines in my Apache logs:
[Mon Jun 19 18:29:01 2017] [warn] pid file /usr/local/apache2/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
My '/usr/local/apache2/conf/httpd.conf' contains
Listen 127.0.0.1:80
I have "Listen 80" configured at '/etc/apache2/ports.conf'
Disk is not full
I've checked that I do not have two or more "Listen" at '/usr/local/apache2/conf/httpd.conf'
Some outputs:
$ sudo ps -ef | grep apache
root 1432 1 0 17:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 1435 1432 0 17:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 1436 1432 0 17:35 ? 00:00:00 /usr/sbin/apache2 -k start
myuserr 1775 1685 0 17:37 pts/1 00:00:00 grep --color=auto apache
$ sudo grep -ri listen /etc/apache2
/etc/apache2/apache2.conf:# supposed to determine listening ports for incoming connections which can be
/etc/apache2/apache2.conf:# Include list of ports to listen on
/etc/apache2/ports.conf:Listen 80
/etc/apache2/ports.conf: Listen 443
/etc/apache2/ports.conf: Listen 443
What can I do to restart Apache? Should I repair 'httpd.pid'?
This error means that something already uses 80 port.
If you really don't have 2 line of Listen 80 in apache configurations then execute this command to see what uses 80 port: netstat -antp | grep 80.
I fixed it by killing the three processes
root 1621 1 0 18:46 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 1624 1621 0 18:46 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 1625 1621 0 18:46 ? 00:00:00 /usr/sbin/apache2 -k start
However, each time I want to reboot my server, I must kill thee processes. What is starting them?
Tell me please is there any way to resolve this issue without installing a login manager please?
I've enabled the auto login for startx, following the steps from the beyond link:
How to make auto login work in Ubuntu? (no display manager)
Auto login is functioning now.
On the device is a command line installation from Minimal Lubuntu 16.10 mini.iso. without any desktop, only the Kernel and some restricted modules. The only environment installed is fluxbox.
After booting in Fluxbox, I can't open gnome-terminal at all, until I will not do the next steps. xterm can start.
ctrl+alt+del in the running Fluxbox, it will redirect me for a second-two in tty, but because auto login is enable it will redirect me back automatically from tty1 to Fluxbox. So, in order to remain in tty I will keep pressing continuously ctrl+c.
Now, being in tty I will
sudo -i
su myusername
startx
Being again in fluxbox, I can run the terminal normally.
Do you please have any clues please, why I can't open the terminal without doing the above?
Trying to start gnome-terminal from xterm when at the first login.
gnome-terminal
Error constructing proxy for org.gnome.Terminal:/org/gnome/Terminal/Factory0: Error calling StartServiceByName for org.gnome.Terminal: Timeout was reached
Excuse me, I am not sure that DBUS_SESSION_BUS_ADDRESS is active or not.
/var/log/Xorg.0.log
Output of env from xterm (after logging manually again to startx)
TERM=xterm
SHELL=/bin/bash
WINDOWID=8388621
XTERM_SHELL=/bin/bash
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SUDO_USER=xdpsx
SUDO_UID=1000
USERNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
MAIL=/var/mail/root
PWD=/home/xdpsx
LANG=en_US.UTF-8
XTERM_LOCALE=en_US.UTF-8
XTERM_VERSION=XTerm(324)
HOME=/root
SUDO_COMMAND=/bin/su
SHLVL=2
LOGNAME=root
LESSOPEN=| /usr/bin/lesspipe %s
DISPLAY=:0.0
SUDO_GID=1000
LESSCLOSE=/usr/bin/lesspipe %s %s
XAUTHORITY=/home/xdpsx/.Xauthority
COLORTERM=truecolor
_=/usr/bin/env
Output of ps aux | grep dbus from xterm (after logging manually again to startx)
message+ 668 0.0 0.0 6420 3936 ? Ss 19:11 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
nobody 811 0.0 0.1 9316 4000 ? S 19:11 0:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --cache-size=0 --conf-file=/dev/null --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
xdpsx 1375 0.0 0.0 6136 3460 ? Ss 19:11 0:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation
xdpsx 1381 0.0 0.0 6136 3316 ? S 19:11 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
xdpsx 1505 0.0 0.0 7004 312 ? S 19:12 0:00 dbus-launch --autolaunch fedd8908d0d244c498876a97f5b34c28 --binary-syntax --close-stderr
xdpsx 1506 0.0 0.0 6136 3060 ? Ss 19:12 0:00 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
xdpsx 1529 0.0 0.0 6136 3356 ? S 19:14 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
root 1834 0.0 0.0 5144 828 pts/1 S+ 19:25 0:00 grep --color=auto dbus
Thank you.
I am seeing a similar problem since upgrading to Ubuntu 16.10. I discovered that I can fix the environment and start gnome-terminal like this:
dbus-update-activation-environment --systemd --all
gnome-terminal &
If that doesn't work, then you could try this:
dbus-launch gnome-terminal &
I'm new to linux.
How can I show a list of all processes that says about each process if it's running or suspended?
I've tried
ps -ef|grep myusername
but it doesn't say if the processes are running or not.
also tried
ps ux
same thing, it doesn't say if the processes are running or not.
I'm looking for something like this list:
I get this list when I move a process to background, I don't know how to see it otherwise...
You can use "ps" to list processes, This (ps aux) will list all the processes. Given an example output of it below.
ps aux | more
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 189160 9376 ? Ss 15:51 0:04 /usr/lib/systemd/systemd --switched-root --system --deserialize 20
root 2 0.0 0.0 0 0 ? S 15:51 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 15:51 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 15:51 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S 15:51 0:06 [rcu_sched]
root 8 0.0 0.0 0 0 ? S 15:51 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S 15:51 0:04 [rcuos/0]
By checking the STAT of the process ( UNDER "STAT" ) you can identify the process states, Below are some possible states codes.
R running or runnable (on run queue)
D uninterruptible sleep (usually IO)
S interruptible sleep (waiting for an event to complete)
Z defunct/zombie, terminated but not reaped by its parent
T stopped, either by a job control signal or because it is being
traced
You can type "man ps" to get more info.
You can use htop to see the list of processes and there is a column for process state
What does a C process status mean in htop?
http://www.howtogeek.com/howto/ubuntu/using-htop-to-monitor-system-processes-on-linux/
ps -p PID -o comm=
Enter the code above where PID is PID of the process.
Following command will be more helpful to you.
Use the command : sudo lsof -i -n -P
This command lists the Application Name, PID, User, IP version, Device ID and the Node with Port Name. It shows both TCP and UDP.
Variations :
To format it in a nice, readable way; use :
sudo lsof -i -n -P | more
To view view only TCP connections :
sudo lsof -i -n -P | grep TCP | more
To view view only UDP connections :
sudo lsof -i -n -P | grep UDP | more