I want to audit commands that are being run by a user inside a running pod.
I know that kube-apiserver supports audit policies that allows you to log every request that is being done towards the API but as far as I know the API audit only records the exec command and not the inner commands run afterwards.
An approach that I thought is to have a sidecar container with auditbeat running but it's too heavy and the user might be able to kill it.
The container should run a single process inside. It is not recommended to run a command inside container exception for testing. Most of our image doesn't have any type of shell.
If you have to spawn a shell and run a command inside, Then you need to think about is it possible to run that outside container? If the main process is terminated but your shell commands are running in a container then k8s might not terminate that pod and recreate a new pod which might impact HA
There are some commercial products that allow to do this. Few weeks ago I did a PoC for one of them.
The way it's implemented is that their product running as a pod (with 1 container inside) on the host level (host namespace / HostPID) and tracks usage of Docker daemon.
Related
I am working on a debian based container which runs a c++ compiled application as a child process to a node.js script. For testing, I am attempting to define a swarm that instantiates this container as a service. When I run "docker service ls", it reports that the service is no running. If I issue "docker ps", however, I can see the identifier for the container and I can use "docker exec" to connect to it.
So far as I can see, the two target processes are running within the container yet, when other containers within the same swarm and using the same networks attempt to connect to this server, I see a name resolution error.
Here is my question: If the intended processes are actively and verifiably running within a container, why would docker consider a service based on that container to not be running? What is Docker's criteria for considering a container to be running as a service?
The status of starting means that it's waiting for the HEALTHCHECK condition to be satisfied.
Examine your Dockerfile's (or compose file's) HEALTHCHECK conditions and investigate why they are not being satisfied.
NOTE: Do not confuse starting with restarting, which means that the program is being launched again (usually after a crash) due to the restart policy.
This question already has answers here:
Is it ok to run docker from inside docker?
(5 answers)
Closed 4 years ago.
We have app and which will spin the short term (short span) docker containers. Right now, it runs in Ubunut16.04 server (VM) and we installed docker, and nodejs in same server. We have nodejs app which runs in same server so whenever a request comes in, then the nodejs app will spin up a docker container and execute a user input inside the docker container. Once after the docker finish its job or if it runs out of admin defined resources then the docker container will be forcefully killed (docker kill) and removed (docker rm).
Now my question is, is it best practices to run the Ubunte16.04 docker container and run nodjes app and the short term docker containers inside the Ubunuter16.04 docker container.
In short run a docker inside other docker container.
Docker-in-Docker is generally considered fragile and hard to maintain and using it isn’t a best practice. https://hub.docker.com/_/docker/ has a little discussion on this.
A straightforward (but potentially dangerous) way to rearrange this is to give the server process access to the host’s Docker socket, with docker run -v /var/run/docker.sock:/var/run/docker.sock. Then it could launch its own Docker containers as needed. Note that if you do this, these sub-containers’ docker run -v options refer to the host’s filesystem, not the calling container’s filesystem, so if you’re trying to use the filesystem to transfer data this can get tricky. Also note that being able to run any Docker command this way gives unlimited access to the host, so you need to be extremely careful about how you launch containers.
A larger redesign would be to introduce some sort of message-queueing system; I’ve successfully used RabbitMQ in the past but there are many other options. Instead of the server process launching a subprocess directly, it writes a message to a queue. Instead of the workers being short-lived processes that start and stop frequently, you have a long-lived worker that reads jobs off the queue and does them. This puts you in a much more established Docker space where nothing needs to dynamically start and stop containers, and you can easily test the Node-Rabbit-worker stack in a non-Docker environment.
I have a simple question hopefully; I'm new to Docker and Linux.
Most of the articles/stackoverflow posts suggest installing cron INSIDE the docker container to get it to work as can be seen at this link
However, based on the picture attached below, we can see that the Docker Engine is an abstraction layer between the HOST OS's system and utility libraries and the application container.
Why are we not REUSING the system cron that comes with the HOST instead of installing cron INSIDE the container? It almost feels redundant.
My understanding of docker is you'd install application level libraries and packages like npm node modules INSIDE your nodejs app container for example but if you need a system utility like cron, then you would somehow call back out to the HOST OS's native cron utility; so why not use the HOST's cron within our container somehow, why reinstall cron inside the container?
Lastly, would you use docker-compose instead and separate out the cron service into its own container then somehow have the cron service talk to the application container and referencing its environment vars?
I mean the environment variables defined in the app container; making those accessible to the cron container?
So that we may follow best practice of one service per container?
Use cron on the host machine if you want, eg
0 0 * * * /usr/local/bin/docker run image
As far as I'm aware, modern container applications use some form of scheduling from the host (eco)system. You could use cron on the host machine to trigger docker run commands. You could use a general purpose scheduler like Airflow. You could use a full-fletched container platform like DC/OS, which come with built in scheduling services.
There is nothing wrong with running cron together with your application inside a container per se. However, if you trigger the application container from a scheduling service outside your application container, your container would terminate after the job is finished, thus releasing any resources to other applications.
Secondly, it's considered good practice to have one container per service. Cron is a service in itself.
I would like to use Supervisor to run multiple processes in my Docker container, as described here, in Docker docs.
It works but the doc does not say anything about what happens when one of the processes I start crashes.
Following docker behavior logic - when a process crashes - container should stop, and probably later it should be restarted by Docker according to restart policy.
But it does not happen, If one (or all) of application I start exits - container keeps working.
How can I tell Supervisor to exit (and stop the container in this way, because I run it in nodaemon=true mode) as well, when one of monitoring processes exits/crashes?
I found this article which describes that its sometimes valid to run multiple processes in one container.
He describes how to use honcho to create the behaviour you would like: stop the whole container when one of the processes fails.
I'am going to try this now, but I'm still a little bit in doubt because supervisord is used so much more in the docker world and is also described on their own site.
if you want to exit the container when your process stops, don't use supervisor (or any other process manager). just run the process in your container, directly.
but more importantly: don't run multiple critical applications in your container. the golden rule of Docker containers is not 1 process per container, but 1 concern per container. that way your container can properly shut down when that 1 concern (application) exits.
even in the example you cite, they are not running 2 critical processes. they are running 1 app process and then hosting sshd in the same container for ssh access. if sshd stops, it's probably not a big deal. if the apache server stops... well, they're using supervisor to handle that and automatically restart it.
to get what you want, separate your concerns into multiple containers and just run the app in the container directly.
I am trying to find the best way to automatically start services inside a docker container once it has been restarted.
I don't mean starting the docker container on restart. I'm trying to achieve the following way:
I stop a container; and
when I start it again, the same services (processes) I was running before will start up again.
I.e. if I am running apache and ssh inside the container starting those service on container restart
That's really not the docker way (multiple processes per container). You can try to go down that path, as I did for several months, but you'll find that you'll be going against the docker team's design principles most of the time. I used the phusion/baseimage base image and it really is well designed, with a good init process and support for run-it and ssh out of the box. Tread carefully, if you go down that path however.