I have been using docker for a couple of months now, and am working on dockerizing various different server images. One consistent issue is that many servers need to run cron jobs. There is a lot of discussion about that online (including on Stackoverflow), but I don't completely understand the mechanics of it.
Currently, I am using the host's cron and docker exec into each container to run a script. I created a convention about the script's name and location; all my containers have the same script. This avoids having the host's cron depending on the containers.
Basically, once a minute, the host's cron does this:
for each container
docker exec -it <containername> /cronscript/minute-script
That works, but makes the containers depend on the host.
What I would like to do is create a cron container that kicks off a script within each of the other containers - but I am not aware of an equivalent to "docker exec" that works from one container to the other.
The specific situations I have right now are running a backup in a MySQL container, and running the cron jobs Moodle requires to be run every minute. Eventually, there will be additional things I need to do via cron. Moodle uses command-line PHP scripts.
What is the "proper" dockerized way to kick off a script from one container in another container?
Update: maybe it helps to mention my specific use cases, although there will be more as time goes on.
Currently, cron needs to do the following:
Perform a database dump from MySQL. I can do that via mysqldump TCP link from a cron container; the drawback here is that I can't limit the backup user to host 127.0.0.1. I might also be able to somehow finagle the MySQL socket into the cron container via a volume.
Perform regular maintenance on a Moodle installation. Moodle includes a php command line script that runs all of the maintenance tasks. This is the biggie for me. I can probably run this script through a volume, but Moodle was not designed with that situation in mind, and I would not rule out race conditions. Also, I do not want my moodle installation in a volume because it makes updating the container much harder (remember that in Docker, volumes are not reinitialized when you update the container with a new image).
Future: perform routine maintenance on a number of other of my servers, such as cleaning out email queues, etc.
My solution is:
install crond inside container
install Your soft
run cron as a daemon
run Your soft
Part of my Dockerfile
FROM debian:jessie
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY .crontab /usr/src/app
# Set timezone
RUN echo "Europe/Warsaw" > /etc/timezone \
&& dpkg-reconfigure --frontend noninteractive tzdata
# Cron, mail
RUN set -x \
&& apt-get update \
&& apt-get install -y cron rsyslog mailutils --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
CMD rsyslogd && env > /tmp/crontab && cat .crontab >> /tmp/crontab && crontab /tmp/crontab && cron -f
Description
Set timezone, because cron need this to proper run tasks
Install cron package - package with cron daemon
Install rsyslog package to log cron task output
Install mailutils package if You want to send e-mails from cron tasks
Run rsyslogd
Copy ENV variables to tmp file, because cron run tasks with minimal ENV and You tasks may need access to containers ENV variables
Append Your .crontab file (with Your tasks) to tmp file
Set root crontab from tmp file
Run cron daemon
I use this in my containers and work very well.
one-process-per-container
If You like this paradigm, then make one Dockerfile per cron task. e.g.
Dockerfile - main program
Dockerfile_cron_task_1 - cron task 1
Dockerfile_cron_task_1 - cron task 2
and build all containers:
docker build -f Dockerfile_cron_task_1 ...
Related
I have an application written in PHP and I need to work with the host machine: create a Linux user, copy some files, etc.
Run this program as root outside a container on the host system.
One of the core features of Docker is that a container isn't normally allowed to, for example, reset the host root user's password by writing /etc/shadow, or more generally read or write the host filesystem at all. A container is similarly normally forbidden from other system-management tasks like changing the host network configuration. This filesystem isolation, keeping a container process from corrupting key system files, is a primary reason to use Docker at all, and it can't be trivially disabled.
So in particular, "create a user" is precisely the class of dangerous-to-the-host operations that a container process by design is forbidden from doing. More generally, "copy files" is merely hard, but a task whose main purpose is reading and writing host files will generally be much easier to run outside a container.
In theory you can accomplish some of this using bind mounts. For the "copy files" part of the task, in principle you can run something like
docker run --rm \
-v "$PWD/here:/input" \
-v "$PWD/there:/output" \
your-image
and in the container, /input and /output will be the host's ./here and ./there directories.
It's possible to mount the entire host filesystem, -v /:/host for example. You could in theory use this to edit /host/etc/passwd, or possibly even to chroot(8) back into the host system and effectively escape the container. But at this point you're not really getting much benefit from Docker, and it'll be much easier to run the task outside a container.
I dit it via SSH & host.docker.internal:
Dockerfile:
RUN apt-get update && apt-get upgrade -y && apt-get install -y ssh
# ...
COPY ./.ssh /root/.ssh
RUN chmod 700 /root/.ssh && chmod 644 /root/.ssh/* && chmod 600 /root/.ssh/id_rsa
docker-compose.yml:
extra_hosts:
- 'host.docker.internal:host-gateway'
I am trying to run multiple js files in a bash script like this. This doesn't work. The container comes up but doesn't run the script. However when I ssh to the container and run this script, the script runs fine and the node service comes up. Can anyone tell me what am I doing wrong?
Dockerfile
FROM node:8.16
MAINTAINER Vivek
WORKDIR /a
ADD . /a
RUN cd /a && npm install
CMD ["./node.sh"]
Script is as below
node.sh
#!/bin/bash
set -e
node /a/b/c/d.js &
node /a/b/c/e.js &
As #hmm mentions your script might be run, but your container is not waiting for your two sub-processes to finish.
You could change your node.sh to:
#!/bin/bash
set -e
node /a/b/c/d.js &
pid1=$!
node /a/b/c/e.js &
pid2=$!
wait pid1
wait pid2
Checkout https://stackoverflow.com/a/356154/1086545 for a more general solution of waiting for sub-processes to finish.
As #DavidMaze is mentioning, a container should generally run one "service". It is of course up to you to decide what constitutes a service in your system. As described officially by docker:
It is generally recommended that you separate areas of concern by using one service per container. That service may fork into multiple processes (for example, Apache web server starts multiple worker processes). It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application.
See https://docs.docker.com/config/containers/multi-service_container/ for more details.
Typically you should run only a single process in a container. However, you can run any number of containers from a single image, and it's easy to set the command a container will run when you start it up.
Set the image's CMD to whatever you think the most common path will be:
CMD ["node", "b/c/d.js"]
If you're using Docker Compose for this, you can specify build: . for both containers, but in the second container, specify an alternate command:.
version: '3'
services:
node-d:
build: .
node-e:
build: .
command: node b/c/e.js
Using bare docker run you can specify an alternate command after the image name
docker build -t me/node-app .
docker run -d --name node-d me/node-app
docker run -d --name node-e me/node-app \
node b/c/e.js
This lets you do things like independently set restart policies for each container; if you run this in a clustered environment like Docker Swarm or Kubernetes, you can independently scale the two containers/pods/processes as well.
I have set up AWS Glue to output Spark event logs so that they can be imported into Spark History Server. AWS provide a CloudFormation stack for this, I just want to run the history server locally and import the event logs. I want to use Docker for this so colleagues can easily run the same thing.
I'm running into problems because the history server is a daemon process, so the container starts and immediately shuts down.
How can I keep the Docker image alive?
My Dockerfile is as follows
ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v2.4.4
FROM ${SPARK_IMAGE}
RUN apk --update add coreutils
RUN mkdir /tmp/spark-events
ENTRYPOINT ["/opt/spark/sbin/start-history-server.sh"]
I start it using:
docker run -v ${PWD}/events:/tmp/spark-events -p 18080:18080 sparkhistoryserver
You need the SPARK_NO_DAEMONIZE environment variable, see here. This will keep the container alive.
Just modify your Dockerfile as follows:
ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v2.4.4
FROM ${SPARK_IMAGE}
RUN apk --update add coreutils
RUN mkdir /tmp/spark-events
ENV SPARK_NO_DAEMONIZE TRUE
ENTRYPOINT ["/opt/spark/sbin/start-history-server.sh"]
See here for a repo with more detailed readme.
I have a simple shell script which is generating delay, jitter, packet loss using the tc-netem package. You can check the script here. My idea is that I want to put it into a container and to execute it on my host using an alias. Something like: echo "alias netsim="sudo docker run -v=pwd:/home/netsim/ docker-image"" >> ~/.bashrc and then when I run netsim on my host to ask me for all the user input. Also the container should control the host network interfaces and I am not even sure if this is possible. My Dockerfile so far is the following:
FROM alpine:3.7
WORKDIR /home/netsim
RUN apk update && apk upgrade
COPY netsim.sh random_netsim.sh /home/netsim/
CMD chmod +x /home/netsim/netsim.sh && chmod +x /home/netsim/random_netsim.sh
The reason why I want to do it is because my host is running Ubuntu 18.04 and in the tc-netem version there is a bug with the jitter generation. I was thinking that it would be pretty cool if I can use containers to do the same, without re-compiling/downgrading the tc-netem version.
I'm pulling my hair out for a week but I am close to giving up. Please share your wisdom.
This is my Docker file:
FROM node
RUN apt-get update
RUN mkdir -p /var/www/stationconnect
RUN mkdir -p /var/log/node
WORKDIR /var/www/stationconnect
COPY stationconnect /var/www/stationconnect
RUN chown node:node /var/log/node
COPY ./stationconnect_fromstage/api/config /var/www/stationconnect/api/config
COPY ./etc/stationconnect /etc/stationconnect
WORKDIR /var/www/stationconnect/api
RUN cd /var/www/stationconnect/api
RUN npm install
RUN apt-get install -y vim nano
RUN npm install supervisor forever -g
EXPOSE 8888
USER node
WORKDIR /var/www/stationconnect/api
CMD ["bash"]
It works fine in docker alone running e.g.
docker run -it 6bcee4528c7c
Any advice?
When create a container, you should have a foreground process to keep the container alive.
What i’ve done is add a shell script line
while true; do sleep 1000; done at the end of my docker-entrypoint.sh, and refer to it in ENTRYPOINT [/docker-entrypoint.sh]
Take a look at this issue to find out more.
There’s an example how to make a Nodejs dockerfile, be sure to check it out.
this is kind of obvious. You are running it with interactive terminal bash session with docker run -it <container>. When you run a container in kube (or in docker without -it) bash will exit immediately, so this is what it is doing in kube deployment. Not crashing per say, just terminating as expected.
Change your command to some long lasting process. Even sleep 1d will do - it will die no longer. Nor will your node app work though... for that you need your magic command to launch your app in foreground.
You could add an ENTRYPOINT command to your Dockerfile that executes something that is run in the background indefinitely, say, for example, you run a script my_service.sh. This, in turn, could start a webserver like nginx as a service or simply do a tail -f /dev/null. This will keep your pod running in kubernetes as the main task of this container is not done yet. In your Dockerfile above, bash is executed, but once it runs it finishes and the container completes. Therefore, when you try to do kubectl run NAME --image=YOUR_IMAGE it fails to connect because k8s is terminating the pod that runs your container almost immediately after the new pod is started. This process will continue like this infinitely.
Please see this answer here for a in-line command that can help you run your image as is for debugging purposes...