Daemonized buildbot start - linux

I'm trying to compose the simplest possible docker buildbot master image that runs buildbot start in ENTRYPOINT/CMD Dockerfile instructions.
I've tried to use a lot of combinations of dumb-init, gosu and exec, but with no success.
The situation is as follows:
When I try to run deamonized buildroot with the command docker run -d -v $local/vol/bldbot/master:/var/lib/buildbot buildbot-master-test, the container starts successfully, but it is terminated abruptly. The log looks as follows:
[timestamp] [-] Log opened.
[timestamp] [-] twistd 16.0.0 (/usr/bin/python 2.7.12) starting up.
[timestamp] [-] reactor class: twisted.internet.epollreactor.EPollReactor.
[timestamp] [-] Starting BuildMaster -- buildbot.version: 0.9.2
[timestamp] [-] Loading configuration from '/var/lib/buildbot/master.cfg'
[timestamp] [-] Setting up database with URL 'sqlite:/state.sqlite'
[timestamp] [-] setting database journal mode to 'wal'
[timestamp] [-] doing housekeeping for master 1 c8aa8b0d5ca3:/var/lib/buildbot
[timestamp] [-] adding 1 new changesources, removing 0
[timestamp] [-] adding 1 new builders, removing 0
[timestamp] [-] adding 2 new schedulers, removing 0
[timestamp] [-] No web server configured on this master
[timestamp] [-] adding 1 new workers, removing 0
[timestamp] [-] PBServerFactory starting on 9989
[timestamp] [-] Starting factory
[timestamp] [-] BuildMaster is running
When I run the container in an interactive mode with the command docker run --rm -it -v $local/vol/bldbot/master:/var/lib/buildbot buildbot-master-test /bin/sh and next I run the command buildbot start all works like charm.
I've already studied the content of official buildbot master docker image, i.e. buildbot/buildbot-master. I see that authors decided to use the command exec twistd -ny $B/buildbot.tac in start_buildbot.sh, not their own buildbot start.
So the question is, how to compose the ENTRYPOINT/CMD instructions in the Dockerfile that runs simply buildbot start.
ADDENDUM 1
Dockerfile content
FROM alpine:3.4
ENV BASE_DIR=/var/lib/buildbot SRC_DIR=/usr/src/buildbot
COPY start $SRC_DIR/
RUN \
echo #testing http://nl.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories && \
echo #community http://nl.alpinelinux.org/alpine/edge/community >> /etc/apk/repositories && \
apk add --no-cache \
python \
py-pip \
py-twisted \
py-cffi \
py-cryptography#community \
py-service_identity#community \
py-sqlalchemy#community \
gosu#testing \
dumb-init#community \
py-jinja2 \
tar \
curl && \
# install pip dependencies
pip install --upgrade pip setuptools && \
pip install "buildbot" && \
rm -r /root/.cache
WORKDIR $BASE_DIR
RUN \
adduser -D -s /bin/sh bldbotmaster && \
chown bldbotmaster:bldbotmaster .
VOLUME $BASE_DIR
CMD ["dumb-init", "/usr/src/buildbot/start","buildbot","master"]
ADDENDUM 2
start script content
#!/bin/sh
set -e
BASE_DIR=/var/lib/buildbot
if [[ "$1" = 'buildbot' && "$2" = 'master' ]]; then
if [ -z "$(ls -A "$BASE_DIR/master.cfg" 2> /dev/null)" ]; then
gosu bldbotmaster buildbot create-master -r $BASE_DIR
gosu bldbotmaster cp $BASE_DIR/master.cfg.sample $BASE_DIR/master.cfg
fi
exec gosu bldbotmaster buildbot start $BASE_DIR
fi
exec "$#"

Buildbot bootstrap is based on Twisted's ".tac" files, which are expected to be started using twistd -y buildbot.tac.
The buildbot start script is actually just a convenience wrapper around twistd. It actually just run twistd, and then watches for the logs to confirm buildbot successfully started. There is no value added beyond this log watching, so it is not strictly mandatory to start buildbot with buildbot start.
You can just start it with twistd -y buildbot.tac.
As you pointed up the official docker image is starting buildbot with twistd -ny buildbot.tac
If you look at the help of twistd, -y means the Twisted daemon will run a .tac file, and the -n means it won't daemonize.
This is because docker is doing process watching by itself, and do not want its entrypoint to daemonize.
The buildbot start command also has a --nodaemon option, which really only is 'exec'ing to twistd -ny.
So for your dockerfile, you can as well us twistd -ny or buildbot start --nodaemon, this will work the same.
Another Docker specific is that the buildbot.tac is different. It configured the twistd logs to output to stdout instead of outputing to twisted.log.
This is because docker design expects logs to be in stdout so that you can configure any fancy cloud log forwarder independently from the application's tech.

I've studied the docker reference and buildbot manual again and have found one hints.
There is a remark with an ngnix example
Do not pass a service x start command to a detached container. For example, this command attempts to start the nginx service.
$ docker run -d -p 80:80 my_image service nginx start
This succeeds in starting the nginx service inside the container. However, it fails the detached container paradigm in that, the root process (service nginx start) returns and the detached container stops as designed. As a result, the nginx service is started but could not be used. Instead, to start a process such as the nginx web server do the following:
$ docker run -d -p 80:80 my_image nginx -g 'daemon off;'
On the other hand there is an option
The --nodaemon option instructs Buildbot to skip daemonizing. The process will start in the foreground. It will only return to the command-line when it is stopped.
Both of the above trails yield
exec gosu bldbotmaster buildbot start --nodaemon $BASE_DIR
line in the start script's line that solves at least abrupt termination phenomenon.

Related

how to run feedconsumers and consumers multiple for kafka in docker?

So I have this docker file and i want to run feed-consumers and consumers multiple times and i tried to do so. We have a node.js application for feed-consumers and consumer and pass user_levels to it.
I just want to ask is this the right approach?
FROM ubuntu:18.04
# Set Apt to noninteractive mode
ENV DEBIAN_FRONTEND noninteractive
# Install Helper Commands
ADD scripts/bin/* /usr/local/bin/
RUN chmod +x /usr/local/bin/*
RUN apt-install-and-clean curl \
build-essential \
git >> /dev/null 2>&1
RUN install-node-12.16.1
RUN mkdir -p /usr/src/app
COPY . /usr/src/app
WORKDIR /usr/src/app
#RUN yarn init-cache
#RUN yarn init-temp
#RUN yarn init-user
RUN yarn install
RUN yarn build
RUN node ./feedsconsumer/consumer.js user_level=0
RUN for i in {1..10}; do node ./feedsconsumer/consumer.js user_level=1; done
RUN for i in {1..20}; do node ./feedsconsumer/consumer.js user_level=2; done
RUN for i in {1..20}; do node ./feedsconsumer/consumer.js user_level=3; done
RUN for i in {1..30}; do node ./feedsconsumer/consumer.js user_level=4; done
RUN for i in {1..40}; do node ./feedsconsumer/consumer.js user_level=5; done
RUN for i in {1..10}; do node ./consumer/consumer.js; done
ENTRYPOINT ["tail", "-f", "/dev/null"]
Or is there any other way around?
Thanks
A container runs exactly one process. Your container's is
ENTRYPOINT ["tail", "-f", "/dev/null"]
This translates to "do absolutely nothing, in a way that's hard to override". I typically recommend using CMD over ENTRYPOINT, and the main container command shouldn't ever be an artificial "do nothing but keep the container running" command.
Before that, you're trying to RUN the process(es) that are the main container process. The RUN only happens during the image build phase, the running process(es) aren't persisted in the image, the build will block until these processes complete, and they can't connect to other containers or data stores. These are the lines you want to be the CMD.
A container only runs one processes, but you can run multiple containers off the same image. It's somewhat easier to add parameters by setting environment variables than by adjusting the command line (you have to replace the whole thing), so in your code look for process.env.USER_LEVEL. Also make sure the process stays as a foreground process and doesn't use a package to daemonize itself.
Then the final part of the Dockerfile just needs to set a default CMD that launches one copy of your application:
...
COPY package.json yarn.lock .
RUN yarn install
COPY . .
RUN yarn build
CMD node ./feedsconsumer/consumer.js
Now you can start a single container running this process
docker build -t my/consumer .
docker run -d --name consumer my/consumer
And you can start multiple containers to run the whole set of them
for user_level in `seq 5`; do
for i in `seq 10`; do
docker run -d \
--name "feed-consumer-$user_level-$i" \
-e "USER_LEVEL=$user_level" \
my/consumer
done
done
for i in `seq 10`; do
docker run -d --name "consumer-$i" \
my/consumer \
node ./consumer/consumer.js
done
Notice this last invocation overrides the CMD to run the alternate script; this becomes a more contorted invocation if it needs to override ENTRYPOINT instead. (docker run --entrypoint node my/consumer ./consumer/consumer.js)
If you're looking forward to cluster environments like Kubernetes, it's often straightforward to run multiple identical copies of a container, which is what you're trying to do here. A Kubernetes Deployment object has a replicas: count, and you can kubectl scale deployment feed-consumer-5 --replicas=40 to change what's in the question, or potentially configure a HorizontalPodAutoscaler to set it dynamically based on the topic length (this last is involved, but possible and rewarding).

azure self hosted agent linux do not run with "--once" parameter

i like to run the self-hosted Linux container only once per pipeline
that means when the pipeline is done i like the container to stop
i saw that there is a parameter called "--once"
please this link in the bottom :
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops
but when i start the docker like this with the once after the run :
docker run --once --rm -it -e AZP_WORK=/home/working_dir -v /home/working_dir:/azp -e AZP_URL=https://dev.azure.com/xxxx -e AZP_TOKEN=nhxxxxxu76mlua -e AZP_AGENT_NAME=ios_dockeragent xxx.xxx.com:2000/azure_self_hosted_agent/agent:latest
I'm getting :
unknown flag: --once
See 'docker run --help'.
also if i put it in the docker file
as
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh --once"]
Im getting error when trying to run the docker :
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"./start.sh --once\": stat ./start.sh --once: no such file or directory": unknown
where do i need to set this "--once" command in dockerized agent?
Is for the agent's run, not the docker run. from the docs:
For agents configured to run interactively, you can choose to have the
agent accept only one job. To run in this configuration:
./run.sh --once
Agents in this mode will accept only one job and then spin down
gracefully (useful for running in Docker on a service like Azure
Container Instances).
So, you need to add it in the bash script you configure the docker image:
FROM ubuntu:18.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl4 \
libicu60 \
libunwind8 \
netcat
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh --once
As far as I know, there's no way to pass it in from the outside; you have to go into the container and edit the start.sh file to add the --once argument to the appropriate line.
exec ./externals/node/bin/node ./bin/AgentService.js interactive --once & wait $!
cleanup
Side note: depending on your requirements, you might also take the opportunity to remove the undocumented web-server from start.sh.

docker run not working: no container created despite making container image

Summary
docker run doesn't seem to build a container (but it also doesn't throw an error) despite docker build successfully building the container image.
Input and Output
1. Successful docker image creation..
$ docker build -t minitwitter:latest .
...
Successfully built da191988e0db
Successfully tagged minitwitter:latest
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
minitwitter latest da191988e0db 6 seconds ago 173MB
python 3.7-alpine b11d2a09763f 9 days ago 98.8MB
2. ..and docker run completes without error..
$ docker run --name minitwitter -d -p 8000:5000 --rm minitwitter:latest
e8835f1b4c72c8e1a8736589c74d56ee2d12ec7bcfb4695531759fb1c2cf0e48
3. ..but docker container doesn't seem to exist.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
And navigating to the port where the app should be exposed, http://localhost:8000, returns the connection error ERR_CONNECTION_REFUSED.
Docker file, boot.sh
The Dockerfile and boot.sh files are pretty simple I think:
Dockerfile
FROM python:3.7-alpine
RUN adduser -D minitwitter
WORKDIR /home/minitwitter
COPY requirements.txt requirements.txt
RUN python -m venv env
RUN env/bin/pip install -r requirements.txt
RUN env/bin/pip install gunicorn
COPY app app
COPY migrations migrations
COPY minitwitter.py config.py boot.sh ./
RUN chmod a+x boot.sh
ENV FLASK_APP minitwitter.py
RUN chown -R minitwitter:minitwitter ./
USER minitwitter
EXPOSE 5000
ENTRYPOINT ["./boot.sh"]
boot.sh
# BOOTS A DOCKER CONTAINER
#!/bin/sh
source env/bin/activate
flask db upgrade
exec gunicorn -b :5000 --access-logfile - --error-logfile - minitwitter:app
Place the 'shebang' -- #!/bin/sh -- on the first line of the boot.sh shell script.
How I found this answer: This blog post which refers to this Stackoverflow post.
The problem: the original script has a comment on the first line and the shebang on the second line.
Note: The title of the 'Question' is misleading: a docker container was built. The container, however, was short-lived and given I used the -rm option in the docker run command, the container was deleted after it terminated within 2 seconds; this is why it didn't appear in the docker images -a command.

Azure webapp running "docker run" twice and fails to deploy

I am trying to deploy my web app on azure using docker. On my local machine it works fine, but when I deploy it in azure I can see that it is running docker run twice (why twice?)
2019-10-02 11:15:26.257 INFO - Status: Image is up to date for *******.azurecr.io/****_****:v2.11
2019-10-02 11:15:26.266 INFO - Pull Image successful, Time taken: 0 Minutes and 1 Seconds
2019-10-02 11:15:26.297 INFO - Starting container for site
2019-10-02 11:15:26.298 INFO - docker run -d -p 27757:8000 --name **********-dv_0_a70e438e -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=8000 -e WEBSITE_SITE_NAME=********-dv -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=********-dv.azurewebsites.net -e WEBSITE_INSTANCE_ID=************************* -e HTTP_LOGGING_ENABLED=1 ********.azurecr.io/*****_*****:v2.11 init.sh
2019-10-02 11:15:28.069 INFO - Starting container for site
2019-10-02 11:15:28.070 INFO - docker run -d -p 6078:8081 --name **********_middleware -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=8000 -e WEBSITE_SITE_NAME=******-dv -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=********** -e WEBSITE_INSTANCE_ID=******73***** -e HTTP_LOGGING_ENABLED=1 appsvc/middleware:1907112318 /Host.ListenUrl=http://0.0.0.0:8081 /Host.DestinationHostUrl=http://172.16.1.3:8000 /Host.UseFileLogging=true
This leads to an error later :
2019-10-02 11:15:30.410 INFO - Initiating warmup request to container drillx-stuckpipe-dv_0_a70e438e for site *********-dv
2019-10-02 11:19:38.791 ERROR - Container *******-dv_0_a70e438e for site ********-dv did not start within expected time limit. Elapsed time = 248.3813377 sec
In the logs stream of the app I can see that the web app has started but du to the fact that the port 8000 is not accessible I get this error :
2019-10-02 11:43:55.827 INFO - Container ********-dv_0_33e8d6cc_middleware for site ********-dv initialized successfully and is ready to serve requests.
2019-10-02 11:43:55.881 ERROR - Container ********-dv_0_33e8d6cc didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.
In my Dockerfile I do have at end EXPOSE 8000. I do not know what I am missing here...
FROM python:3.6
# ssh
ENV SSH_PASSWD "root:PWD!"
RUN apt-get update \
&& apt-get -y install \
apt-transport-https \
apt-utils \
curl \
openssh-server \
&& apt-get clean \
&& echo "$SSH_PASSWD" | chpasswd
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get -y install \
msodbcsql17 \
unixodbc-dev \
libxmlsec1-dev \
&& apt-get clean
RUN mkdir /code
WORKDIR code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
WORKDIR /code/
RUN ls -ltr
COPY sshd_config /etc/ssh/
COPY init.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/init.sh
EXPOSE 8000
ENTRYPOINT ["init.sh"]
Init.sh :
#!/bin/bash
set -e
echo "Starting SSH ..."
service ssh start
gunicorn --bind 0.0.0.0:8000 wsgi
If we correctly look at the logs, its actually not running twice. The docker run logs are for different images.
Your application image
Middleware - "appsvc/middleware" is the image used to handle Easy Auth/MSI/CORS on Web App Linux.
https://hajekj.net/2019/01/21/exploring-app-service-authentication-on-linux/
Now coming to the actual issue, if we take a look at the second set of logs. It states that your application failed to start in expected time limit.
This is by default 230 seconds on Web App Linux and can be increased using WEBSITES_CONTAINER_START_TIME_LIMIT application setting. Maximum value can be upto 1800 seconds.
How does Azure verify that application has started or not? : Azure will ping to a PORT and will wait for a HTTP response. If it receives one, then container will be started otherwise docker run will be executed again and process continues.
Which PORT: https://blogs.msdn.microsoft.com/waws/2017/09/08/things-you-should-know-web-apps-and-linux/#NoPing

Docker cannot run on build when running container with a different user

I don't know the specifics why the node application does not run. Basically I added a dockerfile in a nodejs app, and here is my Dockerfile
FROM node:0.10-onbuild
RUN mv /usr/src/app /ghost && useradd ghost --home /ghost && \
cd /ghost
ENV NODE_ENV production
VOLUME ["/ghost/content"]
WORKDIR /ghost
EXPOSE 2368
CMD ["bash", "start.bash"]
Where start.bash looks like this:
#!/bin/bash
GHOST="/ghost"
chown -R ghost:ghost /ghost
su ghost << EOF
cd "$GHOST"
NODE_ENV={$NODE_ENV:-production} npm start
EOF
I usually run docker like so:
docker run --name ghost -d -p 80:2368 user/ghost
With that I cannot see what is going on, and I decided to run it like this:
docker run --name ghost -it -p 80:2368 user/ghost
And I got this output:
> ghost#0.5.2 start /ghost
> node index
Seems, like starting, but as I check the status of the container docker ps -a , it is stopped.
Here is the repo for that but, the start.bash and dockerfile is different, because I haven't committed the latest, since both are not working:
JoeyHipolito/Ghost
I manage to make it work, there is no error in the start bash file nor in the Dockerfile, it's just that I failed to build the image again.
With that said, you can checkout the final Dockerfile and start.bash file in my repository:
Ghost-blog__Docker (https://github.com/joeyhipolito/ghost)
At the time I write this answer, you can see it in the feature-branch, feature/dockerize.

Resources