Azure webapp running "docker run" twice and fails to deploy - azure

I am trying to deploy my web app on azure using docker. On my local machine it works fine, but when I deploy it in azure I can see that it is running docker run twice (why twice?)
2019-10-02 11:15:26.257 INFO - Status: Image is up to date for *******.azurecr.io/****_****:v2.11
2019-10-02 11:15:26.266 INFO - Pull Image successful, Time taken: 0 Minutes and 1 Seconds
2019-10-02 11:15:26.297 INFO - Starting container for site
2019-10-02 11:15:26.298 INFO - docker run -d -p 27757:8000 --name **********-dv_0_a70e438e -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=8000 -e WEBSITE_SITE_NAME=********-dv -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=********-dv.azurewebsites.net -e WEBSITE_INSTANCE_ID=************************* -e HTTP_LOGGING_ENABLED=1 ********.azurecr.io/*****_*****:v2.11 init.sh
2019-10-02 11:15:28.069 INFO - Starting container for site
2019-10-02 11:15:28.070 INFO - docker run -d -p 6078:8081 --name **********_middleware -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=8000 -e WEBSITE_SITE_NAME=******-dv -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=********** -e WEBSITE_INSTANCE_ID=******73***** -e HTTP_LOGGING_ENABLED=1 appsvc/middleware:1907112318 /Host.ListenUrl=http://0.0.0.0:8081 /Host.DestinationHostUrl=http://172.16.1.3:8000 /Host.UseFileLogging=true
This leads to an error later :
2019-10-02 11:15:30.410 INFO - Initiating warmup request to container drillx-stuckpipe-dv_0_a70e438e for site *********-dv
2019-10-02 11:19:38.791 ERROR - Container *******-dv_0_a70e438e for site ********-dv did not start within expected time limit. Elapsed time = 248.3813377 sec
In the logs stream of the app I can see that the web app has started but du to the fact that the port 8000 is not accessible I get this error :
2019-10-02 11:43:55.827 INFO - Container ********-dv_0_33e8d6cc_middleware for site ********-dv initialized successfully and is ready to serve requests.
2019-10-02 11:43:55.881 ERROR - Container ********-dv_0_33e8d6cc didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.
In my Dockerfile I do have at end EXPOSE 8000. I do not know what I am missing here...
FROM python:3.6
# ssh
ENV SSH_PASSWD "root:PWD!"
RUN apt-get update \
&& apt-get -y install \
apt-transport-https \
apt-utils \
curl \
openssh-server \
&& apt-get clean \
&& echo "$SSH_PASSWD" | chpasswd
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get -y install \
msodbcsql17 \
unixodbc-dev \
libxmlsec1-dev \
&& apt-get clean
RUN mkdir /code
WORKDIR code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
WORKDIR /code/
RUN ls -ltr
COPY sshd_config /etc/ssh/
COPY init.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/init.sh
EXPOSE 8000
ENTRYPOINT ["init.sh"]
Init.sh :
#!/bin/bash
set -e
echo "Starting SSH ..."
service ssh start
gunicorn --bind 0.0.0.0:8000 wsgi

If we correctly look at the logs, its actually not running twice. The docker run logs are for different images.
Your application image
Middleware - "appsvc/middleware" is the image used to handle Easy Auth/MSI/CORS on Web App Linux.
https://hajekj.net/2019/01/21/exploring-app-service-authentication-on-linux/
Now coming to the actual issue, if we take a look at the second set of logs. It states that your application failed to start in expected time limit.
This is by default 230 seconds on Web App Linux and can be increased using WEBSITES_CONTAINER_START_TIME_LIMIT application setting. Maximum value can be upto 1800 seconds.
How does Azure verify that application has started or not? : Azure will ping to a PORT and will wait for a HTTP response. If it receives one, then container will be started otherwise docker run will be executed again and process continues.
Which PORT: https://blogs.msdn.microsoft.com/waws/2017/09/08/things-you-should-know-web-apps-and-linux/#NoPing

Related

azure self hosted agent linux do not run with "--once" parameter

i like to run the self-hosted Linux container only once per pipeline
that means when the pipeline is done i like the container to stop
i saw that there is a parameter called "--once"
please this link in the bottom :
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops
but when i start the docker like this with the once after the run :
docker run --once --rm -it -e AZP_WORK=/home/working_dir -v /home/working_dir:/azp -e AZP_URL=https://dev.azure.com/xxxx -e AZP_TOKEN=nhxxxxxu76mlua -e AZP_AGENT_NAME=ios_dockeragent xxx.xxx.com:2000/azure_self_hosted_agent/agent:latest
I'm getting :
unknown flag: --once
See 'docker run --help'.
also if i put it in the docker file
as
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh --once"]
Im getting error when trying to run the docker :
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"./start.sh --once\": stat ./start.sh --once: no such file or directory": unknown
where do i need to set this "--once" command in dockerized agent?
Is for the agent's run, not the docker run. from the docs:
For agents configured to run interactively, you can choose to have the
agent accept only one job. To run in this configuration:
./run.sh --once
Agents in this mode will accept only one job and then spin down
gracefully (useful for running in Docker on a service like Azure
Container Instances).
So, you need to add it in the bash script you configure the docker image:
FROM ubuntu:18.04
# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl4 \
libicu60 \
libunwind8 \
netcat
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh --once
As far as I know, there's no way to pass it in from the outside; you have to go into the container and edit the start.sh file to add the --once argument to the appropriate line.
exec ./externals/node/bin/node ./bin/AgentService.js interactive --once & wait $!
cleanup
Side note: depending on your requirements, you might also take the opportunity to remove the undocumented web-server from start.sh.

How do you deploy multiple docker containers to gcloud using Travis CI?

I am having trouble accessing my gcloud compute engine via Travis-CI so I can have CI/CD capabilities.
So far using my current code I am able to use my git repository to start up docker containers on Travis CI to see that they work.
I am then able to get them to build, tag, and deploy to the google cloud container registry with no issues.
However, when I get to the step where I want to ssh into my compute instance to pull and run my containers I run into issues.
I have tried using the gcloud compute ssh --command but I run into issues with gcloud not being installed on my instance. Error received:
If I try running a gcloud command it just says gcloud is missing.
bash: gcloud: command not found
The command "gcloud compute ssh --quiet --project charged-formula-262616 --zone us-west1-b instance-1 --command="gcloud auth configure-docker "" failed and exited with 127 during.
I have also tried downloading the gcloud sdk and running the docker config again but I start receiving the Error bellow.
bash
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Using default tag: latest
I am able to ssh into it using putty as another user to pull from the repository with no issues and start the containers and have the gcloud command exists.
The only thing I could think of is the two accounts used for ssh are different but both keys are added to the instance and I don't see where I can control their permissions. I also created a service account for travis ci and granted it all the same permissions as the compute service account and still no dice...
Any help or advice would be much appreciated!
My travis file looks like this
sudo: required
language: generic
services:
- docker
env:
global:
- SHA=$(git rev-parse HEAD)
- CLOUDSDK_CORE_DISABLE_PROMPTS=1
cache:
directories:
- "$HOME/google-cloud-sdk/"
before_install:
- openssl aes-256-cbc -K $encrypted_0c35eebf403c_key -iv $encrypted_0c35eebf403c_iv
-in secrets.tar.enc -out secrets.tar -d
- tar xvf secrets.tar
- if [ ! -d "$HOME/google-cloud-sdk/bin" ]; then rm -rf $HOME/google-cloud-sdk; export
CLOUDSDK_CORE_DISABLE_PROMPTS=1; curl https://sdk.cloud.google.com | bash; fi
- source $HOME/google-cloud-sdk/path.bash.inc
- gcloud auth activate-service-account --key-file service-account.json
- gcloud components update
- gcloud components install docker-credential-gcr
- gcloud version
- eval $(ssh-agent -s)
- chmod 600 deploy_key_open
- echo -e "Host $SERVER_IP_ADDRESS\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config
- ssh-add deploy_key_open
- gcloud auth configure-docker
# - sudo docker pull gcr.io/charged-formula-262616/web-client
# - sudo docker pull gcr.io/charged-formula-262616/web-nginx
deploy:
provider: script
script: bash ./deploy.sh
on:
branch: master
and the bash script is
# docker build -t gcr.io/charged-formula-262616/web-client:latest -t gcr.io/charged-formula-262616/web-client:$SHA -f ./client/Dockerfile ./client
# docker build -t gcr.io/charged-formula-262616/web-nginx:latest -t gcr.io/charged-formula-262616/web-nginx:$SHA -f ./nginx/Dockerfile ./nginx
# docker build -t gcr.io/charged-formula-262616/web-server:latest -t gcr.io/charged-formula-262616/web-server:$SHA -f ./server/Dockerfile ./server
docker push gcr.io/charged-formula-262616/web-client
docker push gcr.io/charged-formula-262616/web-nginx
docker push gcr.io/charged-formula-262616/web-server
# curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-274.0.1-linux-x86_64.tar.gz
# tar zxvf google-cloud-sdk-274.0.1-linux-x86_64.tar.gz google-cloud-sdk
# ./google-cloud-sdk/install.sh
# sudo docker container stop $(docker container ls -aq)
# echo "1 " | gcloud init
ssh -o StrictHostKeyChecking=no -i deploy_key_open travis-ci#104.196.226.118 << EOF
source /home/travis-ci/google-cloud-sdk/path.bash.inc
gcloud auth configure-docker
sudo docker-credential-gcloud list
sudo docker pull gcr.io/charged-formula-262616/web-nginx
sudo docker pull gcr.io/charged-formula-262616/web-client
sudo docker pull gcr.io/charged-formula-262616/web-server
sudo docker run --rm -d -p 3000:3000 gcr.io/charged-formula-262616/web-client
sudo docker run --rm -d -p 80:80 -p 443:443 gcr.io/charged-formula-262616/web-nginx
sudo docker run --rm -d -p 5000:5000 gcr.io/charged-formula-262616/web-server
sudo docker run --rm -d -v /database_data:/var/lib/postgresql/data -e POSTGRES_USER -e POSTGRES_PASSWORD -e POSTGRES_DB postgres
EOF
The error you posted includes a link to Authentication methods where suggests some mechanisms to authenticate docker such as:
gcloud auth configure-docker
And other more advanced authentication methods. I recommend that you check this out as it will guide you to solve your issue.
To install the gcloud command you can follow the guide in Installing Google Cloud SDK. That for Linux are this.

Unable to ssh localhost within a running Docker container

I'm building a Docker image for an application which requires to ssh into localhost (i.e ssh user#localhost)
I'm working on a Ubuntu desktop machine and started with a basic ubuntu:16.04 container.
Following is the content of my Dockerfile:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y \
openjdk-8-jdk \
ssh && \
groupadd -r custom_group && useradd -r -g custom_group -m user1
USER user1
RUN ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N "" && \
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Then I build this container using the command:
docker build -t test-container .
And run it using:
docker run -it test-container
The container opens with the following prompt and the keys are generated correctly to enable ssh into localhost:
user1#0531c0f71e0a:/$
user1#0531c0f71e0a:/$ cd ~/.ssh/
user1#0531c0f71e0a:~/.ssh$ ls
authorized_keys id_rsa id_rsa.pub
Then ssh into localhost and greeted by the error:
user1#0531c0f71e0a:~$ ssh user1#localhost
ssh: connect to host localhost port 22: Cannot assign requested address
Is there anything I'm doing wrong or any additional network settings that needs to be configured? I just want to ssh into localhost within the running container.
First you need to install the ssh server in the image building script:
RUN sudo apt-get install -y openssh-server
Then you need to start the ssh server:
RUN sudo /etc/init.d/ssh start
or probably even in the last lines of the Dockerfile ( you must have one binary instantiated to keep the container running ... )
USER root
CMD [ "sh", "/etc/init.d/ssh", "start"]
on the host than
# init a container from an the image
run -d --name my-ssh-container-name-01 \
-v /opt/local/dir:/opt/container/dir my-image-01
As #user2915097 stated in the OP comments, this was due to the ssh instance in the container was attempting to connect to the host using IPv6.
Forcing connection over IPv4 using -4 solved the issue.
$ docker run -it ubuntu ssh -4 user#hostname
For Docker Compose I was able to add the following to my .yml file:
network_mode: "host"
I believe the equivalent in Docker is:
--net=host
Documentation:
https://docs.docker.com/compose/compose-file/compose-file-v3/#network_mode
https://docs.docker.com/network/#network-drivers
host: For standalone containers, remove network isolation between the
container and the Docker host, and use the host’s networking directly.
See use the host network.
I also faced this error today, here's how to fix it:
If(and only if) you are facing this error inside a running container that isn't in production.
Do this:
docker exec -it -u 0 [your container id here] /bin/bash
then when you entered the container in god mode, run this:
service ssh start
then you can run your ssh based commands.
Of course it is best practice to do it in your Dockerfile before all these, but no need to sweat if you are not done with your image built process just yet.

Daemonized buildbot start

I'm trying to compose the simplest possible docker buildbot master image that runs buildbot start in ENTRYPOINT/CMD Dockerfile instructions.
I've tried to use a lot of combinations of dumb-init, gosu and exec, but with no success.
The situation is as follows:
When I try to run deamonized buildroot with the command docker run -d -v $local/vol/bldbot/master:/var/lib/buildbot buildbot-master-test, the container starts successfully, but it is terminated abruptly. The log looks as follows:
[timestamp] [-] Log opened.
[timestamp] [-] twistd 16.0.0 (/usr/bin/python 2.7.12) starting up.
[timestamp] [-] reactor class: twisted.internet.epollreactor.EPollReactor.
[timestamp] [-] Starting BuildMaster -- buildbot.version: 0.9.2
[timestamp] [-] Loading configuration from '/var/lib/buildbot/master.cfg'
[timestamp] [-] Setting up database with URL 'sqlite:/state.sqlite'
[timestamp] [-] setting database journal mode to 'wal'
[timestamp] [-] doing housekeeping for master 1 c8aa8b0d5ca3:/var/lib/buildbot
[timestamp] [-] adding 1 new changesources, removing 0
[timestamp] [-] adding 1 new builders, removing 0
[timestamp] [-] adding 2 new schedulers, removing 0
[timestamp] [-] No web server configured on this master
[timestamp] [-] adding 1 new workers, removing 0
[timestamp] [-] PBServerFactory starting on 9989
[timestamp] [-] Starting factory
[timestamp] [-] BuildMaster is running
When I run the container in an interactive mode with the command docker run --rm -it -v $local/vol/bldbot/master:/var/lib/buildbot buildbot-master-test /bin/sh and next I run the command buildbot start all works like charm.
I've already studied the content of official buildbot master docker image, i.e. buildbot/buildbot-master. I see that authors decided to use the command exec twistd -ny $B/buildbot.tac in start_buildbot.sh, not their own buildbot start.
So the question is, how to compose the ENTRYPOINT/CMD instructions in the Dockerfile that runs simply buildbot start.
ADDENDUM 1
Dockerfile content
FROM alpine:3.4
ENV BASE_DIR=/var/lib/buildbot SRC_DIR=/usr/src/buildbot
COPY start $SRC_DIR/
RUN \
echo #testing http://nl.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories && \
echo #community http://nl.alpinelinux.org/alpine/edge/community >> /etc/apk/repositories && \
apk add --no-cache \
python \
py-pip \
py-twisted \
py-cffi \
py-cryptography#community \
py-service_identity#community \
py-sqlalchemy#community \
gosu#testing \
dumb-init#community \
py-jinja2 \
tar \
curl && \
# install pip dependencies
pip install --upgrade pip setuptools && \
pip install "buildbot" && \
rm -r /root/.cache
WORKDIR $BASE_DIR
RUN \
adduser -D -s /bin/sh bldbotmaster && \
chown bldbotmaster:bldbotmaster .
VOLUME $BASE_DIR
CMD ["dumb-init", "/usr/src/buildbot/start","buildbot","master"]
ADDENDUM 2
start script content
#!/bin/sh
set -e
BASE_DIR=/var/lib/buildbot
if [[ "$1" = 'buildbot' && "$2" = 'master' ]]; then
if [ -z "$(ls -A "$BASE_DIR/master.cfg" 2> /dev/null)" ]; then
gosu bldbotmaster buildbot create-master -r $BASE_DIR
gosu bldbotmaster cp $BASE_DIR/master.cfg.sample $BASE_DIR/master.cfg
fi
exec gosu bldbotmaster buildbot start $BASE_DIR
fi
exec "$#"
Buildbot bootstrap is based on Twisted's ".tac" files, which are expected to be started using twistd -y buildbot.tac.
The buildbot start script is actually just a convenience wrapper around twistd. It actually just run twistd, and then watches for the logs to confirm buildbot successfully started. There is no value added beyond this log watching, so it is not strictly mandatory to start buildbot with buildbot start.
You can just start it with twistd -y buildbot.tac.
As you pointed up the official docker image is starting buildbot with twistd -ny buildbot.tac
If you look at the help of twistd, -y means the Twisted daemon will run a .tac file, and the -n means it won't daemonize.
This is because docker is doing process watching by itself, and do not want its entrypoint to daemonize.
The buildbot start command also has a --nodaemon option, which really only is 'exec'ing to twistd -ny.
So for your dockerfile, you can as well us twistd -ny or buildbot start --nodaemon, this will work the same.
Another Docker specific is that the buildbot.tac is different. It configured the twistd logs to output to stdout instead of outputing to twisted.log.
This is because docker design expects logs to be in stdout so that you can configure any fancy cloud log forwarder independently from the application's tech.
I've studied the docker reference and buildbot manual again and have found one hints.
There is a remark with an ngnix example
Do not pass a service x start command to a detached container. For example, this command attempts to start the nginx service.
$ docker run -d -p 80:80 my_image service nginx start
This succeeds in starting the nginx service inside the container. However, it fails the detached container paradigm in that, the root process (service nginx start) returns and the detached container stops as designed. As a result, the nginx service is started but could not be used. Instead, to start a process such as the nginx web server do the following:
$ docker run -d -p 80:80 my_image nginx -g 'daemon off;'
On the other hand there is an option
The --nodaemon option instructs Buildbot to skip daemonizing. The process will start in the foreground. It will only return to the command-line when it is stopped.
Both of the above trails yield
exec gosu bldbotmaster buildbot start --nodaemon $BASE_DIR
line in the start script's line that solves at least abrupt termination phenomenon.

Can't curl linked container in Docker even though I can ping it

I have a Docker container called backend that exposes a port, 8200, and runs a django server behind gunicorn inside of it. This is my Dockerfile:
FROM debian:wheezy
RUN rm /bin/sh && \
ln -s /bin/bash /bin/sh && \
apt-get -y update && \
apt-get install -y -q \
curl \
procps \
python=2.7.3-4+deb7u1 \
git \
python-pip=1.1-3 \
python-dev \
libpq-dev && \
rm -rf /var/lib/{apt,dpkg,cache,log}
RUN pip install virtualenv && \
virtualenv mockingbird && \
/bin/bash -c "source mockingbird/bin/activate"
ADD ./requirements.txt /mockingbird/backend/requirements.txt
RUN /mockingbird/bin/pip install -r /mockingbird/backend/requirements.txt
ADD ./src /mockingbird/backend/src
CMD ["/mockingbird/bin/gunicorn", "--workers", "8", "--pythonpath", "/mockingbird/backend/src/", "--bind", "localhost:8200", "backend.wsgi"]
I'm running this container like so:
vagrant#10:~$ sudo docker run --name backend --env-file /mockingbird/apps/backend/env/dev -d --restart always --expose 8200 mockingbird/backend
I know that the django server is up and responding on the correct port by doing the following and getting a response:
vagrant#10:~$ sudo docker exec -it backend /bin/bash
root#b488874c204d:/# curl localhost:8200
I then start a new container linking to backend as follows:
sudo docker run -it --link backend:backend debian:wheezy /bin/bash
But when I try to curl backend, it doesn't work:
root#72946da3dff9:/# apt-get update && apt-get install curl
root#72946da3dff9:/# curl backend:8200
curl: (7) couldn't connect to host
I am, however, able to ping backend:
root#72946da3dff9:/# ping backend
PING backend (172.17.0.41): 48 data bytes
56 bytes from 172.17.0.41: icmp_seq=0 ttl=64 time=0.116 ms
56 bytes from 172.17.0.41: icmp_seq=1 ttl=64 time=0.081 ms
Anyone know anything else I can try to debug why I can't connect to the service running in my linked Docker container? Is there something I am missing here to be able to curl backend:8200 from the linked container?
this might be a problem: "--bind", "localhost:8200" as connections to backend hostname won't be accepted. You might want to change it to "0.0.0.0:8200" or maybe ":8200", depending on the notation supported.

Resources