This question already has answers here:
Docker Compose wait for container X before starting Y
(20 answers)
Closed 2 years ago.
I have a microservice application that uses RabbitMQ. How can I run the RabbitMQ consumer from the application backend container only after the RabbitMQ is up and running. My compose file is as below.
certichain_backend:
depends_on:
- rabbitmq
working_dir: /app/backend/src
command: sh sleep 20 & nohup node /app/backend/src/services/amqp_consumer.js && npm run start;
rabbitmq:
image: "rabbitmq:3-management"
hostname: "rabbitmq"
restart: always
expose:
- 15672
- 5672
labels:
NAME: "rabbitmq"
volumes:
- ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
I have given the backend the 'depends_on' to rabbitmq. But what I have observed is, the rabbitmq container starting process is initiated. But the backend container is not waiting for the rabbitmq container to up completely. My consumer is running along with the backend. So the consumer cannot connect to amqp server since it is not running at that moment. Thus I added a sleep parameter. So that it gets some time while rabbitmq to bringing up.
This method is very inconsistent. I'm sure this is not the right way to achieve this.
In your nodejs code you can add feature to terminate process with exit 1, if rabbitmq container is unreachable.
rabbitmq.connect('rabbitmq://guest:guest#rabbitmq',{})
.then(function(connection){
console.log('Rabbitmq connection established');
// other code here
})
.catch(function(errror) {
console.error('%s while dialing rabbitmq', error.message);
process.exit(1);
});
and in docker-compose file you can add restart: on-failure, so, if rabbitmq container have not started, nodejs application fails to start and is restarted until rabbitmq container is ready.
It can be worth to make rabbitmq connection establishment one of first actions nodejs application does - so, if there is no rabbitmq, nothing starts.
Related
Good morning guys.
I'm having a problem connecting a nodejs application, in a container, to another container that contains a redis server. On my local machine I can connect the application to this redis container without any problem. However, when trying to upload this application in a container, a timeout error is returned.
I'm new to docker and I don't understand why I can connect to this docker container in the application running locally on my machine but that same connection doesn't work when I upload the application in a container.
I tried using docker-compose, but from what I understand it will upload in another container to the redis server, instead of using the redis container that is already in docker.
To connect to redis I'm using the following code:
createClient({
socket: {
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT)
}
});
Where REDIS_HOST is the address of my container running on the server and REDIS_PORT is the port where this container is running on my server.
To run redis on docker I used the following guide: https://redis.io/docs/stack/get-started/install/docker/
I apologize if my problem was not very clear, I'm still studying docker.
You mentioned you are using Docker Compose. Here's an example showing how to start Redis in a container, and make your Node application wait for that container then use an environment variable in your Node application to specify the name of the host to connect to Redis on. In this example it connects to the container running Redis that I've called "redis":
version: "3.9"
services:
redis:
container_name: redis_kaboom
image: "redislabs/redismod"
ports:
- 6379:6379
volumes:
- ./redisdata:/data
entrypoint:
redis-server
--loadmodule /usr/lib/redis/modules/rejson.so
--appendonly yes
deploy:
replicas: 1
restart_policy:
condition: on-failure
node:
container_name: node_kaboom
build: .
volumes:
- .:/app
- /app/node_modules
command: sh -c "npm run load && npm run dev"
depends_on:
- redis
ports:
- 8080:8080
environment:
- REDIS_HOST=redis
So in your Node code you'd then use the value of process.env.REDIS_HOST to connect to the right Redis host. Here, I'm not using a password or a non-standard port, you could also supply those as environment variables that match the configuration of the Redis container in Docker Compose too if you needed to.
Disclosure: I work for Redis.
What I have:
I run celery with rabbitMQ as a broker and redis as the result backend. I have an app that sends tasks and workers that process the tasks.
I deployed this as follow:
The app, redis, rabbitMQ and a worker (let's call him 'local_worker') are running on an azure VM using a docker-compose so I use the docker version of rabbitMQ and redis (6.2.5). rabbitMQ and redis ports are open on the VM and those containers are configured with username and password.
I add workers using azure container instances that connects to the redis and rabbitMQ running on the VM.
First if you have recommendation on this architecture I would be glad to get advice.
The problem:
Everything works well, the tasks are dispatched on the different workers which send back the results etc etc...
When a task is sent after 30 minutes with no task running I observe redis latency of 2 minutes when the task is not sent to the 'local_worker'.
I know this must come from redis because I can see the logs of the task in the worker container instance immediately after sending the task.
I monitor this architecture with flower and graphana with celery prometheus exporter so I can monitor the latency of the tasks. On flower the latent task stays with the 'processing' status.
There is exactly 120 seconds more on a task which is the first one after a no task interval and which is not processed by the 'local_worker'.
This does not happen when the task is processed by the 'local_worker' which runs on the same VM as redis.
It is like redis or the VM was sleeping for 2 minutes before sending back the result. As it is exactly 120 seconds (2 minutes) I expect that it is something wanted by redis, celery or azure (something deterministic)
I don't use a redis conf file, only default settings (except for the password) to run the redis server.
Thanks for your help and feedback on my architecture and problems.
Here is a screenshot of what I see in flower. The three tasks are the same (removing a directory).
The first and the third tasks have been processed by the local worker. The second one has been processed by an external worker. On the logs of the external worker I put a print line just befor returning the results and this line has been printed at 14:14:23. So there has been 120 seconds between this print and the official end of the task.
EDIT:
I found that the default value for redis_socket_timeout was 120 seconds.
I removed the line redis_retry_on_timeout = True and added the line redis_socket_keepalive = True in my celery config file. Now the error I get is that the task failed with redis.exceptions.TimeoutError: Timeout reading from socket.
I don't know why the socket times out whereas the result is ready. Is it a problem with the network of my container instance?
Here is my docker-compose:
version: "3.5"
services:
rabbitmq:
image: rabbitmq:3.8-management
restart: always
ports:
- 5672:5672
labels:
- traefik.enable=true
- traefik.http.services.rabbitmq-ui.loadbalancer.server.port=15672
- traefik.http.routers.rabbitmq-ui-http.entrypoints=http
- traefik.http.routers.rabbitmq-ui-http.rule=(Host(`rabbitmq.${HOSTNAME?Variable not set}.example.app`))
- traefik.docker.network=traefik-public
- traefik.http.routers.rabbitmq-ui-https.entrypoints=https
- traefik.http.routers.rabbitmq-ui-https.rule=Host(`rabbitmq.${HOSTNAME?Variable not set}.example.app`)
- traefik.http.routers.rabbitmq-ui-https.tls=true
- traefik.http.routers.rabbitmq-ui-https.tls.certresolver=le
- traefik.http.routers.rabbitmq-ui-http.middlewares=https-redirect
env_file:
- .env
environment:
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
networks:
- traefik-public
redis:
image: redis:6.2.5
restart: always
command: ["redis-server", "--requirepass", "${RABBITMQ_DEFAULT_PASS:-password}"]
ports:
- 6379:6379
networks:
- traefik-public
flower:
image: mher/flower:0.9.5
restart: always
labels:
- traefik.enable=true
- traefik.http.services.flower-ui.loadbalancer.server.port=5555
- traefik.http.routers.flower-ui-http.entrypoints=http
- traefik.http.routers.flower-ui-http.rule=Host(`flower.${HOSTNAME?Variable not set}.example.app`)
- traefik.docker.network=traefik-public
- traefik.http.routers.flower-ui-https.entrypoints=https
- traefik.http.routers.flower-ui-https.rule=Host(`flower.${HOSTNAME?Variable not set}.example.app`)
- traefik.http.routers.flower-ui-https.tls=true
- traefik.http.routers.flower-ui-https.tls.certresolver=le
- traefik.http.routers.flower-ui-http.middlewares=https-redirect
- traefik.http.routers.flower-ui-https.middlewares=traefik-admin-auth
env_file:
- .env
command:
- "--broker=amqp://${RABBITMQ_DEFAULT_USER:-guest}:${RABBITMQ_DEFAULT_PASS:-guest}#rabbitmq:5672//"
depends_on:
- rabbitmq
- redis
networks:
- traefik-public
local_worker:
build:
context: ..
dockerfile: ./setup/devops/docker/app.dockerfile
image: swtools:app
restart: always
volumes:
- ${SWTOOLSWORKINGDIR:-/tmp}:${SWTOOLSWORKINGDIR:-/tmp}
command: ["celery", "--app=app.worker.celery_app:celery_app", "worker", "-n", "local_worker#%h"]
env_file:
- .env
environment:
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
- RABBITMQ_HOST=rabbitmq
- REDIS_HOST=${HOSTNAME?Variable not set}
depends_on:
- rabbitmq
- redis
networks:
- traefik-public
dashboard_app:
image: swtools:app
restart: always
labels:
- traefik.enable=true
- traefik.http.services.dash-app.loadbalancer.server.port=${DASH_PORT-8080}
- traefik.http.routers.dash-app-http.entrypoints=http
- traefik.http.routers.dash-app-http.rule=Host(`dashboard.${HOSTNAME?Variable not set}.example.app`)
- traefik.docker.network=traefik-public
- traefik.http.routers.dash-app-https.entrypoints=https
- traefik.http.routers.dash-app-https.rule=Host(`dashboard.${HOSTNAME?Variable not set}.example.app`)
- traefik.http.routers.dash-app-https.tls=true
- traefik.http.routers.dash-app-https.tls.certresolver=le
- traefik.http.routers.dash-app-http.middlewares=https-redirect
- traefik.http.middlewares.operator-auth.basicauth.users=${OPERATOR_USERNAME?Variable not set}:${HASHED_OPERATOR_PASSWORD?Variable not set}
- traefik.http.routers.dash-app-https.middlewares=operator-auth
volumes:
- ${SWTOOLSWORKINGDIR:-/tmp}:${SWTOOLSWORKINGDIR:-/tmp}
command: ['waitress-serve', '--port=${DASH_PORT:-8080}', 'app.order_dashboard:app.server']
env_file:
- .env
environment:
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
- RABBITMQ_HOST=rabbitmq
- REDIS_HOST=${HOSTNAME?Variable not set}
networks:
- traefik-public
depends_on:
- rabbitmq
- redis
networks:
traefik-public:
external: true
and my celery config file:
import os
import warnings
from pathlib import Path
# result backend use redis
result_backend_host = os.getenv('REDIS_HOST', 'localhost')
result_backend_pass = os.getenv('REDIS_PASS', 'password')
result_backend = 'redis://:{password}#{host}:6379/0'.format(password=result_backend_pass, host=result_backend_host)
# redis_retry_on_timeout = True
redis_socket_keepalive = True
# broker use rabbitmq
rabbitmq_user = os.getenv('RABBITMQ_DEFAULT_USER', 'guest')
rabbitmq_pass = os.getenv('RABBITMQ_DEFAULT_PASS', 'guest')
rabbitmq_host = os.getenv('RABBITMQ_HOST', 'localhost')
broker_url = 'amqp://{user}:{password}#{host}:5672//'.format(user=rabbitmq_user, password=rabbitmq_pass, host=rabbitmq_host)
include = ['app.worker.tasks', 'app.dashboard.example1', 'app.dashboard.example2']
#task events
worker_send_task_events = True
task_send_sent_event = True
All the env variables are defined and it works well except my socket timeout problem! When I deploy a new worker on a container instance, I set the env variables so it connects to the rabbitmq and redis running on the docker-compose.
Here is my celery file that defines the celery app:
from celery import Celery
from app.worker import celery_config
celery_app = Celery()
celery_app.config_from_object(celery_config)
I guess that you have some firewall between your Redis instance and your worker.
Can you login to that SandboxHost... and ensure that you can connect your redis?
You can do that with telnet, for example:
telnet <your_redis_hostname> <your_redis_port>
or with redis-cli:
redis-cli -h <your_redis_hostname> -p <your_redis_port>
EDIT:
Seems like you're missing result_backend:
result_backend = f"redis://username:{result_backend_pass}#{result_backend_host}:6379/0"
and make sure that your REDIS_HOST=${HOSTNAME?Variable not set} is valid...
EDIT2:
Can you add the bind to your Redis command:
["redis-server", "--bind", "0.0.0.0", "--requirepass", "${RABBITMQ_DEFAULT_PASS:-password}"]
Please be aware of its security implications!
Finally changing the backend to rpc resolved the problem. I tried different things with redis that didn't work. A way to dig would be to inspect the sockets with tcp-dump to see where it blocks but I didn't try that I my problem was solved with the rpc backend.
i want launch three containers for my web application.
The containers are: frontend, backend and mongo database.
To do this i write the following docker-compose.yml
version: '3.7'
services:
web:
image: node
container_name: web
ports:
- "3000:3000"
working_dir: /node/client
volumes:
- ./client:/node/client
links:
- api
depends_on:
- api
command: npm start
api:
image: node
container_name: api
ports:
- "3001:3001"
working_dir: /node/api
volumes:
- ./server:/node/api
links:
- mongodb
depends_on:
- mongodb
command: npm start
mongodb:
restart: always
image: mongo
container_name: mongodb
ports:
- "27017:27017"
volumes:
- ./database/data:/data/db
- ./database/config:/data/configdb
and update connection string on my .env file
MONGO_URI = 'mongodb://mongodb:27017/test'
I run it with docker-compose up -d and all go on.
The problem is when i run docker logs api -f for monitoring the backend status: i have MongoNetworkError: failed to connect to server [mongodb:27017] on first connect error, because my mongodb container is up but not in waiting connections (he goes up after backend try to connect).
How can i check if mongodb is in waiting connections status before run api container?
Thanks in advance
Several possible solutions in order of preference:
Configure your application to retry after a short delay and eventually timeout after too many connection failures. This is an ideal solution for portability and can also be used to handle the database restarting after your application is already running and connected.
Use an entrypoint that waits for mongo to become available. You can attempt a full mongo client connect + login, or a simple tcp port check with a script like wait-for-it. Once that check finishes (or times out and fails) you can continue the entrypoint to launching your application.
Configure docker to retry starting your application with a restart policy, or deploy it with orchestration that automatically recovers when the application crashes. This is a less than ideal solution, but extremely easy to implement.
Here's an example of option 3:
api:
image: node
deploy:
restart_policy:
condition: unless-stopped
Note, looking at your compose file, you have a mix of v2 and v3 syntax in your compose file, and many options like depends_on, links, and container_name, are not valid with swarm mode. You are also defining settings like working_dir, which should really be done in your Dockerfile instead.
I am under a weird dilemma. I have created a node application and this application needs to connect to MongoDB (through docker container) I created a docker-compose file as follows:
version: "3"
services:
mongo:
image: mongo
expose:
- 27017
volumes:
- ./data/db:/data/db
my-node:
image: "<MY_IMAGE_PATH>:latest"
deploy:
replicas: 1
restart_policy:
condition: on-failure
working_dir: /opt/app
ports:
- "2000:2000"
volumes:
- ./mod:/usr/app
networks:
- webnet
command: "node app.js"
networks:
webnet:
I am using mongo official image. I have ommited my docker image from the above configuration .I have set up with many configuration but i am unable to connect to mongoDB (yes i have changed the MongoDB uri inside the node.js application too). but whenever i am deploying my docker-compose my application on start up gives me always MongoNetworkError of TransientTransactionError. I am unable to understand where is the problem since many hours.
One more weird thing is when i running my docker-compose file i receive following logs:
Creating network server_default
Creating network server_webnet
Creating service server_mongo
Creating service server_feed-grabber
Could it be that both services are in a different network? If yes then how to fix that?
Other Info:
In node.js application MongoDB uri that i tried is
mongodb://mongo:27017/MyDB
I am running my docker-compose with the command: docker stack deploy -c docker-compose.yml server
My node.js image is Ubuntu 18
Anyone can help me with this?
Ok So i have tried few things and i figured out at last after spending many many hours. There were two things i was doing wrong and they were hitting me on last point:
First thing logging of the startup of docker showed me that it was creating two networks server_default and server_webnet this is the first mistake. Both containers should be in the same network while working.
The second thing I needed to run the Mongo container first as my Node.js application depend_on the Mongo container to be run first. This is exactly what I did in my docker-compose configuration by introducing depend_on property.
For me it was:
1- get your ip by running command
docker-machine ip
2- don't go to localhost/port, go to your ip/port, example : http://192.168.99.100:8080
I am using Docker to create multiple containers, one of which contains a RabbitMQ instance and another contains the node.js action that should respond to queue activity. Traversing the docker-compose logs, I see a lot of ECONNREFUSED errors, before I see where the line begins indicating that RabbitMQ has started in its container. This seems to indicate that RabbitMQ seems to be starting after the service that needs it.
As a sidebar, just to eliminate any other possible causes here is the connection string for node.js to connect to RabbitMQ:
amqp://rabbitmq:5672
and here is the entry for RabbitMQ in the docker-compose.yaml file:
rabbitmq:
container_name: "myapp_rabbitmq"
tty: true
image: rabbitmq:management
ports:
- 15672:15672
- 15671:15671
- 5672:5672
volumes:
- /rabbitmq/lib:/var/lib/rabbitmq
- /rabbitmq/log:/var/log/rabbitmq
- /rabbitmq/conf:/etc/rabbitmq/
service1:
container_name: "service1"
build:
context: .
dockerfile: ./service1.dockerfile
links:
- mongo
- rabbitmq
depends_on:
- mongo
- rabbitmq
service2:
container_name: "service2"
build:
context: .
dockerfile: ./service2/dockerfile
links:
- mongo
- rabbitmq
depends_on:
- mongo
- rabbitmq
What is the fix for this timing issue?
How could I get RabbitMQ to start before the consuming container starts?
Might this not be a timing issue, but a configuration issue in the docker-compose.yml entry I have listed?
It doesn't look like you have included a complete docker-compose file. I would expect to also see your node container in the compose. I think the problem is that you need a
depends_on:
- "rabbitmq"
In the node container part of your docker compose
More info on compose dependancies here: https://docs.docker.com/compose/startup-order/
note, as this page suggests you should do this in conjunction with making your app resilient to outages on external services.
You need to control the boot-up process of your dependent containers. Below documents the same
https://docs.docker.com/compose/startup-order/
I usually use wait-for-it.sh file from below project
https://github.com/vishnubob/wait-for-it
So I will have a below command in my service1
wait-for-it.sh rabbitmq:5672 -t 90 -- command with args to launch service1