I'm trying to dockerize Node.js application which connects to MongoDB using mongoose. It succeeds anytime I run node index.js from the shell when the connection URL is: mongodb://localhost:27017/deposit.
If I restart my computer and then try to run the dockerized project (with mongo instead of localhost in url) with the command docker-compose up it fails to connect to MongoDB. But after I try again the same command, then it succeeds.
So my question is why node cannot connect to MongoDB on first try after the computer is restarted?
PS. Docker is running when I'm trying it
connection.js
const mongoose = require('mongoose');
const connection = "mongodb://mongo:27017/deposit";
const connectDb = () => {
mongoose.connect(connection, {useNewUrlParser: true, useUnifiedTopology: true}).then(res => console.log("Connected to DB"))
.catch(err => console.log('>> Failed to connect to MongoDB, retrying...'));
};
module.exports = connectDb;
Dockerfile
FROM node:latest
RUN mkdir -p /app
WORKDIR /app
#/usr/src/app
COPY package.json /app
RUN npm install
COPY . /app
EXPOSE 7500
# ENTRYPOINT ["node"]
CMD ["node", "src/index.js"]
docker-compose.yml
version: "3"
services:
deposit:
container_name: deposit
image: test/deposit
restart: always
build: .
network_mode: host
ports:
- "7500:7500"
depends_on:
- mongo
mongo:
container_name: mongo
image: mongo
volumes:
- /data:/data/db
network_mode: host
ports:
- '27017:27017'
In your case, node application starts before mongo being ready. There are two approaches to tackle this problem: To handle it in your docker-compose or your application.
You can use wait-for-it.sh or write a wrapper script (both described here) to make sure that your node application starts after the db is ready.
But as quoted from docker documentation, it is better to handle this in your application:
To handle this, design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.
The best solution is to perform this check in your application code,
both at startup and whenever a connection is lost for any reason
You can implement mongo retry as below (Described in this answer):
var connectWithRetry = function() {
return mongoose.connect(mongoUrl, function(err) {
if (err) {
console.error('Failed to connect to mongo on startup - retrying in 5 sec', err);
setTimeout(connectWithRetry, 5000);
}
});
};
connectWithRetry();
As mentioned in the comment there might be the possibility the DB is not yet ready to accept the connection, so one way is to add retry logic or the other option is
serverSelectionTimeoutMS -
With useUnifiedTopology, the MongoDB driver will try to find a server
to send any given operation to, and keep retrying for
serverSelectionTimeoutMS milliseconds. If not set, the MongoDB driver
defaults to using 30000 (30 seconds).
So try with below option
const mongoose = require('mongoose');
const uri = 'mongodb://mongo:27017/deposit?retryWrites=true&w=majority';
mongoose.connect(uri, {
useNewUrlParser: true,
useUnifiedTopology: true,
serverSelectionTimeoutMS: 50000
}).catch(err => console.log(err.reason));
But again if you init DB script getting bigger it will take more time, so you go with retry logic if that did not work. in the above script it will wait for 50 seconds.
Related
I'm trying link my node server with my db in Mongo, but doesn't work...
This is my Docker-Compose.yml:
version: '3.9'
services:
human:
build:
context: ./human-app
container_name: human
ports:
- "3001:3001"
mongodb:
image: mongo
container_name: mongodb
env_file:
- ./human-app/srcs/.env
environment:
- MONGO_INITDB_ROOT_USERNAME=Pepe
- MONGO_INITDB_ROOT_PASSWORD=PepePass
- MONGO_INITDB_DATABASE=Pool
ports:
- "27017:27017"
volumes:
- ./volumes_mongo:/data/db
And when i run the command:
docker-compose up -d
Everything work correctly but separate, i check the status with the command "docker ps -a" and the result is this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9335c5cd4940 mongo "docker-entrypoint.s…" 4 seconds ago Up 3 seconds 0.0.0.0:27017->27017/tcp mongodb
2a28635d2caa human-selection-app_human "node server" 4 seconds ago Up 3 seconds 0.0.0.0:3001->3001/tcp human
If i enter in my container of mongo, and follow the next steps, thats happen:
1- docker exec -it mongodb bash
2- mongo
3- show dbs "result no dbs"
So, i can't link nothing with my node server, probably i need create a network for the 2 services? can anyone help me? thanks.
Extra, this is how i connect from my server to MongoDB:
async _connectDB() {
//! We create the connection to the database
const dbUser = process.env.DBUSER;;
const dbPassword = process.env.DBPASSWORD;
const dbName = process.env.DBNAME;
const dbUriLocal = `mongodb://${dbUser}:${dbPassword}#127.0.0.1:27017/${dbName}`;
mongoose.connect(dbUriLocal, { useNewUrlParser: true, useUnifiedTopology: true })
.then(() => console.log(clc.cyan('Debug: Database connected successfully')))
.catch(e => console.log(e));
}
As your application is also running inside of a Docker container being in the same network with the MongoDB Service, it means you should use MongoDB's service name as the connection address instead of the localhost IP address.
Change your dbUriLocal variable to use the MongoDB Service name (in your case, it is "mongodb" named after the service name in docker-compose.yml) as the connection address instead of "127.0.0.1":
const dbUriLocal = `mongodb://${dbUser}:${dbPassword}#mongodb:27017/${dbName}`;
See "Networking" in the Docker Compose File Referrence for more information.
I'm trying to use Docker Compose to connect a Node.js container to a Postgres container. I can run the Node server fine, and can also connect to the Postgres container fine from my local as I've mapped the ports, but I'm unable to get the Node container to connect to the database.
Here's the compose YAML file:
version: "3"
services:
api_dev:
build: ./api
command: sh -c "sleep 5; npm run dev" # I added the sleep in to wait for the DB to load but it doesn't work either way with or without this sleep part
container_name: pc-api-dev
depends_on:
- dba
links:
- dba
ports:
- 8001:3001
volumes:
- ./api:/home/app/api
- /home/app/api/node_modules
working_dir: /home/app/api
restart: on-failure
dba:
container_name: dba
image: postgres
expose:
- 5432
ports:
- '5431:5432'
env_file:
- ./api/db.env
In my Node container, I'm waiting for the Node server to spin up and attempting to connect to the database in the other container like so:
const { Client } = require('pg')
const server = app.listen(app.get('port'), async () => {
console.log('App running...');
const client = new Client({
user: 'db-user',
host: 'dba', // host is set to the service name of the DB in the compose file
database: 'db-name',
password: 'db-pass',
port: 5431,
})
try {
await client.connect()
console.log(client) // x - can't see this
client.query('SELECT NOW()', (err, res) => {
console.log(err, res) // x - can't see this
client.end()
})
console.log('test') // x - can't see this
} catch (e) {
console.log(e) // x - also can't see this
}
});
After reading up on it today in depth, I've seen the DB host in the connection code above can't be localhost as that refers to the container which is currently running, so it must be set to the service name of the container we're connecting to (dba in this case). I've also mapped the ports, and can see the DB is ready accepting connections well before my Node server starts.
However, not only can I not connect to the database from Node, I'm also unable to see any success or error console logs from the try catch. It's as if the connection is not resolving, and doesn't ever time out, but I'm not sure.
I've also seen that the "listen_addresses" needs to be updated so other containers can connect to the Postgres container, but struggling to find out how to do this and test when I can't debug the actual issue due to lack of logs.
Any direction would be appreciated, thanks.
You are setting the container name and can reference that container by it. For example,
db:
container_name: container_db
And host:port
DB_URL: container_db:5432
This is a part of my node app
app.configure('development', function() {
app.set('db-uri', 'mongodb://localhost/nodepad-development');
app.use(express.errorHandler({ dumpExceptions: true }));
app.set('view options', {
pretty: true
});
});
app.configure('test', function() {
app.set('db-uri', 'mongodb://localhost/nodepad-test');
app.set('view options', {
pretty: true
});
});
app.configure('production', function() {
app.set('db-uri', 'mongodb://localhost/nodepad-production');
});
Edited to
app.set('db-uri', 'mongodb://mongoDB:27017/nodepad-development');
Still the same error.
I have already created container for my node app which runs on local host but I am unable to connect it to another mongo container due to which I cannot do POST requests to the app.
This is my docker compose file
version: '3'
services:
apptest:
container_name: apptest
restart: always
image: ekamzf/nodeapp:1.1
ports:
- '8080:8080
depends_on:
- mongoDB
mongoDB:
container_name: mongoDB
image: mongo
volumes:
- ./data:/usr/src/app
ports:
- '27017:27017'
And the error I get when I try to register account details in my node app is
Error: Timeout POST /users
at null._onTimeout (/usr/src/app/node_modules/connect-timeout/index.js:12:22)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
What am I missing?
Basically, how should I connect this type of code with mongodb anyway ?
What's the role of 'db-uri'?
Docker Compose aliases service names to host names.
Because you named your container running the mongo image as mongoDB, mongodb (case insensitive) will be the name of the host and the name by which you should refer to it from your Node.JS container and app.
Replace localhost in the URI with mongodb
The MongoDB database defaults to port 27017. Unless you've changed the port, you should specify this value.
Add the port to URI so that you have mongodb:27017
Optional but good practice, refactor the app to use environment variables rather than hard-coded values.
This has at least 2 benefits:
a. Your code becomes more flexible;
b. Your Compose file, by then specifying these values, will be clearer.
See the DockerHub documentation for the image here
See MongoDB"s documentation on connection strings here
A Google search returns many examples using Compose, MongoDB and Node.JS
Update: repro
I'm confident your issue may be related to the timing of the Compose containers. I believe your app tries to connect to the DB before the DB container is ready. This is a common issue with Compose and is not solved using Compose's depends_on. Instead you must find a MongoDB (or perhaps other database) solution to this.
In my repro, index.js (see below) introduces a false delay before it tries to connect to the database. 5 seconds is sufficient time for the DB container to be ready and thus this works:
docker-compose build --no-cache
docker-compose up
Then:
docker-compose logs app
Attaching to 60359441_app_1
app_1 | URL: mongodb://mongodb:27017/example
app_1 | Connected
yields Connected which is good.
Alternatively, to prove the timing issue, you may run the containers separately (and you could remove the sleep function to ensure the database is ready before the app:
HOST=localhost # No DNS naming
PORT=37017 # An arbitrary port to prove the point
# In one session
docker run \
--interactive --tty --rm \
--publish=${PORT}:27017 \
mongo
# In another session
docker run \
--interactive --tty --rm \
--net=host \
--env=HOST=${HOST} --env=PORT=${PORT} --env=DATA=example --env=WAIT=0 \
app
URL: mongodb://localhost:37017/example
Connected
docker-compose.yaml:
version: "3"
services:
app:
image: app
build:
context: ./app
dockerfile: Dockerfile
environment:
- HOST=mongodb
- PORT=27017
- DATA=example
- WAIT=5000
volumes:
- ${PWD}/app:/app
mongodb:
image: mongo
restart: always
# environment:
# MONGO_INITDB_ROOT_USERNAME: root
# MONGO_INITDB_ROOT_PASSWORD: example
mongo-express:
image: mongo-express
restart: always
ports:
- 8081:8081
environment:
ME_CONFIG_MONGODB_SERVER: mongodb
# ME_CONFIG_MONGODB_ADMINUSERNAME: root
# ME_CONFIG_MONGODB_ADMINPASSWORD: example
NB Because I followed your naming of mongodb, the mongo-express container must be provided this host name through ME_CONFIG_MONGODB_SERVER
NB The other environment variables shown with mongo and mongo-express images are the defaults and thus optional.
Dockerfile:
FROM node:13.8.0-slim
WORKDIR /usr/src/app
COPY package.json .
RUN npm install
COPY . .
ENTRYPOINT ["node","index.js"]
index.js:
// Obtain config from the environment
const HOST = process.env.HOST;
const PORT = process.env.PORT;
const DATA = process.env.DATA;
const WAIT = parseInt(process.env.WAIT, 10);
// Create MongoDB client
var MongoClient = require("mongodb").MongoClient;
let url = `mongodb://${HOST}:${PORT}/${DATA}`;
console.log(`URL: ${url}`);
// Artificially delay the code
setTimeout(function() {
MongoClient.connect(url, function(err, db) {
if(!err) {
console.log("Connected");
}
});
}, WAIT);
NB index.js uses the environment (HOST,PORT,DB) for its config which is good practice.
Including mongo-express provides the ability to browse the mongo server to readily observe what's going on:
I'm trying to connect my app to redis, but i get:
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379
when i do:
docker exec -it ed02b7e19810 ping test_redis_1
i've received all packets.
also the redis container declares:
* Running mode=standalone, port=6379
* Ready to accept connections
( i get the WARNINGS but i don't think its related:
Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128
this is my docker-compose.yaml:
version: '3'
services:
test-service:
build: .
volumes:
- ./:/usr/test-service/
ports:
- 5001:3000
depends_on:
- redis
redis:
image: "redis:alpine"
DockerFile
FROM node:8.11.2-alpine
WORKDIR /usr/test-service/
COPY . /usr/test-service/
RUN yarn install
EXPOSE 3000
CMD ["yarn", "run", "start"]
app.js
const Redis = require('ioredis');
const redis = new Redis();
redis.set('foo', 'bar');
redis.get('foo').then(function (result) {
console.log(result);
});
i've also tried with redis package but still can't connect:
var redis = require("redis"),
client = redis.createClient();
client.on("error", function (err) {
console.log("Error " + err);
});
getting:
Error Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
For that specific docker-compose.yml there is no redis on 127.0.0.1, you should use redis as host, since services on the same Docker network are able to find each other using the service names as DNS.
const Redis = require('ioredis');
const redis = new Redis({ host: 'redis' });
Furthermore, depends_on does not wait for redis container to be ready before starting, it will only launch it first, so it is your job to wait before starting app.js or just handle that inside app.js
io-redis comes with a reconnection strategy, so you may want to try that first.
You can see my answer here regarding that issue:
Wait node.js until Logstash is ready using containers
I have a Node express server consuming a Mongo database.
I'm trying to create a container for each of them using docker-compose.
Here's my docker-compose.yml file:
version: "2"
services:
server:
container_name: server
restart: always
build: .
ports:
- "3000:3000"
depends_on:
- db
db:
container_name: db
image: mongo
volumes:
- /var/lib/mongodb:/data/db
ports:
- "27017:27017"
And my Dockerfile:
FROM node:latest
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app
RUN npm install
COPY . /usr/src/app
RUN npm run build-run
EXPOSE 3000
I saw on many tutorials that, when using Docker to create a Mongo container, the connection string should be updated in mongoose.connect to use Docker containers naming resolution.
So I changed my connection string according to my docker-compose file:
private readonly CONNECTION_STRING: String = 'mongodb://db/search-people-db'
public connect(): void {
mongoose.connect(this.CONNECTION_STRING)
this._db.on('error', (err) => {
console.log(`mongoose server failed to start: ${err}`)
})
this._db.once('open', () => {
console.log(`mongoose server running using ${this.CONNECTION_STRING}`)
})
}'
However, when running sudo docker-compose up, I keep getting the following error:
Mongoose server failed to start: MongoNetworkError: failed to connect to server [db:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND db db:27017]
What am I doing wrong ? Thanks in advance
MongoDB's container boots up but MongoDB itself needs more time start. so your application will not connect to it until it's fully started.
as Docker's documents suggested, you should set a wait time for your application and then run your code.
I suggest to make mongoose try to reconnect if couldn't connect at the first time or let the application crash if it couldn't connect. Docker will run your container again.
Replace depends_on with links in your docker-compose.yml and try to run command again.