Simple Node app and mongo containers created using docker-compose below... What am I missing?
mongodb://user:password#mongo:27017/
version: '3.8'
services:
mongo:
image: mongo
restart: always
environment:
- MONGO_INITDB_ROOT_USERNAME=user
- MONGO_INITDB_ROOT_PASSWORD=password
app:
image: app
build:
context: ./app
dockerfile: Dockerfile
ports:
- "3000:3000"
depends_on:
- mongo
I've read several posts on the same issue and the official mongo docker page and seem to be doing everything correct. Keep getting the following msg.
app_1 | mongodb://user:password#mongo:27017/ {
app_1 | autoIndex: false,
app_1 | poolSize: 10,
app_1 | bufferMaxEntries: 0,
app_1 | useNewUrlParser: true,
app_1 | useUnifiedTopology: true
app_1 | }
app_1 | MongoDB connection with retry
app_1 | MongoDB connection unsuccessful, retry after 5 seconds. 2
I prepared interesting test for you
version: '3.8'
services:
mongo:
image: mongo
environment:
- MONGO_INITDB_ROOT_USERNAME=user
- MONGO_INITDB_ROOT_PASSWORD=password
healthcheck:
test: "echo 'db.runCommand(\"ping\").ok'"
interval: 5s
timeout: 5s
retries: 3
app:
image: mongo
command: "mongosh mongodb://user:password#mongo:27017/admin --eval \"printjson(db.test.insertOne({'a': 1}))\""
ports:
- "3000:3000"
depends_on:
- mongo
It will print
app_1 | {
app_1 | acknowledged: true,
app_1 | insertedId: ObjectId("63696c4e99703eb4ab9fba62")
app_1 | }
but if you will change
command: "mongosh mongodb://user:password#mongo:27017/admin --eval \"printjson(db.test.insertOne({'a': 1}))\""
to
command: "mongosh mongodb://user:password#mongo:27017/local --eval \"printjson(db.test.insertOne({'a': 1}))\""
you will see error. Even if you will add MONGO_INITDB_DATABASE to mongo with value local.
You can test it running in second console command:
docker-compose run app bash
and then try
mongosh mongodb://user:password#mongo:27017/admin --eval "printjson(db.test.insertOne({'a': 1}))"
with success
and
mongosh mongodb://user:password#mongo:27017/local --eval "printjson(db.test.insertOne({'a': 1}))"
that failing. You can see logs from mongo like
mongo_1 | {"t":{"$date":"2022-11-07T20:40:35.686+00:00"},"s":"I", "c":"ACCESS", "id":20249, "ctx":"conn6","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-256","speculative":true,"principalName":"user","authenticationDatabase":"local","remote":"172.20.0.3:50382","extraInfo":{},"error":"UserNotFound: Could not find user \"user\" for db \"local\""}}
mongo_1 | {"t":{"$date":"2022-11-07T20:40:35.687+00:00"},"s":"I", "c":"ACCESS", "id":20249, "ctx":"conn6","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"user","authenticationDatabase":"local","remote":"172.20.0.3:50382","extraInfo":{},"error":"UserNotFound: Could not find user \"user\" for db \"local\""}}
mongo_1 | {"t":{"$date":"2022-11-07T20:40:35.688+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn5","msg":"Connection ended","attr":{"remote":"172.20.0.3:50374","uuid":"b77fff1f-b832-4900-9c2d-1e7fd1e79424","connectionId":5,"connectionCount":1}}
mongo_1 | {"t":{"$date":"2022-11-07T20:40:35.699+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn6","msg":"Connection ended","attr":{"remote":"172.20.0.3:50382","uuid":"3995bcbf-706d-4bed-92a2-04736305b7c2","connectionId":6,"connectionCount":0}}
this problem is described in topic User not found on MongoDB Docker image with authentication
You can authenticate with user,password against admin db, not mydb
You can read more about creating database with user and password here:
How to create a DB for MongoDB container on start up?
This issue has to do with the fact that the file exists on the backend container but not the postgres container. How could I transfer the file between containers automatically?
I am currently trying to execute the following script:
COPY climates(
station_id,
date,
element,
data_value,
m_flag,
q_flag,
s_flag,
obs_time
)
FROM '/usr/api/2017.csv`
DELIMITER ','
CSV HEADER;
within a docker container running a sequelize backend connecting to a postgres:14.1-alpine container.
The following error is returned:
db_1 | 2022-08-30 04:23:58.358 UTC [29] ERROR: could not open file "/usr/api/2017.csv" for reading: No such file or directory
db_1 | 2022-08-30 04:23:58.358 UTC [29] HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
db_1 | 2022-08-30 04:23:58.358 UTC [29] STATEMENT: COPY climates(
db_1 | station_id,
db_1 | date,
db_1 | element,
db_1 | data_value,
db_1 | m_flag,
db_1 | q_flag,
db_1 | s_flag,
db_1 | obs_time
db_1 | )
db_1 | FROM '/usr/api/2017.csv'
db_1 | DELIMITER ','
db_1 | CSV HEADER;
ebapi | Unable to connect to the database: MigrationError: Migration 20220829_02_populate_table.js (up) failed: Original error: could not open file "/usr/api/2017.csv" for reading: No such file or directory
ebapi | at /usr/api/node_modules/umzug/lib/umzug.js:151:27
ebapi | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
ebapi | at async Umzug.runCommand (/usr/api/node_modules/umzug/lib/umzug.js:107:20)
ebapi | ... 2 lines matching cause stack trace ...
ebapi | at async start (/usr/api/index.js:14:3) {
ebapi | cause: Error
ebapi | at Query.run (/usr/api/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)
ebapi | at /usr/api/node_modules/sequelize/lib/sequelize.js:311:28
ebapi | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
ebapi | at async Object.up (/usr/api/migrations/20220829_02_populate_table.js:10:5)
ebapi | at async /usr/api/node_modules/umzug/lib/umzug.js:148:21
ebapi | at async Umzug.runCommand (/usr/api/node_modules/umzug/lib/umzug.js:107:20)
ebapi | at async runMigrations (/usr/api/util/db.js:52:22)
ebapi | at async connectToDatabase (/usr/api/util/db.js:32:5)
ebapi | at async start (/usr/api/index.js:14:3) {
ebapi | name: 'SequelizeDatabaseError',
...
Here is my docker-compose.yml
# set up a postgres database version: "3.8" services: db:
image: postgres:14.1-alpine
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
ports:
- "5432:5432"
volumes:
- db:/var/lib/postgresql/data
- ./db/init.sql:/docker-entrypoint-initdb.d/create_tables.sql api:
container_name: ebapi
build:
context: ./energybot
depends_on:
- db
ports:
- 3001:3001
environment:
DATABASE_URL: postgres://postgres:postgres#db:5432/postgres
DB_HOST: db
DB_PORT: 5432
DB_USER: postgres
DB_PASSWORD: postgres
DB_NAME: postgres
links:
- db
volumes:
- "./energybot:/usr/api"
volumes: db:
driver: local
I have 2 dockers composes. 1 implements an infrastructure with a reverse proxy, databases, database admins, etc.
version: '3.7'
volumes:
mysql80_volume:
name: MySql80-Volume
services:
reverse-proxy:
image: traefik:v2.2
container_name: "reverse-proxy"
restart: always
network_mode: bridge
ports:
- "80:80"
- "443:443"
# etc...
mysql80:
container_name: mysql80
image: mysql:8.0
restart: always
labels:
- traefik.enable=false
network_mode: bridge
ports:
- 3380:3306
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
volumes:
- mysql80_volume:/var/lib/mysql
- ./my.cnf:/etc/my.cnf
And there is a docker-compose service, which describes the configs for the service.
version: '3.7'
services:
server:
container_name: project-api.com
build:
context: .
dockerfile: docker/Dockerfile
sysctls:
- net.ipv4.ip_unprivileged_port_start=0
restart: always
labels:
- traefik.enable=true
#labels for traefik
extra_hosts:
- "project-api.com www.project-api.com:127.0.0.1"
external_links:
- mysql80:mysql # <---- My link
network_mode: bridge
volumes:
- .:/app
I want to connect the database from docker-compose 1 to the docker-compose 2 file and pass it to the connection in the Typeform config.
Config for typeorm
type: "mysql",
host: process.env.TYPEORM_ENDPOINT,
database: process.env.TYPEORM_DB_NAME,
port: parseInt(process.env.TYPEORM_PORT) || 3306,
username: process.env.TYPEORM_USERNAME,
password: process.env.TYPEORM_PASSWORD,
.env
TYPEORM_ENDPOINT=mysql
TYPEORM_DB_NAME=red_driver
TYPEORM_USERNAME=root
TYPEORM_PASSWORD=root
TYPEORM_PORT=3306
But I am getting an error.
project-api.com | [nodemon] 2.0.7
project-api.com | [nodemon] to restart at any time, enter `rs`
project-api.com | [nodemon] watching path(s): *.*
project-api.com | [nodemon] watching extensions: ts,graphql
project-api.com | [nodemon] starting `ts-node index.ts`
project-api.com | Error: getaddrinfo ENOTFOUND mysql
project-api.com | at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:71:26)
project-api.com | --------------------
project-api.com | at Protocol._enqueue (/home/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
project-api.com | at Protocol.handshake (/home/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
project-api.com | at PoolConnection.connect (/home/app/node_modules/mysql/lib/Connection.js:116:18)
project-api.com | at Pool.getConnection (/home/app/node_modules/mysql/lib/Pool.js:48:16)
project-api.com | at /home/app/src/driver/mysql/MysqlDriver.ts:894:18
project-api.com | at new Promise (<anonymous>)
project-api.com | at MysqlDriver.createPool (/home/app/src/driver/mysql/MysqlDriver.ts:891:16)
project-api.com | at MysqlDriver.<anonymous> (/home/app/src/driver/mysql/MysqlDriver.ts:344:36)
project-api.com | at step (/home/app/node_modules/typeorm/node_modules/tslib/tslib.js:141:27)
project-api.com | at Object.next (/home/app/node_modules/typeorm/node_modules/tslib/tslib.js:122:57) {
project-api.com | errno: -3008,
project-api.com | code: 'ENOTFOUND',
project-api.com | syscall: 'getaddrinfo',
project-api.com | hostname: 'mysql',
project-api.com | fatal: true
project-api.com | }
The config property of typeorm must be the container name of MySQL. (host: 'mysql80')
What I'm trying to do
I'm trying to set up a docker-compose definition, where I have a mongoDB container, and a nodeJS container that connects to it.
version: "3.9"
services:
events-db:
image: mongo
volumes:
- db-volume:/data/db
environment:
MONGO_INITDB_ROOT_USERNAME: $SANDBOX_DB_USER
MONGO_INITDB_ROOT_PASSWORD: $SANDBOX_DB_PASS
MONGO_INITDB_DATABASE: sandboxdb
app:
image: node:15.12.0
user: node
working_dir: /home/node/app
volumes:
- ./:/home/node/app:ro
environment:
MDB_CONNECTION: mongodb://$SANDBOX_DB_USER:$SANDBOX_DB_PASS#events-db:27017/sandboxdb
command: node myapp
depends_on:
- events-db
volumes:
db-volume:
Along with a .env file that declares the credentials (planning to use proper env variables when I deploy this to a production environment):
SANDBOX_DB_USER=myuser
SANDBOX_DB_PASS=myp4ss
Finally, my nodejs script, myapp.js is simply trying to connect, grab a reference to a collection, and insert a document:
require('dotenv').config()
const { MongoClient } = require('mongodb')
async function main () {
console.log('Connecting')
const client = new MongoClient(process.env.MDB_CONNECTION, {
connectTimeoutMS: 10000,
useUnifiedTopology: true,
})
await client.connect()
const db = client.db()
const events = db.collection('events')
console.log('Inserting an event')
await events.insertOne({
type: 'foo',
timestamp: new Date(),
})
console.log('Done.')
process.exit(0)
}
if (require.main === module) {
main()
}
Result
When I run docker-compose config I see the following output, so I would expect it to work:
$ docker-compose config
services:
app:
command: node myapp
depends_on:
events-db:
condition: service_started
environment:
MDB_CONNECTION: mongodb://myuser:myp4ss#events-db:27017/sandboxdb
image: node:15.12.0
user: node
volumes:
- C:\workspace\dcsandbox:/home/node/app:ro
working_dir: /home/node/app
events-db:
environment:
MONGO_INITDB_DATABASE: sandboxdb
MONGO_INITDB_ROOT_PASSWORD: myp4ss
MONGO_INITDB_ROOT_USERNAME: myuser
image: mongo
volumes:
- db-volume:/data/db:rw
version: '3.9'
volumes:
db-volume: {}
However, when I run docker-compose up I see that my node container is unable to connect to the mongoDB to insert an event:
events-db_1 | {"t":{"$date":"2021-04-07T13:57:36.793+00:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
app_1 | Connecting
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.811+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.27.0.3:34164","connectionId":1,"connectionCount":1}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.816+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.27.0.3:34164","client":"conn1","doc":{"driver":{"name":"nodejs","version":"3.6.6"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"4.19.128-microsoft-standard"},"platform":"'Node.js v15.12.0, LE (unified)"}}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.820+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.27.0.3:34166","connectionId":2,"connectionCount":2}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.822+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.27.0.3:34166","client":"conn2","doc":{"driver":{"name":"nodejs","version":"3.6.6"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"4.19.128-microsoft-standard"},"platform":"'Node.js v15.12.0, LE (unified)"}}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.822+00:00"},"s":"I", "c":"ACCESS", "id":20251, "ctx":"conn2","msg":"Supported SASL mechanisms requested for unknown user","attr":{"user":"myuser#sandboxdb"}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.823+00:00"},"s":"I", "c":"ACCESS", "id":20249, "ctx":"conn2","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-256","principalName":"myuser","authenticationDatabase":"sandboxdb","client":"172.27.0.3:34166","result":"UserNotFound: Could not find user \"myuser\" for db \"sandboxdb\""}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.824+00:00"},"s":"I", "c":"ACCESS", "id":20249, "ctx":"conn2","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","principalName":"myuser","authenticationDatabase":"sandboxdb","client":"172.27.0.3:34166","result":"UserNotFound: Could not find user \"myuser\" for db \"sandboxdb\""}}
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.826+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn1","msg":"Connection ended","attr":{"remote":"172.27.0.3:34164","connectionId":1,"connectionCount":1}}
app_1 | /home/node/app/node_modules/mongodb/lib/cmap/connection.js:268
app_1 | callback(new MongoError(document));
app_1 | ^
app_1 |
app_1 | MongoError: Authentication failed.
app_1 | at MessageStream.messageHandler (/home/node/app/node_modules/mongodb/lib/cmap/connection.js:268:20)
app_1 | at MessageStream.emit (node:events:369:20)
app_1 | at processIncomingData (/home/node/app/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
app_1 | at MessageStream._write (/home/node/app/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
app_1 | at writeOrBuffer (node:internal/streams/writable:395:12)
app_1 | at MessageStream.Writable.write (node:internal/streams/writable:340:10)
app_1 | at Socket.ondata (node:internal/streams/readable:750:22)
app_1 | at Socket.emit (node:events:369:20)
app_1 | at addChunk (node:internal/streams/readable:313:12)
app_1 | at readableAddChunk (node:internal/streams/readable:288:9) {
app_1 | ok: 0,
app_1 | code: 18,
app_1 | codeName: 'AuthenticationFailed'
app_1 | }
events-db_1 | {"t":{"$date":"2021-04-07T13:57:38.832+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2","msg":"Connection ended","attr":{"remote":"172.27.0.3:34166","connectionId":2,"connectionCount":0}}
dcsandbox_app_1 exited with code 1
I've put the full output at https://pastebin.com/uNyJ6tiy
and the example code at this repo: https://github.com/akatechis/example-docker-compose-mongo-node-auth
After some more digging, I managed to figure it out. The issue is that the MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD variables simply set the root user's credentials, and the MONGO_INITDB_DATABASE simply sets the initial database for scripts in /docker-entrypoint-initdb.d.
By default, the root user is added to the admin database, so by removing the /sandboxdb part of the connection string, I was able to have my node app authenticate against the admin DB as the root user.
While this doesn't quite accomplish what I wanted initially (to create a separate, non-root user for my database, and use that to authenticate), I think this puts me on the right path to using an init script to set up the user accounts I want to have.
I faced an error while running MongoDB in docker. When I was running my docker-composer file. The node-app-setup container gave the following error and my HTTP server was crashed. But, my mongo-express container working fine. On the other hand. When I MongoDB atlas the error was gone. And my app(node-app-setup) was running well.
How can I solve this problem?
Error message from Docker log:
> nodemon ./bin/www
[nodemon] 2.0.6
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node ./bin/www`
mongodb://db:27017/mydb?authSource=admin
Master 35 is running
test-app is listening on http://localhost:8089
connected mongoose
events.js:292
throw er; // Unhandled 'error' event
^
MongoError: The $changeStream stage is only supported on replica sets
at MessageStream.messageHandler (/usr/src/app/node_modules/mongodb/lib/cmap/connection.js:268:20)
at MessageStream.emit (events.js:315:20)
at processIncomingData (/usr/src/app/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
at MessageStream._write (/usr/src/app/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
at writeOrBuffer (_stream_writable.js:352:12)
at MessageStream.Writable.write (_stream_writable.js:303:10)
at Socket.ondata (_stream_readable.js:719:22)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:284:9)
at Socket.Readable.push (_stream_readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Emitted 'error' event on ChangeStream instance at:
at ChangeStream.<anonymous> (/usr/src/app/node_modules/mongoose/lib/cursor/ChangeStream.js:41:51)
at ChangeStream.emit (events.js:315:20)
at processError (/usr/src/app/node_modules/mongodb/lib/change_stream.js:571:38)
at ChangeStreamCursor.<anonymous> (/usr/src/app/node_modules/mongodb/lib/change_stream.js:435:5)
at ChangeStreamCursor.emit (events.js:315:20)
at /usr/src/app/node_modules/mongodb/lib/core/cursor.js:343:16
at /usr/src/app/node_modules/mongodb/lib/core/cursor.js:736:9
at /usr/src/app/node_modules/mongodb/lib/change_stream.js:329:9
at done (/usr/src/app/node_modules/mongodb/lib/core/cursor.js:458:7)
at /usr/src/app/node_modules/mongodb/lib/core/cursor.js:533:11
at executeCallback (/usr/src/app/node_modules/mongodb/lib/operations/execute_operation.js:70:5)
at callbackWithRetry (/usr/src/app/node_modules/mongodb/lib/operations/execute_operation.js:122:14)
at /usr/src/app/node_modules/mongodb/lib/operations/command_v2.js:88:9
at /usr/src/app/node_modules/mongodb/lib/cmap/connection_pool.js:348:13
at handleOperationResult (/usr/src/app/node_modules/mongodb/lib/core/sdam/server.js:558:5)
at MessageStream.messageHandler (/usr/src/app/node_modules/mongodb/lib/cmap/connection.js:268:11) {
ok: 0,
code: 40573,
codeName: 'Location40573'
}
[nodemon] app crashed - waiting for file changes before starting...
My Docker Compose code:
version: '3.8'
networks:
my-network-nodeApp:
services:
db:
container_name: mongo
image: mongo:4.2.0
restart: always
ports:
- 27017:27017
# environment:
# - MONGO_INITDB_ROOT_USERNAME=root
# - MONGO_INITDB_ROOT_PASSWORD=example
networks:
- my-network-nodeApp
db-mangement:
container_name: mongo-express
image: mongo-express
restart: always
depends_on:
- db
ports:
- 8081:8081
environment:
# ME_CONFIG_MONGODB_ADMINUSERNAME: root
# ME_CONFIG_MONGODB_ADMINPASSWORD: example
ME_CONFIG_MONGODB_SERVER: db
networks:
- my-network-nodeApp
app:
build:
context: .
dockerfile: ./docker/node/Dockerfile.Dev
image: "node-app"
depends_on:
- db
- db-mangement
container_name: "node-app-setup"
restart: always
# command: npm start
ports:
- "8089:8089"
volumes:
- ./:/usr/src/app
# - /app/code/node_modules
networks:
- my-network-nodeApp
My .env variable:
MONGODB_HOST=mongodb://db:27017/mydb?authSource=admin
MY app.js DB CONNECTION CODE
const mongoose = require("mongoose");
console.log(process.env.MONGODB_HOST);
mongoose.connect(process.env.MONGODB_HOST, {
useNewUrlParser: true,
useFindAndModify: false,
useUnifiedTopology: true,
});
const db = mongoose.connection;
db.on("error", console.error.bind(console, "connection error:"));
db.once("open", function () {
console.log("connected mongoose");
});