Sqlalchemy does not see Postgresql db - python-3.x

I develop a web-app using Flask under Python3. I have a problem with connecting to postgresql via sqlalchemy.
Dockerfile:
FROM ubuntu:20.04
ENV LANG en_US.UTF-8
RUN apt update
RUN apt install -y python3-pip python3.8-dev
RUN apt install default-libmysqlclient-dev -y
RUN apt install libpq-dev -y
WORKDIR /app
ADD ./requirements.txt /app/requirements.txt
RUN pip3 install psycopg2
RUN --mount=type=cache,target=/root/.cache/pip pip3 install -r requirements.txt
RUN pip3 install -r requirements.txt
COPY . /app
ENV PYTHONPATH "${PYTHONPATH}:/app/src"
CMD ["gunicorn", "-c", "python:config.gunicorn", "--worker-class", "aiohttp.worker.GunicornWebWorker", "--pythonpath", "/app/src/", "src.main:app"]
docker-compose.yml:
services:
backend:
environment:
TZ: 'UTC'
volumes:
- ./src:/app/src
build:
context: .
restart: always
ports:
- '80:5000'
depends_on:
- postgres
links:
- postgres:postgres
postgres:
restart: always
image: postgres:latest
volumes:
- /var/lib/postgresql
environment:
POSTGRES_PASSWORD: 1234
POSTGRES_USER: postgres
POSTGRES_DB: postgres
POSTGRES_INITDB_ARGS: "-A md5"
ports:
- "5432:5432"
Postgresql logs:
2022-08-03 14:21:46.773 UTC [1] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-08-03 14:21:46.776 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-03 14:21:46.776 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-03 14:21:46.781 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-03 14:21:46.787 UTC [27] LOG: database system was shut down at 2022-08-03 14:20:52 UTC
2022-08-03 14:21:46.794 UTC [1] LOG: database system is ready to accept connections
2022-08-03 14:21:50.342 UTC [1] LOG: received fast shutdown request
2022-08-03 14:21:50.344 UTC [1] LOG: aborting any active transactions
2022-08-03 14:21:50.347 UTC [1] LOG: background worker "logical replication launcher" (PID 33) exited with exit code 1
2022-08-03 14:21:50.347 UTC [28] LOG: shutting down
2022-08-03 14:21:50.365 UTC [1] LOG: database system is shut down
Python app logs:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "0.0.0.0" and accepting
TCP/IP connections on port 5432?
As you can see postgresql is running on 0.0.0.0:5432 and sqlalchemy tries to connect to 0.0.0.0:5432. Therefore, it is not very clear what it does not like.

If I remember correctly, when connecting to another service within a docker compose network, you have to connect using the service name as the host name - so postgres:5432 in your case.

Related

P1001: Can't reach database server at `localhost`:`5432` error [duplicate]

This question already has answers here:
Docker - Can't reach database server at `localhost`:`3306` with Prisma service
(2 answers)
ECONNREFUSED for Postgres on nodeJS with dockers
(7 answers)
Closed 17 days ago.
This post was edited and submitted for review 16 days ago and failed to reopen the post:
Original close reason(s) were not resolved
I am trying to build a docker image from my project and run it in a container,
The project is keystone6 project connecting to a postgres database, everything worked well when I normally run the project and it connects successfully to the database.
Here is my dockerfile:
FROM node:18.13.0-alpine3.16
ENV NODE_VERSION 18.13.0
ENV NODE_ENV=development
LABEL Name="di-wrapp" Version="1"
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . .
RUN npm install
COPY .env .
EXPOSE 9999
CMD ["npm", "run", "dev"]
I am building an image using the command docker build -t di-wrapp:1.0 .
after that I run docker-compose file which contains the following code:
version: "3.8"
services:
postgres:
image: postgres:15-alpine
container_name: localhost
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=di_wrapp
ports:
- "5432:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
dashboard:
image: di-wrapp:1.0
container_name: di-wrapp-container
restart: always
environment:
- DB_CONNECTION=postgres
- DB_PORT=5432
- DB_HOST=localhost
- DB_USER=postgres
- DB_PASSWORD=postgres
- DB_NAME=di_wrapp
tty: true
depends_on:
- postgres
ports:
- 8273:9999
links:
- postgres
command: "npm run dev"
volumes:
- /usr/src/app
volumes:
postgres-data:
And this is the connection URI used to connect my project to postgres:
DATABASE_URL=postgresql://postgres:postgres#localhost:5432/di_wrapp
which I am using to configure my db setting in keystone config file like this:
export const db: DatabaseConfig<BaseKeystoneTypeInfo> = {
provider: "postgresql",
url: String(DATABASE_URL!),
};
when I run the command docker-compose -f docker-compose.yaml up
This is what I receive:
localhost | 2023-02-03 13:43:35.034 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
localhost | 2023-02-03 13:43:35.034 UTC [1] LOG: listening on IPv6 address "::", port 5432
localhost | 2023-02-03 13:43:35.067 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
localhost | 2023-02-03 13:43:35.121 UTC [24] LOG: database system was shut down at 2023-02-03 13:43:08 UTC
localhost | 2023-02-03 13:43:35.155 UTC [1] LOG: database system is ready to accept connections
di-wrapp-container | > keystone-app#1.0.2 dev
di-wrapp-container | > keystone dev
di-wrapp-container |
di-wrapp-container | ✨ Starting Keystone
di-wrapp-container | ⭐️ Server listening on :8273 (http://localhost:8273/)
di-wrapp-container | ⭐️ GraphQL API available at /api/graphql
di-wrapp-container | ✨ Generating GraphQL and Prisma schemas
di-wrapp-container | Error: P1001: Can't reach database server at `localhost`:`5432`
di-wrapp-container |
di-wrapp-container | Please make sure your database server is running at `localhost`:`5432`.
di-wrapp-container | at Object.createDatabase (/usr/src/app/node_modules/#prisma/internals/dist/migrateEngineCommands.js:115:15)
di-wrapp-container | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
di-wrapp-container | at async ensureDatabaseExists (/usr/src/app/node_modules/#keystone-6/core/dist/migrations-e3b5740b.cjs.dev.js:262:19)
di-wrapp-container | at async Object.pushPrismaSchemaToDatabase (/usr/src/app/node_modules/#keystone-6/core/dist/migrations-e3b5740b.cjs.dev.js:68:3)
di-wrapp-container | at async Promise.all (index 1)
di-wrapp-container | at async setupInitialKeystone (/usr/src/app/node_modules/#keystone-6/core/scripts/cli/dist/keystone-6-core-scripts-cli.cjs.dev.js:984:3)
di-wrapp-container | at async initKeystone (/usr/src/app/node_modules/#keystone-6/core/scripts/cli/dist/keystone-6-core-scripts-cli.cjs.dev.js:762:35)di-wrapp-container exited with code 1
even though I receive that the database server is up on port 5432, my app container can't connect to it.
any help is appreciated.

Deploying a multi-container app on Azure App Services

I have been struggling for some days now to deploy an Azure App Service with two minimalist Docker containers.
My first image is a basic PostgreSQL image and my second image is built with the following Dockerfile:
FROM python:3.7
RUN pip install streamlit
COPY app.py /streamlit-docker/
WORKDIR /streamlit-docker/
CMD streamlit run app.py
and where app.py is:
import streamlit as st
st.write('Hello world')
The docker-compose.yml file I use for creating my Azure App service is the following:
version: '3.3'
services:
db:
image: atestcr.azurecr.io/postgres:latest
environment:
POSTGRES_PASSWORD: postgres
app:
image: atestcr.azurecr.io/my_app:latest
ports:
- '8501:8501'
I get an 'Application Error' whenever I try to access the webapp URL. However, when I deploy this app with a single docker container, using this docker-compose.yml:
version: '3.3'
services:
app:
image: atestcr.azurecr.io/my_app:latest
ports:
- '8501:8501'
everything works fine.
Here is a GitHub repo to recreate the bug.
These are the Azure logs:
2021-08-09T17:29:34.382Z INFO - Starting multi-container app..
2021-08-09T17:29:35.089Z INFO - Pulling image: atestcr.azurecr.io/postgres:latest
2021-08-09T17:29:35.313Z INFO - latest Pulling from postgres
2021-08-09T17:29:35.315Z INFO - Digest: sha256:b6df1345afa5990ea32866e5c331eefbf2e30a05f2a715c3a9691a6cb18fa253
2021-08-09T17:29:35.317Z INFO - Status: Image is up to date for atestcr.azurecr.io/postgres:latest
2021-08-09T17:29:35.319Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2021-08-09T17:29:35.335Z INFO - Starting container for site
2021-08-09T17:29:35.335Z INFO - docker run -d -p 8477:5432 --name testapp32_db_0_6662435d -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=testapp32 -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=testapp32.azurewebsites.net -e WEBSITE_INSTANCE_ID=42c09ff46e6c54cb467d28e88f2ab5b1e8971ee4daf2e883f44401bde67fe89f -e HTTP_LOGGING_ENABLED=1 atestcr.azurecr.io/postgres:latest
2021-08-09T17:29:35.781Z INFO - Pulling image: atestcr.azurecr.io/my_app:latest
2021-08-09T17:29:35.984Z INFO - latest Pulling from my_app
2021-08-09T17:29:35.987Z INFO - Digest: sha256:659937a52a6223b938b3d429901ab8648497870bf8068b5dcc05816050db5eaf
2021-08-09T17:29:35.988Z INFO - Status: Image is up to date for atestcr.azurecr.io/my_app:latest
2021-08-09T17:29:35.993Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2021-08-09T17:29:36.014Z INFO - Starting container for site
2021-08-09T17:29:36.014Z INFO - docker run -d -p 0:8501 --name testapp32_app_0_6662435d -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=testapp32 -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=testapp32.azurewebsites.net -e WEBSITE_INSTANCE_ID=42c09ff46e6c54cb467d28e88f2ab5b1e8971ee4daf2e883f44401bde67fe89f -e HTTP_LOGGING_ENABLED=1 atestcr.azurecr.io/my_app:latest
2021-08-09T17:33:26.741Z ERROR - multi-container unit was not started successfully
2021-08-09T17:33:26.846Z INFO - Container logs from testapp32_db_0_6662435d = 2021-08-09T17:29:40.459721366Z The files belonging to this database system will be owned by user "postgres".
2021-08-09T17:29:40.491740899Z This user must also own the server process.
2021-08-09T17:29:40.493019808Z
2021-08-09T17:29:40.497263739Z The database cluster will be initialized with locale "en_US.utf8".
2021-08-09T17:29:40.502456876Z The default database encoding has accordingly been set to "UTF8".
2021-08-09T17:29:40.503203182Z The default text search configuration will be set to "english".
2021-08-09T17:29:40.503218482Z
2021-08-09T17:29:40.503223282Z Data page checksums are disabled.
2021-08-09T17:29:40.506809008Z
2021-08-09T17:29:40.521275113Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2021-08-09T17:29:40.523912632Z creating subdirectories ... ok
2021-08-09T17:29:40.525237242Z selecting dynamic shared memory implementation ... posix
2021-08-09T17:29:40.642905697Z selecting default max_connections ... 100
2021-08-09T17:29:40.733900958Z selecting default shared_buffers ... 128MB
2021-08-09T17:29:41.039050775Z selecting default time zone ... Etc/UTC
2021-08-09T17:29:41.047757638Z creating configuration files ... ok
2021-08-09T17:29:44.416903507Z running bootstrap script ... ok
2021-08-09T17:29:50.023628737Z performing post-bootstrap initialization ... ok
2021-08-09T17:30:04.217544961Z syncing data to disk ... ok
2021-08-09T17:30:04.218321267Z
2021-08-09T17:30:04.219189873Z initdb: warning: enabling "trust" authentication for local connections
2021-08-09T17:30:04.219206473Z You can change this by editing pg_hba.conf or using the option -A, or
2021-08-09T17:30:04.219223973Z --auth-local and --auth-host, the next time you run initdb.
2021-08-09T17:30:04.219491175Z
2021-08-09T17:30:04.219513275Z Success. You can now start the database server using:
2021-08-09T17:30:04.219519575Z
2021-08-09T17:30:04.219523675Z pg_ctl -D /var/lib/postgresql/data -l logfile start
2021-08-09T17:30:04.219527775Z
2021-08-09T17:30:08.340584036Z waiting for server to start.......2021-08-09 17:30:08.340 UTC [44] LOG: starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-09T17:30:08.478247580Z 2021-08-09 17:30:08.410 UTC [44] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-09T17:30:08.680599067Z 2021-08-09 17:30:08.680 UTC [45] LOG: database system was shut down at 2021-08-09 17:29:49 UTC
2021-08-09T17:30:08.753589768Z 2021-08-09 17:30:08.753 UTC [44] LOG: database system is ready to accept connections
2021-08-09T17:30:08.755965684Z done
2021-08-09T17:30:08.760382514Z server started
2021-08-09T17:30:09.790821978Z
2021-08-09T17:30:09.799723839Z /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2021-08-09T17:30:09.802079455Z
2021-08-09T17:30:09.840304817Z 2021-08-09 17:30:09.834 UTC [44] LOG: received fast shutdown request
2021-08-09T17:30:09.862102366Z waiting for server to shut down....2021-08-09 17:30:09.861 UTC [44] LOG: aborting any active transactions
2021-08-09T17:30:09.950488072Z 2021-08-09 17:30:09.950 UTC [44] LOG: background worker "logical replication launcher" (PID 51) exited with exit code 1
2021-08-09T17:30:09.954360498Z 2021-08-09 17:30:09.950 UTC [46] LOG: shutting down
2021-08-09T17:30:13.063848848Z ...2021-08-09 17:30:13.063 UTC [44] LOG: database system is shut down
2021-08-09T17:30:13.138558463Z done
2021-08-09T17:30:13.140082774Z server stopped
2021-08-09T17:30:13.144102302Z
2021-08-09T17:30:13.162430928Z PostgreSQL init process complete; ready for start up.
2021-08-09T17:30:13.165654351Z
2021-08-09T17:30:14.083504086Z 2021-08-09 17:30:13.992 UTC [1] LOG: starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-09T17:30:14.084760295Z 2021-08-09 17:30:14.011 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-09T17:30:14.084774095Z 2021-08-09 17:30:14.015 UTC [1] LOG: listening on IPv6 address "::", port 5432
2021-08-09T17:30:14.084779495Z 2021-08-09 17:30:14.082 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-09T17:30:14.156076187Z 2021-08-09 17:30:14.132 UTC [63] LOG: database system was shut down at 2021-08-09 17:30:12 UTC
2021-08-09T17:30:14.198749982Z 2021-08-09 17:30:14.176 UTC [1] LOG: database system is ready to accept connections
2021-08-09T17:30:15.006882460Z 2021-08-09 17:30:14.998 UTC [70] LOG: invalid length of startup packet
2021-08-09T17:30:16.112919592Z 2021-08-09 17:30:16.112 UTC [71] LOG: invalid length of startup packet
2021-08-09T17:30:17.178350344Z 2021-08-09 17:30:17.174 UTC [72] LOG: invalid length of startup packet
...
2021-08-09T17:33:25.735293291Z 2021-08-09 17:33:25.735 UTC [260] LOG: invalid length of startup packet
2021-08-09T17:33:29.142Z INFO - Container logs from testapp32_app_0_6662435d = 2021-08-09T17:30:24.010212694Z
2021-08-09T17:30:24.011085300Z You can now view your Streamlit app in your browser.
2021-08-09T17:30:24.011101800Z
2021-08-09T17:30:24.012092107Z Network URL: http://172.16.232.3:8501
2021-08-09T17:30:24.012855612Z External URL: http://20.40.148.207:8501
2021-08-09T17:30:24.013407416Z
2021-08-09T17:33:36.002Z INFO - Stopping site testapp32 because it failed during startup.
It's worth noting your shared example doesn't work because your docker-compose.yml is using your own private container registries.
However, tinkering around the edges I have managed to get your dummy example working with changes to the following files:
Your app dockerfile:
FROM python:3.7
RUN pip install streamlit
COPY app.py /streamlit-docker/
WORKDIR /streamlit-docker/
EXPOSE 80
CMD streamlit run app.py --server.port 80
And adjusting your docker-compose.yml
version: '3.3'
services:
db:
image: postgis/postgis:13-master
environment:
- POSTGRES_DB=counterfactualcovid
- POSTGRES_USER=django
- POSTGRES_PASSWORD=django
app:
image: testazurecontainerregistryajc.azurecr.io/mcmegaapp:latest
ports:
- '80:80'
Ignore the fact i've used a generic postgres docker image and my own private container registry for the images and you'll see the key changes are:
in the Dockerfile exposing port 80, and adding --server.port 80 to the streamlit CMD call
in the docker-compose.yml setting the port forwarding to 80:80
I /think/ your issues arise from the preview limitations for the multi-container web apps feature, so setting everything to work via port 80 appears to resolve this.

Can't reach postgres container from app container

I have a docker-composer.yml that is setting up two services: server and db. The Node.js server, which is the server service, uses pg to connect to the PostgreSQL database; and the db service is a PostgreSQL image.
On the server startup, it tries to connect to the database but gets a timeout.
docker-compose.yml
version: '3.8'
services:
server:
image: myapi
build: .
container_name: server
env_file: .env
environment:
- PORT=80
- DATABASE_URL=postgres://postgres:postgres#db:15432/mydb
- REDIS_URL=redis://redis
ports:
- 3000:80
depends_on:
- db
command: node script.js
restart: unless-stopped
db:
image: postgres
container_name: db
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: mydb
ports:
- 15432:15432
volumes:
- db-data:/var/lib/postgresql/data
command: -p 15432
restart: unless-stopped
volumes:
db-data:
Edit: code above changed to remove links and expose.
db service output:
db |
db | PostgreSQL Database directory appears to contain a database; Skipping initialization
db |
db | 2020-11-05 20:18:15.865 UTC [1] LOG: starting PostgreSQL 13.0 (Debian 13.0-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db | 2020-11-05 20:18:15.865 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 15432
db | 2020-11-05 20:18:15.865 UTC [1] LOG: listening on IPv6 address "::", port 15432
db | 2020-11-05 20:18:15.873 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.15432"
db | 2020-11-05 20:18:15.880 UTC [25] LOG: database system was shut down at 2020-11-05 20:18:12 UTC
db | 2020-11-05 20:18:15.884 UTC [1] LOG: database system is ready to accept connections
script.js - used by the command from the server service.
const pg = require('pg');
console.log(process.env.DATABASE_URL);
const pool = new pg.Pool({
connectionString: process.env.DATABASE_URL,
connectionTimeoutMillis: 5000,
});
pool.connect((err, _, done) => {
if (err) {
console.error(err);
done(err);
}
done();
});
pool.query('SELECT NOW()', (err, res) => {
console.log(err, res);
pool.end();
});
const client = new pg.Client({
connectionString: process.env.DATABASE_URL,
connectionTimeoutMillis: 5000,
});
client.connect(console.error);
client.query('SELECT NOW()', (err, res) => {
console.log(err, res);
client.end();
});
server service output:
NOTE: The first line is the output of the first console.log call from script.js.
NOTE: Since the server service is set up with restart: unless-stopped, it will repeat this output forever.
server | postgres://postgres:postgres#db:15432/mydb
server | Error: Connection terminated due to connection timeout
server | at Connection.<anonymous> (/home/node/app/node_modules/pg/lib/client.js:255:9)
server | at Object.onceWrapper (events.js:421:28)
server | at Connection.emit (events.js:315:20)
server | at Socket.<anonymous> (/home/node/app/node_modules/pg/lib/connection.js:78:10)
server | at Socket.emit (events.js:315:20)
server | at emitCloseNT (net.js:1659:8)
server | at processTicksAndRejections (internal/process/task_queues.js:79:21)
server | at runNextTicks (internal/process/task_queues.js:62:3)
server | at listOnTimeout (internal/timers.js:523:9)
server | at processTimers (internal/timers.js:497:7)
server | Error: Connection terminated due to connection timeout
server | at Connection.<anonymous> (/home/node/app/node_modules/pg/lib/client.js:255:9)
server | at Object.onceWrapper (events.js:421:28)
server | at Connection.emit (events.js:315:20)
server | at Socket.<anonymous> (/home/node/app/node_modules/pg/lib/connection.js:78:10)
server | at Socket.emit (events.js:315:20)
server | at emitCloseNT (net.js:1659:8)
server | at processTicksAndRejections (internal/process/task_queues.js:79:21)
server | at runNextTicks (internal/process/task_queues.js:62:3)
server | at listOnTimeout (internal/timers.js:523:9)
server | at processTimers (internal/timers.js:497:7) undefined
server | Error: timeout expired
server | at Timeout._onTimeout (/home/node/app/node_modules/pg/lib/client.js:95:26)
server | at listOnTimeout (internal/timers.js:554:17)
server | at processTimers (internal/timers.js:497:7)
server | Error: Connection terminated unexpectedly
server | at Connection.<anonymous> (/home/node/app/node_modules/pg/lib/client.js:255:9)
server | at Object.onceWrapper (events.js:421:28)
server | at Connection.emit (events.js:315:20)
server | at Socket.<anonymous> (/home/node/app/node_modules/pg/lib/connection.js:78:10)
server | at Socket.emit (events.js:315:20)
server | at emitCloseNT (net.js:1659:8)
server | at processTicksAndRejections (internal/process/task_queues.js:79:21) undefined
server | postgres://postgres:postgres#db:15432/mydb
...
From the host computer, I can reach the PostgreSQL database at the db service, connecting successfully, using the same script from the server service.
The output of the script running from the host computer:
➜ node script.js
postgres://postgres:postgres#localhost:15432/mydb
null Client { ... }
undefined Result { ... }
null Result { ... }
This output means the connection succeeded.
In summary:
I can't reach the db container from the server container, getting timeouts on the connection, but I can reach the db container from the host computer, connecting successfully.
Considerations
First, thanks for the answer so far. Addressing some points raised:
Missing network:
It isn't required because docker-compose has a default network. A tested with a custom network but it didn't work either.
Order of initialization:
I'm using depends_on to ensure the db container is started first but I know it isn't ensuring the database is in fact initialized first then the server. It isn't the problem because the server breaks when a timeout happens and it runs again because it is set up with restart: unless-stopped. So if the database is still initializing on the first or second try to start the server, there is no problem because the server will continue to be restarted until it succeeds in the connection (which never happened.)
UPDATE:
From the server container, I could reach the database at the db service using psql. I still can't connect from the Node.js app there.
The DATABASE_URL isn't the problem because the URI I used in the psql command is the same URI used by the script.js and printed by the first console.log call there.
Command-line used:
docker exec -it server psql postgres://postgres:postgres#db:15432/mydb
Edit: Improved code by removing the dependency for Sequelize. Now it uses only pg and calls the script directly.
Thanks for providing the source to reproduce the issue.
No issues in the docker-compose file as you have already ruled out.
The problem lies between your Dockerfile and the version of node-pg that you are using.
You are using node:14-alpine and pg: 7.18.2.
Turns out there is a bug on node 14 and earlier versions of node-pg.
Solution is either downgrade to node v12 or use latest version of node-pg which is currently 8.4.2 (fix went in on v8.0.3).
I have verified both these solutions on the branch you provided and they work.
This isn't a complete answer; I don't have your code handy so I can't actually test the compose file. However, there are a few issues there I'd like to point out:
The links directive is deprecated.
The links is a legacy option that was used before Docker introduced user-defined networks and automatic DNS support. You can just get rid of it. Containers in a compose file are able to refer to each other by name without.
The expose directive does nothing. It can be informative in for example a Dockerfile as a way of saying, "this image will expose a service on this port", but it doesn't actually make anything happen. It's almost entirely useless in a compose file.
The depends_on directive is also less useful than you would think. It will indeed cause docker-compose to bring up the database container first, but it the container is considered "up" as soon as the first process has started. It doesn't cause docker-compose to wait for the database to be ready to service requests, which means you'll still run into errors if your application tries to connect before the database is ready.
The best solution to this is to built database re-connection logic into your application so that if the database ever goes down (e.g. you restart the postgres container to activate a new configuration or upgrade the postgres version), the app will retry connections until it is successful.
An acceptable solution is to include code in your application startup that blocks until the database is responding to requests.
The problem has nothing to do with docker. To test that, perform following actions :
By using this docker-compose.yml file:
version: '3.8'
services:
app:
image: ubuntu
container_name: app
command: sleep 8h
db:
image: postgres
container_name: db
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: mydb
expose:
- '15432'
ports:
- 15432:15432
volumes:
- db-data:/var/lib/postgresql/data
command: -p 15432
restart: unless-stopped
volumes:
db-data:
Perform a docker exec -it app bash to go into container app then install postgresql-client with apt install -y postgresql-client`.
Command psql -h db -p 15432 -U postgres -W succeeded !
Check pg configuration
You say that pg use environment variable DATABASE_URL to reach postgresql. I'm not sure :
From https://node-postgres.com/features/connecting, we can found this example :
$ PGUSER=dbuser \
PGHOST=database.server.com \
PGPASSWORD=secretpassword \
PGDATABASE=mydb \
PGPORT=3211 \
node script.js
And this sentence :
node-postgres uses the same environment variables as libpq to connect to a PostgreSQL server.
In libpq documentation, no DATABASE_URL.
To adapt example provided in pg documentation with your docker-compose.yml file, try with following file (I only changed environments variables of app service) :
version: '3.8'
services:
server:
image: myapi
build: .
container_name: server
env_file: .env
environment:
- PORT=80
- PGUSER=postgres
- PGPASSWORD=postgres
- PGHOST=db
- PGDATABASE=mydb
- PGPORT=15432
- REDIS_URL=redis://redis
ports:
- 3000:80
depends_on:
- db
command: node script.js
restart: unless-stopped
db:
image: postgres
container_name: db
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: mydb
ports:
- 15432:15432
volumes:
- db-data:/var/lib/postgresql/data
command: -p 15432
restart: unless-stopped
volumes:
db-data:

Dockerized Node App unable to connect to Dockerized Postgres Database

I have tried many possible solutions to it, Not able to make any of them work.
Here it goes:
I built a nodejs container and a postgres docker container. Used docker compose to configure them both and used Dockerfile to build the nodejs/typescript application.
docker-compose.yml
version: "3"
volumes:
pg_vol:
networks:
app-network:
driver: bridge
services:
db:
container_name: db
image: postgres
ports:
- "5432:5432"
environment:
- POSTGRES_DB=db22
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=q123
volumes:
- pg_vol:/var/lib/postgresql/data
networks:
- app-network
webapp:
container_name: webapp
build:
context: ./
dockerfile: Dockerfile
environment:
- DATABASE_URL=postgres://postgres:q123#db:5432/db22
ports:
- "5000:5000"
networks:
- app-network
depends_on:
- db
Dockerfile
FROM node:14
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY ./dist .
CMD [ "npm", "start" ]
When I do docker-compose up --build, it shows an error while connecting to the postgres db. DB starts alright.
Error STDOUT
Starting db ... done
Starting webapp ... done
Attaching to db, webapp
db |
db | PostgreSQL Database directory appears to contain a database; Skipping initialization
db |
db | 2020-11-24 16:32:25.918 UTC [1] LOG: starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db | 2020-11-24 16:32:25.918 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db | 2020-11-24 16:32:25.918 UTC [1] LOG: listening on IPv6 address "::", port 5432
db | 2020-11-24 16:32:25.923 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db | 2020-11-24 16:32:25.928 UTC [25] LOG: database system was shut down at 2020-11-24 16:32:18 UTC
db | 2020-11-24 16:32:25.933 UTC [1] LOG: database system is ready to accept connections
webapp |
webapp | > backend-postgres#1.0.0 start /usr/src/app
webapp | > node app.js
webapp |
webapp | undefined
webapp | undefined
webapp | running in-code config
webapp | Error: connect EHOSTUNREACH 172.19.0.2:5432
webapp | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
webapp | errno: -113,
webapp | code: 'EHOSTUNREACH',
webapp | syscall: 'connect',
webapp | address: '172.19.0.2',
webapp | port: 5432
webapp | }
webapp | npm ERR! code ELIFECYCLE
webapp | npm ERR! errno 1
webapp | npm ERR! backend-postgres#1.0.0 start: `node app.js`
webapp | npm ERR! Exit status 1
webapp | npm ERR!
webapp | npm ERR! Failed at the backend-postgres#1.0.0 start script.
webapp | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
webapp |
webapp | npm ERR! A complete log of this run can be found in:
webapp | npm ERR! /root/.npm/_logs/2020-11-24T16_32_26_955Z-debug.log
webapp exited with code 1
Please help with correcting me.
System:
Fedora 33 (Workstation Edition)
EDIT:
I restarted the docker container with command docker start webapp. Same error as above.
Judging by logs: the app exits with 1 because the host at port 5432 was unreachable,maybe the database was not ready yet (you can verify it by checking if db22 is created on db container ).What you can is : add restart policy on webapp ,using this :
webap : ..
restart_policy: condition: on-failure
then check then if the problem persists
My system is Fedora 33 (Workstation Edition). This had some issues with Docker.
Turns out:
With the release of Fedora 33, Docker is not supported by Fedora 33 officially. Oct 29, 2020
I uninstalled docker. Rebooted my laptop.
Then install Podman and Podman Compose. Everything works perfectly fine.
Thank you all for help.

Setting up Docker with Knex.js and PostgreSQL

Today I've been trying to setup Docker for my GraphQL API that runs with Knex.js, PostgreSQL and Node.js. The problem that I'm facing is that Knex.js is timing out when trying to access data in my database. I believe that it's due to my incorrect way of trying to link them together.
I've really tried to do this on my own, but I can't figure this out. I'd like to walk through each file that play a part of making this work so (hopefully) someone could spot my mistake(s) and help me.
knexfile.js
In my knexfile I have a connection key which used to look like this:
connection: 'postgres://localhost/devblog'
This worked just fine, but this wont work if I want to use Docker. So I modified it a bit and ended up with this:
connection: {
host: 'db' || 'localhost',
port: process.env.DB_PORT || 5432,
user: process.env.DB_USER || 'postgres',
password: process.env.DB_PASSWORD || undefined,
database: process.env.DATABASE // DATABASE = devblog
}
I've noticed that something is wrong with host. Since it always times-out when I have something else (in this case, db) than localhost.
Dockerfile
My Dockerfile looks like this:
FROM node:9
WORKDIR /app
COPY package-lock.json /app
COPY package.json /app
RUN npm install
COPY dist /app
COPY wait-for-it.sh /app
CMD node index.js
EXPOSE 3010
docker-compose.yml
And this file looks like this:
version: "3"
services:
redis:
image: redis
networks:
- webnet
db:
image: postgres
networks:
- webnet
environment:
POSTGRES_PASSWORD: password
POSTGRES_USER: martinnord
POSTGRES_DB: devblog
web:
image: node:8-alpine
command: "node /app/dist/index.js"
volumes:
- ".:/app"
working_dir: "/app"
depends_on:
- "db"
ports:
- "3010:3010"
environment:
DB_PASSWORD: password
DB_USER: martinnord
DB_NAME: devblog
DB_HOST: db
REDIS_HOST: redis
networks:
webnet:
When I try to run this with docker-compose up I get the following output:
Starting backend_db_1 ... done
Starting backend_web_1 ... done
Attaching to backend_db_1, backend_redis_1, backend_web_1
redis_1 | 1:C 12 Feb 16:05:21.303 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 12 Feb 16:05:21.303 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
db_1 | 2018-02-12 16:05:21.337 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
redis_1 | 1:C 12 Feb 16:05:21.303 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 12 Feb 16:05:21.311 * Running mode=standalone, port=6379.
db_1 | 2018-02-12 16:05:21.338 UTC [1] LOG: listening on IPv6 address "::", port 5432
redis_1 | 1:M 12 Feb 16:05:21.311 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 12 Feb 16:05:21.314 # Server initialized
redis_1 | 1:M 12 Feb 16:05:21.315 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
db_1 | 2018-02-12 16:05:21.348 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-02-12 16:05:21.367 UTC [20] LOG: database system was shut down at 2018-02-12 16:01:17 UTC
redis_1 | 1:M 12 Feb 16:05:21.317 * DB loaded from disk: 0.002 seconds
redis_1 | 1:M 12 Feb 16:05:21.317 * Ready to accept connections
db_1 | 2018-02-12 16:05:21.374 UTC [1] LOG: database system is ready to accept connections
web_1 | DB_HOST db
web_1 |
web_1 | App listening on 3010
web_1 | Env: undefined
But when I try to make a query with GraphQL I get:
"message": "Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?"
I really don't know why this is not working for me and it's driving me nuts. If anyone could help me with this I would be delighted. I have also added a link to my project below.
Thanks a lot for reading! Cheers.
LINK TO PROJECT: https://github.com/Martinnord/DevBlog/tree/master/backend
Updated docker-compose file:
version: "3"
services:
redis:
image: redis
networks:
- webnet
db:
image: postgres
networks:
- webnet
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: martinnord
POSTGRES_DB: devblog
ports:
- "15432:5432"
web:
image: devblog-server
ports:
- "3010:3010"
networks:
- webnet
environment:
DB_PASSWORD: password
DB_USER: martinnord
DB_NAME: devblog
DB_HOST: db
REDIS_HOST: redis
command: ["./wait-for-it.sh", "db:5432", "--", "node", "index.js"]
networks:
webnet:
Maybe like this:
version: "3"
services:
redis:
image: redis
db:
image: postgres
environment:
POSTGRES_PASSWORD: password
POSTGRES_USER: martinnord
POSTGRES_DB: devblog
ports:
- "15432:5432"
web:
image: node:8-alpine
command: "node /app/dist/index.js"
volumes:
- ".:/app"
working_dir: "/app"
depends_on:
- "db"
ports:
- "3010:3010"
links:
- "db"
- "redis"
environment:
DB_PASSWORD: password
DB_USER: martinnord
DB_NAME: devblog
DB_HOST: db
REDIS_HOST: redis
And you should be able to connect from webapp to postgres with:
postgres://martinnord:password#db/devblog
or
connection: {
host: process.DB_HOST,
port: process.env.DB_PORT || 5432,
user: process.env.DB_USER || 'postgres',
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME || 'postgres'
}
I also added row that exposes postgres running in docker to a port 15432 so you can try to connect it first directly from your host machine with
psql postgres://martinnord:password#localhost:15432/devblog

Resources