My problem is I have a script that should scrap data and put it inside postgres database, however it has a problem to reach out postgres container.
When I run my docker-compose here is the result:
Name Command State Ports
------------------------------------------------------------------------------------------
orcsearch_dev-db_1 docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
orcsearch_flask_1 gunicorn wsgi:application ... Up 0.0.0.0:80->80/tcp, 8000/tcp
We can clearly see that postgres is on 5432 port.
This is my python script database setting:(ofcourse I removed password for obvious reason)
class App():
settings = {
'db_host': 'db',
'db_user': 'postgres',
'db_pass': '',
'db_db': 'orc',
}
db = None
proxies = None
and this is my docker-compose.yml
version: '2'
services:
flask:
build:
context: ./backend
dockerfile: Dockerfile.dev
volumes:
- ./backend:/app
- pip-cache:/root/.cache
ports:
- "80:80"
links:
- "dev-db:db"
environment:
- DATABASE_URL=postgresql://postgres#db:5432/postgres
stdin_open: true
command: gunicorn wsgi:application -w 1 --bind 0.0.0.0:80 --log-level debug --reload
networks:
app:
aliases:
- flask
dev-db:
image: postgres:9.5
ports:
- "5432:5432"
networks:
app:
aliases:
- dev-db
volumes:
pip-cache:
driver: local
networks:
app:
When going into exec flask bash(inside flask container) and running script command I get this error:
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Obviously there is postgres running on this port and I cant figure out what wrong do I do. Any help would be nice!
Probably you are using DSN instead of URI, and PostgreSQL thinks that "db" is not a host because it's hard to tell if "db" is host or path to socket. To fix it, use URI instead of DSN if you use >=9.2 version of PostgreSQL.
Example of URI:
postgresql://[user[:password]#][netloc][:port][/dbname][?param1=value1&...]
https://www.postgresql.org/docs/9.2/static/libpq-connect.html#LIBPQ-CONNSTRING
In your App class it should be 'db_host': 'dev-db', Seems like that hostname is exposed, not db.
I think that the problem is related to the fact that you're using the network and the link together. Try remove the link and change the postgres address to dev-db or change the alias to:
networks:
app:
aliases:
- dev-db
- db
Related
Internal connection between my Spring Boot app and postgres db works fine but when I want to connect to db through init_database.sh and pg_restore I get pg_restore: error: connection to server at "db" (192.168.224.2), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?. In this .sh script I use the same host and port as in application.properties (where connection works fine as I said before).
I simply want to do a pg_restore from init_database.sh in my Docker container.
docker-compose.yml
services:
db:
container_name: db
hostname: db
image: postgres:14-alpine
command: -c 'max_connections=250'
ports:
- "5430:5432"
environment:
POSTGRES_DB: endlessblow_db
POSTGRES_USER: kuba
POSTGRES_PASSWORD: pass
volumes:
- ./init.dump:/init.dump
- ./init_database.sh:/docker-entrypoint-initdb.d/init_database.sh
restart: always
app:
container_name: app
image: 'endlessblow-server:latest'
build: ./
ports:
- "8000:8080"
depends_on:
- db
init_database.sh
#!/bin/sh
pg_restore --no-privileges --no-owner -h db -p 5432 -U kuba -d endlessblow_db init.dump
application.properties
spring.datasource.jdbc-url=jdbc:postgresql://db:5432/endlessblow_db
spring.datasource.url=jdbc:postgresql://db:5432/endlessblow_db
spring.datasource.username=kuba
spring.datasource.password=pass
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.platform=postgres
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect
How to solve that? Thank you!
in config of service "app", try following:
app:
...
depends_on:
- db
links:
- db
without links, docker network will isolate containers without explicitly exposing them to one another.
I have the following compose file
version: '3'
services:
db:
image: "postgres:12.4"
ports:
- "15432:5432"
env_file:
- ./db/database.env
volumes:
- ./db/data:/var/lib/postgresql/data
- ./db/init:/init
- ./db/init/run.sh:/docker-entrypoint-initdb.d/initialise.sh
pgadmin:
container_name: pgadmin4_container
image: dpage/pgadmin4
network_mode: bridge
environment:
PGADMIN_DEFAULT_EMAIL: xxx
PGADMIN_DEFAULT_PASSWORD: xxx
ports:
- "5050:80"
extra_hosts:
- "host.docker.internal:host-gateway"
Now when I run this locally on my Mac, all is good. I get a database which my node API can reach on localhost, and I get a pgadmin, which can connect to the database by using host.docker.internal.
When I run this on ubuntu linux, however, pgadmin tells me it can't connect to the database. Bear in a mind the non-docker node api can connect to the db just fine on localhost.
I've tried:
with and without the extra-hosts bit
on pgadmin trying to connect to the db with 'localhost', '127.0.0.1', 'host.docker.internal', '172.17.0.1' and the public IP
changing network mode to bridge, host and removing it entirely, all while testing the aforementioned host names.
This is a minimal setup that works. You can use it as a starting point:
version: '3'
services:
db:
image: "postgres:12.4"
environment:
POSTGRES_PASSWORD: "password"
pgadmin:
image: "dpage/pgadmin4"
environment:
PGADMIN_DEFAULT_EMAIL: xxx#xxx.xxx
PGADMIN_DEFAULT_PASSWORD: xxxxxx
ports:
- "5050:80"
Then navigate to http://localhost:5050 and login with xxx#xxx.xxx / xxxxxx.
Add a new server where the host is db (service name in docker-compose.yml), user is postgres and password is password (set in docker-compose.yml).
Notes:
You use pgadmin to access the database so no need to expose database ports.
Services see each other by name: pgadmin and db in your case.
Services run on a separate, "hidden" network and you only access to that is through a mapped port: 5050 in this case.
I am a beginner in node, nest, and docker but somehow I got assigned a job to dockerized all the existing node js applications.
I followed one of the youtube tutorial and successfully deployed the basic hello world via docker but in the next youtube tutorial when I am trying to add Postgres to the docker I am facing some issues in connecting to Postgres.
I am using docker desktop on mac.
Here is my docker-compose.yml file code snippet
version: "3.9" # optional since v1.27.0
services:
api:
build:
dockerfile: Dockerfile
context: .
depends_on:
- postgres
environment:
DATABASE_URL: postgres://user:password#postgres:5432/db
NODE_ENV: developement
PORT: 3000
ports:
- "8080:3000"
postgres:
image: postgres:14.0
ports:
- "35000:5432"
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: password
POSTGRES_DB: db
Here is the entire error log
Github Repository of this project
Thank you for helping in advance :)
Your problem for a typo in DATABASE_URL. In code for connect database use DATABSE_URL word but in docker-compose used DATABASE_URL.
You should change url: process.env.DATABSE_URL to url: process.env.DATABASE_URL
Make sure your connection string is correct in your docker-compose.yml. Just pass the host, port, user and pass seperated and let TypeOrm handle the connection.
// app.module.ts
TypeOrmModule.forRoot({
type: 'postgres',
host: process.env.POSTGRES_HOST,
port: process.env.POSTGRES_PORT,
username: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: process.env.POSTGRES_DB,
})
And your docker-compose.yml:
# docker-compose.yml
version: '3.9'
services:
api:
build:
dockerfile: Dockerfile
context: .
depends_on:
- postgres
environment:
- POSTGRES_HOST=postgres
- POSTGRES_PASSWORD=promo-pass
- POSTGRES_USER=promo-user
- POSTGRES_DB=promo-api-db
- POSTGRES_PORT=5432
postgres:
container_name: postgres
image: postgres
environment:
POSTGRES_USER: promo-user
POSTGRES_PASSWORD: promo-pass
POSTGRES_DB: promo-api-db
In the normal case, without Docker, i. e. you are using node and postgresql on you development or production machine, you just want to start postgres service and enable it if you want.
In order to start your postgres service, type the following command:
sudo systemctl start postgresql
To enable :
sudo systemctl enable postgresql
Note:
"Enable" means enbable the postgres server at boot time.
In my environment, i am using Red Hat. So if commands doesn't work, find the corresponding command on your linux distribution or on your specific OS.
I hope this can help someone else!
I'm running an instance of a web application in my Docker container and am also running a MongoDB container so when I launch the web app I can easily connect to the DB on the app's connection page.
The issue is that I'm not sure how to reach the Mongo container from my web app and am not sure if my host/port connection info is correct.
My Docker Setup
As you can see the container is up and running with both mongo and web app services running without errors
I build the two through docker-compose.yml
version: "3.3"
services:
web:
image: grafana-asw-v3
container_name: grafana-asw-v3
restart: always
build: .
ports:
- "13000:3000"
volumes:
- grafana-storage:/var/lib/grafana
stdin_open: true
tty: true
db:
container_name: mongo
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
volumes:
- grafana-mongo-db:/var/lib/mongo
ports:
- "27018:27017"
volumes:
grafana-mongo-db: {}
grafana-storage: {}
Issue
With everything up and running I'm attempting to connect through the web app, but I seem to be using the wrong connection info...
I assumed to use "hostMachine:port" (roxane:27018), but it's not connecting. Is there something I overlooked here?
There were two changes I had to make to fix this issue:
Modify the bind_ip in mongod.conf via making this change to my docker-compose file
db:
container_name: mongo
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
volumes:
- grafana-mongo-db:/var/lib/mongo
ports:
- "27018:27017"
command: mongod --bind_ip 0.0.0.0
I needed to refer to the IP address instead of the hostname in the cli in my we application. (Thanks to this answer for help with this one)
Short answer
db service is in the same network than web service not in host network.
As you named your services via container_name you shoud be able to use the connection string mongodb://mongo:27017
Explanation
By default, docker containers run under a bridge network allowing them to communicate without viewing your host network.
When using ports in a compose file, you define that you want to map an internal port of the container to the host port
"27018:27017" => I want to expose the container port number 27017 to the host port number 27018.
As a result, you could expose your web frontend without exposing your mongo service :
version: "3.3"
services:
web:
image: grafana-asw-v3
container_name: grafana-asw-v3
restart: always
build: .
ports:
- "13000:3000"
volumes:
- grafana-storage:/var/lib/grafana
stdin_open: true
tty: true
db:
container_name: mongo
image: mongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
volumes:
- grafana-mongo-db:/var/lib/mongo
volumes:
grafana-mongo-db: {}
grafana-storage: {}
Code first, it will be easier to explain what I'm after.
docker-compose.yml
version: '3.4'
services:
db:
user: '${UID}:${GID}'
image: postgres
container_name: postgres
ports:
- '5432:5432'
restart: always
environment:
POSTGRES_HOST: db
POSTGRES_USER: root
POSTGRES_PASSWORD: secret
POSTGRES_DATABASE: foo
PGDATA: /var/lib/postgresql/data/db
volumes:
- ./db/data:/var/lib/postgresql/data/db
- db-init.sh:/docker-entrypoint-initdb.d/:ro
cache:
image: redis:alpine
container_name: redis
sysctls:
net.core.somaxconn: '511'
ports:
- '6379:6379'
command: ['--requirepass "secret"']
api:
image: node:alpine
container_name: api
working_dir: /var/www/app
command: sh -c "npm start"
ports:
- '5000:5000'
volumes:
- node_modules:/var/www/app/node_modules
- .:/var/www/app
env_file: .env
depends_on:
- db
- cache
volumes:
node_modules:
postgres connection settings for node.js app:
export const {
POSTGRES_USER = 'root',
POSTGRES_PASSWORD = 'secret',
POSTGRES_HOST = 'db',
POSTGRES_PORT = 5432,
POSTGRES_DATABASE = 'foo',
} = process.env
issue:
When using service or container name (db or postgres) for the POSTGRES_HOST setting of node app:
I can successfully connect to, and query, the database.
I'm not able to run commands from host which affect the container. For example, seeding db won't work:
npx knex --esm seed:run
This makes sense, as the DNS resolution for db / postgres is taken care of by docker and those only have meaning on the network connecting the containers. Commands run from the host and targeting that container will fail as host doesn't know how to resolve the DNS here.
On the other hand, when using localhost for the POSTGRES_HOST setting of node app:
Queries to postgres from api will fail.
Commands run from the host, like npx knex --esm seed:run, will succeed.
Again, this makes perfect sense. Addressing container as localhost from host will work thanks to the port forwarding in docker-compose.yml. But in the context of the container, it refers to that very container: for api localhost means itself, and its trying to find a database on localhost:5432 or api:5432.
I want to have working inter-container network and also run commands from the host, addressing the said containers. I'm aware of two approaches to achieve that:
Use container / service name as POSTGRES_HOST, and run commands against the containers with:
docker exec -it <container_name> <command>
Assign static ips to the containers and use those instead of service / container names.
Do I have any other options here?
since you are exposing the database ports on the host machine you can do the following.
Use service or container name (db or postgres) for the POSTGRES_HOST, this way it will work for Docker containers.
when you run the seed command form the host, overwrite the POSTGRES_HOST. This can be done in this way
$ export POSTGRES_HOST=127.0.0.1
$ npx knex --esm seed:run
or in one step
$ POSTGRES_HOST=127.0.0.1 npx knex --esm seed:run