In sequelize connection I am getting operation timeout error. How to fix this issue? - node.js

I am trying to run node JS server & Postgres inside docker & using sequalize for DB Connection. However, Seems like my Node JS Server is not able to communicate with Postgres DB inside docker.
Before someone mark it as Duplicate, Please note that I have already checked other answers & none of them worked out for me.
I already tried implementing Retry Strategy for Sequalize connection.
Here's my docker-compose file:
version: "3.8"
services:
rapi:
container_name: rapi
image: rapi/latest
build: .
ports:
- "3001:3001"
environment:
- EXTERNAL_PORT=3001
- PGUSER=rapiuser
- PGPASSWORD=12345
- PGDATABASE=postgres
- PGHOST=rapi_db # NAME OF THE SERVICE
depends_on:
- rapi_db
rapi_db:
container_name: rapi_db
image: "postgres:12"
ports:
- "5432:5432"
environment:
- POSTGRES_USER=rapiuser
- POSTGRES_PASSWORD=12345
- POSTGRES_DB=postgres
volumes:
- rapi_data:/var/lib/postgresql/data
volumes:
rapi_data: {}
Here's my Dockerfile:
FROM node:16
EXPOSE 3000
# Use latest version of npm
RUN npm i npm#latest -g
COPY package.json package-lock.json* ./
RUN npm install --no-optional && npm cache clean --force
# copy in our source code last, as it changes the most
WORKDIR /
COPY . .
CMD [ "node", "index.js" ]
My DB Credentials:
credentials = {
PGUSER :process.env.PGUSER,
PGDATABASE :process.env.PGNAME,
PGPASSWORD : process.env.PGPASSWORD,
PGHOST : process.env.PGHOST,
PGPORT:process.env.PGPORT,
PGNAME:'postgres'
}
console.log("env Users: " + process.env.PGUSER + " env Database: " + process.env.PGDATABASE + " env PGHOST: " + process.env.PGHOST + " env PORT: " + process.env.EXTERNAL_PORT)
}
//else credentials = {}
module.exports = credentials;
Sequalize DB code:
const db = new Sequelize(credentials.PGDATABASE,credentials.PGUSER,credentials.PGPASSWORD, {
host: credentials.PGHOST,
dialect: credentials.PGNAME,
port:credentials.PGPORT,
protocol: credentials.PGNAME,
dialectOptions: {
},
logging: false,
define: {
timestamps: false
}
,
pool: {
max: 10,
min: 0,
acquire: 100000,
},
retry: {
match: [/Deadlock/i, Sequelize.ConnectionError], // Retry on connection errors
max: 3, // Maximum retry 3 times
backoffBase: 3000, // Initial backoff duration in ms. Default: 100,
backoffExponent: 1.5, // Exponent to increase backoff each try. Default: 1.1
},
});
module.exports = db;

Your process.env.PGPORT does not exists. Add an enviroment variable in the docker-compose for service rapi or set it to 5432 in your credentials file.

Related

ampqlib Error:"Frame size exceeds frame max" inside docker container

I am trying to do simple application with backend on node.js + ts and rabbitmq, based on docker. So there are 2 containers: rabbitmq container and backend container with 2 servers running - producer and consumer. So now I am trying to get an access to rabbitmq server, but I get this error "Frame size exceeds frame max".
The full code is:
My producer server code is:
import express from 'express';
import amqplib, { Connection, Channel, Options } from 'amqplib';
const producer = express();
const sendRabbitMq = () =>{
amqplib.connect('amqp://localhost', function(error0: any, connection: any) {
if(error0){
console.log('Some error...')
throw error0
}
})
}
producer.post('/send', (_req, res) => {
sendRabbitMq();
console.log('Done...');
res.send("Ok")
})
export { producer };
It is connected to main file index.ts and running inside this file.
Also maybe I have some bad configuration inside docker. My Dockerfile is
FROM node:16
WORKDIR /app/backend/src
COPY *.json ./
RUN npm install
COPY . .
And my docker-compose include this code:
version: '3'
services:
backend:
build: ./backend
container_name: 'backend'
command: npm run start:dev
restart: always
volumes:
- ./backend:/app/backend/src
- ./conf/myrabbit.conf:/etc/rabbitmq/rabbitmq.config
ports:
- 3000:3000
environment:
- PRODUCER_PORT=3000
- CONSUMER_PORT=5672
depends_on:
- rabbitmq
rabbitmq:
image: rabbitmq:3.9.13
container_name: 'rabbitmq'
ports:
- 5672:5672
- 15672:15672
environment:
- RABBITMQ_DEFAULT_USER=user
- RABBITMQ_DEFAULT_PASS=user
I will be very appreciated for your help

TypeORM migration:run not working errno: -3008,getaddrinfo ENOTFOUND

Do you know why I get the following error when I try to run typeorm:run to execute migration?
node --require ts-node/register ./node_modules/typeorm/cli.js migration:run
Error during migration run:
Error: getaddrinfo ENOTFOUND users-service-db
at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:69:26) {
errno: -3008,
code: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: 'users-service-db',
fatal: true
}
error Command failed with exit code 1.
my config is
users-service-db:
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=db
image: mysql:5.7.20
ports:
- "7201:3306"
the users-service-db is running does this Error: getaddrinfo ENOTFOUND users-service-db say that the host doesn't know what to do. Can you help?
After trying Answer 1 and 2 still getting the same error don't know what to do it worked before?
version: "3"
services:
api-gateway:
build:
context: "."
dockerfile: "./api-gateway/Dockerfile"
depends_on:
- chat-service
- users-service
ports:
- "7000:7000"
volumes:
- ./api-gateway:/opt/app
chat-service:
build:
context: "."
dockerfile: "./chat-service/Dockerfile"
depends_on:
- chat-service-db
ports:
- "7100:7100"
volumes:
- ./chat-service:/opt/app
chat-service-db:
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=db
image: mysql:5.7.20
ports:
- "7200:3306"
phpmyadmin:
image: phpmyadmin/phpmyadmin
ports:
- "7300:80"
volumes:
- ./phpmyadmin/config.user.inc.php:/etc/phpmyadmin/config.user.inc.php
users-service:
build:
context: "."
dockerfile: "./users-service/Dockerfile"
depends_on:
- users-service-db
ports:
- "7101:7101"
volumes:
- ./users-service:/opt/app
users-service-db:
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=db
image: mysql:5.7.20
ports:
- "7201:3306"
hostname: 'localhost'
finally I resolved the error thanks to #Eranga Heshan
I created an additional ormConfig.js file at pasted this:
export = {
"type": "mysql",
"host": "localhost",
"port": 7201,
"username": "root",
"password": "password",
"database": "db",
"synchronize": true,
"logging": false,
"entities": [
"src/entities/**/*.ts"
],
"migrations": [
"./src/db/migrations/**/*.ts"
],
"cli": {
"entitiesDir": "src/db/entities",
"migrationsDir": "src/db/migrations"
}
}
then
node --require ts-node/register ./node_modules/typeorm/cli.js migration:run --config src/db/migrations/ormConfig
Your VS Code terminal is running inside your machine. So it can't resolve users-service-db host.
You can do this in two ways.
1. Using a new config file and execute migrations from your localhost
Create a new typeorm connection config file migrationsOrmConfig.ts and put it inside your project (Let's say you put it in src/migrations directory)
export = {
host: 'localhost',
port: '7201',
type: 'mysql',
user : 'root',
password : 'password',
database : 'db' ,
};
Now you can modify the command you used earlier to run migrations
node --require ts-node/register ./node_modules/typeorm/cli.js migration:run --config src/migrations/migrationsOrmConfig
2. Execute migrations from a terminal within the container
In your VSCode terminal type
docker ps -a
Get the CONTAINER ID of user-service (Let's say it is CONTAINER_ID)
Open up a terminal inside the container
docker exec -it CONTAINER_ID /bin/bash
Execute the command you used earlier to run migrations (if the following command complained about typeorm node module not being found, you can install it inside the container)
node --require ts-node/register ./node_modules/typeorm/cli.js migration:run

(Docker-Compose) UnhandledPromiseRejectionWarning when connecting node and postgres

I am trying to connect the containers for postgres and node. Here is my setup:
yml file:
version: "3"
services:
postgresDB:
image: postgres:alpine
container_name: postgresDB
ports:
- "5432:5432"
environment:
- POSTGRES_DB=myDB
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=Thisisngo1995!
express-server:
build: ./
environment:
- DB_SERVER=postgresDB
links:
- postgresDB
ports:
- "3000:3000"
Dockerfile:
FROM node:12
WORKDIR /usr/src/app
COPY package.json ./
RUN npm install
COPY . .
COPY ormconfig.docker.json ./ormconfig.json
EXPOSE 3000
CMD ["npm", "start"]
connect to postgres:
let { Pool, Client } = require("pg");
let postgres = new Pool({
host: "postgresDB",
port: 5432,
user: "postgres",
password: "Thisisngo1995!",
database: "myDB",
});
module.exports = postgres;
and here is how I handled my endpoint:
exports.postgres_get_controller = (req, resp) => {
console.log("Reached Here");
postgres
.query('SELECT * FROM public."People"')
.then((results) => {
console.log(results);
resp.send({ allData: results.rows });
})
.catch((e) => console.log(e));
};
Whenever I try to touch the endpoint above, I get this error in the container:
Reasons why?
Note: I am able to have everything functioning on my local machine (without docker) simply by changing "host: localhost"
Your postgres database name and username should be the same
You can use docker-compose-wait to make sure interdependent services are launched in proper order.
See below on how to use it for your case.
update the final part of your Dockerfile as below;
# ...
# this will be used to check if DB is up
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.7.3/wait ./wait
RUN chmod +x ./wait
CMD ./wait && npm start
Update some parts of your docker-compose.yml as below:
express-server:
build: ./
environment:
- DB_SERVER=postgresDB
- WAIT_HOSTS=postgresDB:5432
- WAIT_BEFORE_HOSTS=4
links:
- postgresDB
depends_on:
- postgresDB
ports:
- "3000:3000"

Not able to connect to Elasticsearch from docker container (node.js client)

I have set up an elasticsearch/kibana docker configuration and I want to connect to elasticsearch from inside of a docker container using the #elastic/elasticsearch client for node. However, the connection is "timing out".
The project is taken with inspiration from Patrick Triest : https://blog.patricktriest.com/text-search-docker-elasticsearch/
However, I have made some modification in order to connect kibana, use a newer ES image and the new elasticsearch node client.
I am using the following docker-compose file:
version: "3"
services:
api:
container_name: mp-backend
build: .
ports:
- "3000:3000"
- "9229:9229"
environment:
- NODE_ENV=local
- ES_HOST=elasticsearch
- PORT=3000
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
container_name: elasticsearch
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "http.cors.allow-origin=*"
- "http.cors.enabled=true"
- "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization"
- "http.cors.allow-credentials=true"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana:7.5.1
ports:
- "5601:5601"
links:
- elasticsearch
networks:
- elastic
depends_on:
- elasticsearch
volumes:
data01:
driver: local
networks:
elastic:
driver: bridge
When building/ bringing the container up, I able to get a response from ES: curl -XGET "localhost:9200", "you know, for search"... And kibana is running and able to connect to the index.
I have the following file located in the backend container (connection.js):
const { Client } = require("#elastic/elasticsearch");
const client = new Client({ node: "http://localhost:9200" });
/*Check the elasticsearch connection */
async function health() {
let connected = false;
while (!connected) {
console.log("Connecting to Elasticsearch");
try {
const health = await client.cluster.health({});
connected = true;
console.log(health.body);
return health;
} catch (err) {
console.log("ES Connection Failed", err);
}
}
}
health();
If I run it outside of the container then I get the expected response:
node server/connection.js
Connecting to Elasticsearch
{
cluster_name: 'es-docker-cluster',
status: 'yellow',
timed_out: false,
number_of_nodes: 1,
number_of_data_nodes: 1,
active_primary_shards: 7,
active_shards: 7,
relocating_shards: 0,
initializing_shards: 0,
unassigned_shards: 3,
delayed_unassigned_shards: 0,
number_of_pending_tasks: 0,
number_of_in_flight_fetch: 0,
task_max_waiting_in_queue_millis: 0,
active_shards_percent_as_number: 70
}
However, if I run it inside of the container:
docker exec mp-backend "node" "server/connection.js"
Then I get the following response:
Connecting to Elasticsearch
ES Connection Failed ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at onResponse (/usr/src/app/node_modules/#elastic/elasticsearch/lib/Transport.js:214:13)
at ClientRequest.<anonymous> (/usr/src/app/node_modules/#elastic/elasticsearch/lib/Connection.js:98:9)
at ClientRequest.emit (events.js:223:5)
at Socket.socketErrorListener (_http_client.js:415:9)
at Socket.emit (events.js:223:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:81:21) {
name: 'ConnectionError',
meta: {
body: null,
statusCode: null,
headers: null,
warnings: null,
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 3,
aborted: false
}
}
}
So, I tried changing the client connection to (I read somewhere that this might help):
const client = new Client({ node: "http://172.24.0.1:9200" });
Then I am just "stuck" waiting for a response. Only one console.log of "Connecting to Elasticsearch"
I am using the following version:
"#elastic/elasticsearch": "7.5.1"
As you probably see, I do not have a full grasp of what is happening here... I have also tried to add:
links:
- elasticsearch
networks:
- elastic
To the api service, without any luck.
Does anyone know what I am doing wrong here? Thank you in advance :)
EDIT:
I did a "docker network inspect" on the network with *_elastic. There I see the following:
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.22.0.0/16",
"Gateway": "172.22.0.1"
}
]
},
Changing the client to connect to the "GateWay" Ip:
const client = new Client({ node: "http://172.22.0.1:9200" });
Then it works! I am still wondering why as this was just "trial and error" Is there any way to obtain this Ip without having to inspect the network?
In Docker, localhost (or the corresponding IPv4 address 127.0.0.1, or the corresponding IPv6 address ::1) generally means "this container"; you can't use that host name to access services running in another container.
In a Compose-based setup, the names of the services: blocks (api, elasticsearch, kibana) are usable as host names. The caveat is that all of the services have to be on the same Docker-internal network. Compose creates one for you and attaches containers to it by default. (In your example api is on the default network but the other two containers are on a separate elastic network.) Networking in Compose in the Docker documentation has some more details.
So to make this work, you need to tell your client code to honor the environment variable you're setting that points at Elasticsearch
const esHost = process.env.ES_HOST || 'localhost';
const esUrl = 'http://' + esHost + ':9200';
const client = new Client({ node: esUrl });
In your docker-compose.yml file delete all of the networks: blocks to use the provided default network. (While you're there, links: is unnecessary and Compose provides reasonable container_name: for you; api can reasonably depends_on: [elasticsearch].)
Since we've provided a fallback for $ES_HOST, if you're working in a host development environment, it will default to using localhost; outside of Docker where it means "the current host" it will reach the published port of the Elasticsearch container.

Docker Health Check and Node.js Application

I've been trying to include healthcheck to my container, however, no matter what i do, container never seems to work. To be precise, i have the following structure:
Traefik Proxy
Node.js Application behind that proxy
All the labels for Traefik are included in the docker-compose.yml file.
Whenever i try to add healthcheck, either in Dockerfile or in docker-compose.yml, application is built and listening for connections on port 443, however, when i try to access address from the browser, it always shows 404 error (when Traefik is unable to proxy container).
Here is simple service configuration:
frontend:
restart: always
build:
context: ./configuration/frontend
dockerfile: Dockerfile
environment:
- application_environment=development
- FRONTEND_DOMAIN=HOST_HERE
volumes:
- ./volumes/frontend:/app:rw
- ./volumes/backend/.env:/.env:ro
- ./volumes/backend/resources/lang:/backend-lang:rw
labels:
- traefik.enable=true
- traefik.frontend.rule=Host:HOST_HERE
- traefik.port=3000
- traefik.docker.network=traefik_proxy
- traefik.frontend.redirect.entryPoint=https
- traefik.frontend.passHostHeader=true
- traefik.frontend.headers.SSLRedirect=true
- traefik.frontend.headers.browserXSSFilter=true
- traefik.frontend.headers.contentTypeNosniff=true
- traefik.frontend.headers.customFrameOptionsValue=SAMEORIGIN
- traefik.frontend.headers.STSPreload=true
- traefik.frontend.headers.STSSeconds=31536000
healthcheck:
test: ["CMD", "cd /app && yarn healthcheck"]
interval: 10s
timeout: 5s
start_period: 60s
networks:
- traefik_proxy
And here is healthcheck.js file which is accessible through the command yarn healthcheck:
const http = require('https');
const options = {
host: process.env.FRONTEND_DOMAIN,
port: 443,
path: '/',
method: 'GET',
timeout: 2000
};
const healthCheck = http.request(options, (response) => {
console.log(`STATUS: ${response.statusCode}`);
if (response.statusCode === 200) {
process.exit(0);
}
else {
process.exit(1);
}
});
healthCheck.on('error', function (error) {
console.error('ERROR', error);
process.exit(1);
});
healthCheck.end();
When I start the container without the HEALTHCHECK options (either Dockerfile or compose), it works just fine, page is displayed and when I manually execute yarn healthcheck, then it shows that everything is fine (I mean in the console, STATUS: 200). However, with automated healthcheck, Traefik will have no access to the container.

Resources