Failed to connect ElastiCache from NodeJS server on Elastic Beanstalk - node.js

We have a nodeJS server with express on AWS Elastic Beanstalk and we are trying to connect it with the Elasticache(Redis clustered) from the NodeJS but getting this error Redis Client Connection Error ClusterAllFailedError: Failed to refresh slots cache.. The error seems very common as a lot of people are facing the same bug. In order to connect to ElastiCache, we are using an npm module named ioredis.
A lot of people recommend using the same VPC and security group for both ElastiCache and Elastic Beanstalk. We are already using the same VPC and on Elastic Beanstalk we are using two security groups one of them matches the security group of ElastiCache. For the default VPC, we have enabled All Traffic for the inbound and outbound rules, but still, we are facing the same bug.
In order to connect to ElastiCache from NodeJS server I am using the following code:
const Redis = require("ioredis");
exports.connect = () => {
const client = new Redis.Cluster(
["xxxxx.xxxxx.clustercfg.use1.cache.amazonaws.com:6379"],
{
slotsRefreshTimeout: 10000,
dnsLookup: (address, callback) => callback(null, address),
redisOptions: {
showFriendlyErrorStack: true,
tls: {
checkServerIdentity: (/*host, cert*/) => {
// skip certificate hostname validation
return undefined;
},
},
},
}
);
client.on("ready", () => {
console.log("Redis Client Ready");
});
client.on("connect", () => {
console.log("Redis Client Connected");
});
client.on("error", (error) => {
console.log("Redis Client Connection Error", error);
});
client.on("reconnecting", () => {
console.log("Redis Client Reconnecting");
});
client.on("end", () => {
console.log("Redis Client Connection ended");
});
return client;
};
ElastiCache Configuration
Default VPC Security Group with Inbound and Outbound rules
Elastic Beanstalk security group(Same as default)
Error information from Elastic Beanstalk
Versions:
Node.js running on 64bit Amazon Linux with platform version 4.15.1
NodeJS version: 12.18.3
ioredis version: 4.17.3
npm version: 6.14.6
express version: 4.17.1
UPDATE: I am able to access the ElastiCache from ElasticBeanstalk if I do ssh and use redis-cli, but unable to access it using ioredis on NodeJS which is running on ElasticBeanstalk.

I have a similar setup and eventually got it working, a few key points:
Elasticbeanstalk and Elasticache have to be in the same VPC
Elasticache's security group should have an inbound rule to allow traffic from Elasticbeanstalk
Here's a code to connect:
import { RedisPubSub } from 'graphql-redis-subscriptions';
import Redis from 'ioredis';
import config from '../../config/env';
const options = {
// AWS host will look like this: somecache-dev-ro.k6sjdj.ng.0001.use1.cache.amazonaws.com
host: config.redis.host || 'localhost',
port: config.redis.port || 6379,
retryStrategy: (times: number): number => {
// reconnect after
return Math.min(times * 50, 2000);
},
};
export const pubsub = new RedisPubSub({
publisher: new Redis(options),
subscriber: new Redis(options),
});

I was debugging a similar issue. To access redis, I had to add tls: {} to the ioredis options:
{
host: process.env.REDIS_HOST,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASSWORD,
tls: {}
}

you can simply create connection
const Redis = require("ioredis");
const client = new Redis(
6379,
"Configiration Endpoint (xxx.xxxx.xxxcache.amazonaws.com)"
);
client.on("ready", () => {
console.log("Redis Client Ready");
client.send(
});
client.on("connect", () => {
console.log("Redis Client Connected");
});
client.on("error", (error) => {
console.log("Redis Client Connection Error", error);
});

Related

AWS Redis Cluster MOVED Error using redis node library

I have created a Redis MemoryDB cluster with 2 nodes in AWS:
I connect to it using redis node library v4.0.0 like this:
import { createCluster } from 'redis';
(async () => {
const REDIS_USERNAME = 'test-username';
const REDIS_PASSWORD = 'test-pass';
const cluster = createCluster({
rootNodes: [
{
url: `rediss://node1.amazonaws.com:6379`,
},
{
url: `rediss://node2.amazonaws.com:6379`,
},
],
defaults: {
url: `rediss://cluster.amazonaws.com:6379`,
username: REDIS_USERNAME,
password: REDIS_PASSWORD,
}
});
cluster.on('error', (err) => console.log('Redis Cluster Error', err));
await cluster.connect();
console.log('connected to cluster...');
await cluster.set('key', 'value');
const value = await cluster.get('key');
console.log('Value', value);
await cluster.disconnect();
})();
But sometimes I get the error ReplyError: MOVED 12539 rediss://node2.amazonaws.com:6379 and I cannot get the value from the key.
Do you have any idea if there is something wrong with the configuration of the cluster or with the code using redis node library?
Edit:
I tried it with ioredis library and it works, so it's something wrong with the redis library.
Node.js Version: 16
Redis Server Version: 6
I had created an issue to redis library, so it's going to be solved soon with this PR.

How to add proxy to connect with Azure Postgresql?

I use this script to connect node.js with Azure Postgresql.
But the ssl verification of our firewall blocks the connection, so in the past I need to use a proxy. Where in the code can I add the proxy settings as like host and port?
Means when I start the code, vscode should connect through the proxy to postgresql.
const pg = require('pg');
const config = {
host: '<your-db-server-name>.postgres.database.azure.com',
// Do not hard code your username and password.
// Consider using Node environment variables.
user: '<your-db-username>',
password: '<your-password>',
database: '<name-of-database>',
port: 5432,
ssl: true
};
const client = new pg.Client(config);
client.connect(err => {
if (err) throw err;
else { queryDatabase(); }
});
function queryDatabase() {
console.log(`Running query to PostgreSQL server: ${config.host}`);
const query = 'SELECT * FROM inventory;';
client.query(query)
.then(res => {
const rows = res.rows;
rows.map(row => {
console.log(`Read: ${JSON.stringify(row)}`);
});
process.exit();
})
.catch(err => {
console.log(err);
});
}
To configure proxy for Visual Studio Code
Edit the settings.json file
Depending on your platform, the user settings file is located here:
Windows: %APPDATA%\Code\User\settings.json
macOS: $HOME/Library/Application Support/Code/User/settings.json
Linux: $HOME/.config/Code/User/settings.json
Modify and Add the below lines to configure your proxy
"http.proxy": "http://user:pass#proxy.com:portnumber",
"https.proxy": "http://user:pass#proxy.com:portnumber",
"http.proxyStrictSSL": false
If your proxy doesn't require authentication, you could simply use
"http.proxy": "http://proxy.com:portnumber",
"https.proxy": "http://proxy.com:portnumber"
"http.proxyStrictSSL": false
Restart VS Code
The documentation related to settings and schema of the settings.json file is here for reference

Connection to postgresql db from node js

I'm tyring to make a connection from my nodejs script to my db connection, but seems like there is a suspicius issue i'm not able to figure out.
At the moment, this is my code:
const { Pool } = require('pg');
const pool = new Pool({
user: 'user',
host: '192.168.1.xxx',
database: 'database',
password: 'password',
port: 5432,
});
pool.on('error', (err, client) => {
console.error('Error:', err);
});
const query = `SELECT * FROM users`;
pool.connect()
.then((client) => {
client.query(query)
.then(res => {
for (let row of res.rows) {
console.log(row);
}
})
.catch(err => {
console.error(err);
});
})
.catch(err => {
console.error(err);
});
The issue seems to be in pool.connect(), but i can't understand what i'm missing because i got no errors in the log. I've installed pg module in the directory of my project with npm install --prefix pg and i know modules are loaded correctly.
I edited postgresql.conf:
# - Connection Settings -
listen_addresses = '*'
and pg_hba.conf
host database user 192.168.1.0/24 md5
to make the database reachable via lan and seems liek it works, because i'm able to connect successfully with apps like DBeaver...but i can't with NodeJS.
It's possible there is some kind of configuration i've to active?

How to handle error of closed MySQL connection in Node.js?

I've built a Node.js API that connects to a MySQL database. I use node-mysql2 as driver. The API and the database run in separate Docker container. At some point after deployment in a Kubernetes cluster I get the following error:
Error: Can't add new command when connection is in closed state
at PromiseConnection.query (/usr/src/app/node_modules/mysql2/promise.js:92:22)
I wonder why this error happens and how to catch and handle it using Node.js. These are code snippets of my Node.js API:
const mysql = require('mysql2/promise')
...
async function main() {
try {
const client = await mysql.createConnection({
host: DATABASE_HOST,
port: DATABASE_PORT,
user: DATABASE_USERNAME,
password: DATABASE_PASSWORD,
database: DATABASE_NAME
})
client.on('error', error => {
process.stderr.write(`${error.code}\n`) // PROTOCOL_CONNECTION_LOST
process.stderr.write(`An error occurred while connecting to the db: ${error.message}\n`)
process.exit(1)
})
} catch (error) {
process.stderr.write(`Error while creating db connection: ${error.code}\n`)
}
...
}
...
main().catch(err => {
process.stderr.write(`Error message: ${err.message}\n`)
process.exit(1)
})
Do you have an idea how to handle this error?
Do you close the connection after finishing with it?
client.end();
Also considered using a pool?
const pool = mysql.createPool({
host: DATABASE_HOST,
user: DATABASE_USERNAME,
database: DATABASE_NAME,
waitForConnections: true,
connectionLimit: 10,
queueLimit: 0
});
More info about pools: https://github.com/sidorares/node-mysql2#using-connection-pools

Unable to use tedious with Azure database

I've a node application which do have a connection to SQL Server.
Also, I'm using database as a service from Azure.
Code Snippet :
import { Connection } from 'tedious';
import { Request } from 'tedious';
var config = {
userName: 'dbuser',
password: 'dbpassword',
server: 'mydatabase.database.windows.net',
options: {
instanceName: 'SQLEXPRESS', // Removed this line while deploying it on server as it has no instance.
database: 'dbname'
}
};
connection = new Connection(config);
connection.on('connect', function(err) {
if (err) {
console.log('error : '+err);
} else {
console.log("Connected to Database");
}
});
It has a successful connection, if done, locally.
Console Output => Connected to Database.
Deep dive done using console log :
-> Connection object is being created, but, the event ".on" is not being able to establish.
-> The connection gets established when deployed locally, while, when deployed on server, it doesn't works.
Based on the documentation here, you need to provide an additional option for encrypted connection.
Please try the following for config:
var config = {
userName: 'dbuser',
password: 'dbpassword',
server: 'mydatabase.database.windows.net',
options: {
database: 'dbname',
encrypt: true //Need to add this for connecting to Azure DB
}
};
Using this configuration, I was able to connect to my database hosted in Azure.

Resources