I am attempting to connect to an ElastiCache cluster that is encrypted in transit from a node script using ioRedis. Sometimes my script works, other times I get Error: 140736319218624:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1216:
Here is all of my code:
var Redis = require('ioredis');
var nodes = [{
host: 'clustercfg.name.xxxxxx.region.cache.amazonaws.com',
port: '6379',
}];
var cluster = new Redis.Cluster(nodes,{
redisOptions: {
tls: {}
}});
cluster.set('aws', 'test');
cluster.get('aws', function (err, res) {
console.log(res);
if (err) {
console.error(err)
}
cluster.disconnect()
});
I believe the ssl handshake error is a side-effect of a race-condition bug in ioredis.
I have been banging my head over the same issue the last several days (ioredis version 4.0.0). I just couldn't reliably connect ioredis to our elasticache cluster. I would see the same intermittent error.
Error: 140618195700616:error:140940E5:SSL routines:ssl3_read_bytes:ssl
handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1216:
You can view ioredis debug output by setting "DEBUG=ioredis:*" in your node environment. Once I did this I could see that when the error occurred it was accompanied by several messages similar to the following:
2018-10-06T18:24:38.287Z ioredis:cluster:connectionPool Disconnect
xxx.usw2.cache.amazonaws.com:6379 because the node does not hold any
slot
I tried node-redis and redis-clustr it works fine with elasticache.
Related
I have a Redis cluster set up that I need to connect to from a Lambda function. I'm using the following code to create the client and connect to Redis:
const getClient = async (clientargs) => {
var params = {
no_ready_check: true,
connect_timeout: clientargs.timeout == null ? 30000 : clientargs.timeout,
enable_offline_queue: false,
db: clientargs.index,
retry_strategy: undefined
};
logger.debug('db getClient: Connecting to redis client.');
logger.debug('db getClient params', params);
const client = redis.createClient(clientargs.host, clientargs.port, params);
await client.connect();
// do stuff once it's connected
}
clientargs.host stores the Elasticache Redis endpoint and I'm using the default port 6379. I've checked that the host and port are being passed in correctly to the getClient function, and that they are being passed into redis.createClient properly as well. I was originally debugging in a local environment and noticed that it was trying to connect to 127.0.0.1 instead of the Elasticache endpoint, so I ran a few tests within the AWS Console but it's still trying on 127.0.0.1:6379.
I have also tried passing the host and port arguments into the client.connect() function, but had no luck with that either.
The Lambda function and the Elasticache Redis cluster are in the same VPC instance and security group, so I don't think that is the issue, but I am fairly new to AWS as a whole so any help/thoughts would be appreciated!
Am trying to connect redis sentinel instance from nodeJS using ioredis. Am not able to connect redis sentinel instance despite trying multiple available options. We have not configured sentinel password. But, able to connect same redis sentinel instance from .net core using StackExchange.Redis. Please find below nodeJS code,
import { AppModule } from './app.module';
import IORedis from 'ioredis';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const ioredis = new IORedis({
sentinels: [
{ host: 'sentinel-host-1' },
{ host: 'sentinel-host-2' },
{ host: 'sentinel-host-3' },
],
name: 'mastername',
password: 'password',
showFriendlyErrorStack: true,
});
try {
ioredis.set('foo', 'bar');
} catch (exception) {
console.log(exception);
}
await app.listen(3000);
}
bootstrap();
Error we got is,
[ioredis] Unhandled error event: Error: connect ETIMEDOUT
node_modules\ioredis\built\redis\index.js:317:37)
at Object.onceWrapper (node:events:475:28)
at Socket.emit (node:events:369:20)
at Socket._onTimeout (node:net:481:8)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
Connection String used from .net core is below,
Redis_Configuration = "host-1,host-2,host-3,serviceName=mastername,password=password,abortConnect=False,connectTimeout=1000,responseTimeout=1000";
Answering this for the benefit of others. Everything is fine, but this nodeJS package is resolving redis instances into private IPs which i cannot access from my local. So, had to put it over subnet group and make it work. However, FYI - .net core package does not resolve into private IPs, hence i was able to access instances from my local itself.
"The arguments passed to the constructor are different from the ones you use to connect to a single node"
Try to replace password with sentinelPassword.
I am trying to connect MongoDB Atlas with mongoose and aws lambda but i get error MongoNetworkError
AWS Lambda
Mongoose
MongoDB Atlas
The same code was tested with serverless-offline and works perfect, the problem is when i deploy it to AWS Lambda.
This is the code snipet
'use strict';
const mongoose = require('mongoose');
const MongoClient = require('mongodb').MongoClient;
let dbuser = process.env.DB_USER;
let dbpass = process.env.DB_PASSWORD;
let opts = {
bufferCommands: false,
bufferMaxEntries: 0,
socketTimeoutMS: 2000000,
keepAlive: true,
reconnectTries: 30,
reconnectInterval: 500,
poolSize: 10,
ssl: true,
};
const uri = `mongodb+srv://${dbuser}:${dbpass}#carpoolingcluster0-bw91o.mongodb.net/awsmongotest?retryWrites=true&w=majority`;
// simple hello test
module.exports.hello = async (event, context, callback) => {
const response = {
body: JSON.stringify({message:'AWS Testing :: '+ `${dbuser} and ${dbpass}`}),
};
return response;
};
// connect using mongoose
module.exports.cn1 = async (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
let conn = await mongoose.createConnection(uri, opts);
const M = conn.models.Test || conn.model('Test', new mongoose.Schema({ name: String }));
const doc = await M.find();
const response = {
body: JSON.stringify({data:doc}),
};
return response;
};
// connect using mongodb
module.exports.cn2 = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
console.log("Connec to mongo using connectmongo ");
MongoClient.connect(uri).then(client => {
console.log("Success connect to mongo DB::::");
client.db('awsmongotest').collection('tests').find({}).toArray()
.then((result)=>{
let response = {
body: JSON.stringify({data:result}),
}
callback(null, response)
})
}).catch(err => {
console.log('=> an error occurred: ', err);
callback(err);
});
};
In the CloudWatch logs i see this error
{
"errorType": "MongoNetworkError",
"errorMessage": "failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
"stack": [
"MongoNetworkError: failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
" at Pool.<anonymous> (/var/task/node_modules/mongodb-core/lib/topologies/server.js:431:11)",
" at Pool.emit (events.js:189:13)",
" at connect (/var/task/node_modules/mongodb-core/lib/connection/pool.js:557:14)",
" at callback (/var/task/node_modules/mongodb-core/lib/connection/connect.js:109:5)",
" at runCommand (/var/task/node_modules/mongodb-core/lib/connection/connect.js:129:7)",
" at Connection.errorHandler (/var/task/node_modules/mongodb-core/lib/connection/connect.js:321:5)",
" at Object.onceWrapper (events.js:277:13)",
" at Connection.emit (events.js:189:13)",
" at TLSSocket.<anonymous> (/var/task/node_modules/mongodb-core/lib/connection/connection.js:350:12)",
" at Object.onceWrapper (events.js:277:13)",
" at TLSSocket.emit (events.js:189:13)",
" at _handle.close (net.js:597:12)",
" at TCP.done (_tls_wrap.js:388:7)"
],
"name": "MongoNetworkError",
"errorLabels": [
"TransientTransactionError"
]
}
Here is example on github to reproduce the error.
https://github.com/rollrodrig/error-aws-mongo-atlas
Just clone it, npm install, add your mongo atlas user, password and push to AWS.
Thanks.
Some extra steps are required to let lambda call external endpoint
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
Your atlas should also whitelist IP address of the servers, from which lambda will be connected.
Another option to consider - VPC peering between your lambda VPC and Atlas.
I have some questions concerning your configuration:
Did you whitelist the AWS Lambda function's IP address in Atlas?
Several posts on SO indicate that users get a MongoNetworkError like this if the IP is not whitelisted. [1][4]
Did you read the best-practices guide by Atlas which states that mongodb connections should be initiated outside the lambda handler? [2][3]
Do you use a public lambda function or a lambda function inside a VPC? There is a substantial difference between them and the latter one is more error-prone since the VPC configuration (e.g. NAT) must be taken into account.
I was able to ping the instances in the Atlas cluster and was able to establish a connection on port 27017. However, when connecting via the mongo shell, I get the following error:
Unable to reach primary for set CarpoolingCluster0-shard-0.
Cannot reach any nodes for set CarpoolingCluster0-shard-0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
When I use your GitHub sample from AWS lambda I get the exact same error message as described in the question.
As the error messages are not authentication-related but network-related, I assume that something is blocking the connection... Please double-check the three config questions above.
[1] What is a TransientTransactionError in Mongoose (or MongoDB)?
[2] https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/
[3] https://blog.cloudboost.io/i-wish-i-knew-how-to-use-mongodb-connection-in-aws-lambda-f91cd2694ae5
[4] https://github.com/Automattic/mongoose/issues/5237
Well, thanks everyone. Finally i found the solution with the help of the mongo support.
Here is the solution for anyone who needs
When you create a Mongo Altas cluster they ask you add your local ip and it is automatically added to the WhiteList. You can see it in
Your cluster > Network Access > IP Whitelist there in the list you will see your IP. It mean that only people from YOUR network will be able to connect to your MongoAtlas. AWS Lambda is NOT in your network, soo aws lambda will never connect to your Mongo Atlas. That is why i get the error MongoNetworkError.
Fix
You need to add the AWS Lambda IP to the Mongo Atlas WhiteListIP
go to your Your cluster > Network Access > IP Whitelist
click in the button ADD IP ADDRESS
click on ALLOW ACCESS FROM ANYWHERE it will add the ip 0.0.0.0/0 to the list, click confirm
Test your call from AWS Lambda and i will work.
FINALLY !
What you did is tell to Mongo Atlas that ANYONE from ANYWHERE can connect to your mongo Atlas.
Of course this is not a good practice. What you need is add only the AWS Lambda IP, here is when VPC comes to scene.
Create a VPC is little complex and it have many steeps, there are good tutorials in the other comments.
But for sure this small guide tacle the MongoNetworkError
I'm having a socket.io app that basically receives signals from a frontend in order to kill and start a new ffmpeg process (based on .spawn()).
Everything works like expected, but often I get a 525 error from cloudflare. The error message is: Cloudflare is unable to establish an SSL connection to the origin server.
It works like 9 out of 10 times.I noticed that more of these errors pop up whenever a kill + spawn is done. Could it be the case that something block the event loop and because of this blocks all incoming requests and cloudflare logs these as a handshake failed error?
Contacting cloudflare support gives me back this info (this is the request they do to my server):
Time id host message upstream
2017-08-16T09:14:24.000Z 38f34880faf04433 xxxxxx.com:2096 peer closed connection in SSL handshake while SSL handshaking to upstream https://xxx.xxx.xxx.xxx:2096/socket.io/?EIO=3&transport=polling&t=LtgKens
I'm debugging for some time now, but can't seem to find a solutions myself.
This is how I initialize my socketIO server.
/**
* Start the socket server
*/
var startSocketIO = function() {
var ssl_options = {
key: fs.readFileSync(sslConfig.keyFile, 'utf8'),
cert: fs.readFileSync(sslConfig.certificateFile, 'utf8')
};
self.app = require('https').createServer(ssl_options, express);
self.io = require('socket.io')(self.app);
self.io.set('transports', ['websocket', 'polling']);
self.app.listen(2096, function() {
console.log('Socket.IO Started on port 2096');
});
};
This is the listener code on the server side
this.io.on('connection', function (socket) {
console.log('new connection');
/**
* Connection to the room
*/
socket.on('changeVideo', function (data) {
//Send to start.js and start.js will kill the ffmpeg process and
start a new one
socket.emit('changeVideo');
});
});
Another thing that I observer while debugging (I only got this a few times):
The text new connection displayed on the server and the connected client emits the changevideo event but nothing happens on the server side instead the client just
keeps reconnecting.
This is a simplified version of the nodejs code. If you have more questions, just let me know.
Thanks!
this is probably not a bug but rather a gap in my understanding but putting it here as afraid havent been able to find a way so far. Appreciate if you can provide your inputs please.
I'm trying to connect to my Riak cluster (hosted on AWS) of 3 nodes via two options - 1) Using an ejabberd server, and 2) using a Node server.
Connecting from the ejabberd server is successful after I put the hostname and port in the ejabberd configuration, but when I use a simple Node server (code below), I get the "Error: No RiakNodes available to execute command." error. Am I missing out on something here please - I can confirm that the 3 nodes are indeed up with Riak running? Note that if I dont do the client ping on the nodes, the server doesnt throw any error, so it is probably got to do with how pings are handled. The same server (without the ping) gives an ECONNREFUSED error if one of the nodes are brought down. So clearly the connection is going through but not the ping.
Apologize if am missing out on something basic here ... even the firewall settings for the Riak nodes have been set to all inbound, so it is not a case of the ejabberd server having access but not the Node server.
var async = require('async');
var assert = require('assert');
var logger = require('winston');
var Riak = require('basho-riak-client');
logger.remove(logger.transports.Console);
logger.add(logger.transports.Console, {
level : 'debug',
colorize : true,
timestamp : true
});
var nodes = [
'ip-xx-xx-xx-xx:8087',
'ip-xx-xx-xx-xx:8087',
'ip-xx-xx-xx-xx:8087'
];
var client = new Riak.Client(nodes, function (err, c) {
logger.info('Now inside Riak.Client');
// NB: at this point the client is fully initialized, and
// 'client' and 'c' are the same object
});
client.ping(function (err, rslt) {
logger.info('Now entered client.ping');
if (err) {
logger.info('There is an error encountered in client.ping');
throw new Error(err);
} else {
// On success, ping returns true
logger.info('client.ping has resulted in success!');
assert(rslt === true);
}
});