How to connect to AWS ElasticSearch using npm elasticsearch and http-aws-es? - node.js

I am using the npm elasticsearch package to search my AWS ES domain. Everything seems to work fine when I use Postman to make POST requests with my AWS IAM credentials.
I wanted to do the same in my code (node.js). I referred to this answer here:
How to make calls to elasticsearch apis through NodeJS?
Here is code:
const elasticsearch = require('elasticsearch');
const awsHttpClient = require('http-aws-es');
const AWS = require('aws-sdk');
const client = new elasticsearch.Client({
host: 'my-aws-es-endpoint',
connectionClass: awsHttpClient,
amazonES: {
region: 'us-east-1',
credentials: new AWS.Credentials('my-access-key','my-secret-key')
}
});
But when I run client.search(), it fails with an error:
Elasticsearch ERROR: 2018-10-31T15:12:22Z
Error: Request error, retrying
POST https://my-endpoint.us-east-1.es.amazonaws.com/my-index/student/_search => Data must be a string or a buffer
It also gives me a warning
Elasticsearch WARNING: 2018-10-31T15:12:22Z
Unable to revive connection: https://my-endpoint.us-east-1.es.amazonaws.com/
When I use just the aws-sdk, it works fine (probably because I sign the request there?)
Can someone suggest what I am I doing wrong here?

I was able to solve this by specifying the region. There is a problem with the elasticsearch client where it's not able to the pick the region which we specify in
amazonES: {
region: 'us-east-1',
credentials: new AWS.Credentials('my-access-key','my-secret-key')
}
}
I solved it by specifying the region using AWS.config.region before the above code
AWS.config.region = 'us-east-1';

Related

Redis client via Amazon Elasticache is ignoring host parameter from function and using default IP address

I have a Redis cluster set up that I need to connect to from a Lambda function. I'm using the following code to create the client and connect to Redis:
const getClient = async (clientargs) => {
var params = {
no_ready_check: true,
connect_timeout: clientargs.timeout == null ? 30000 : clientargs.timeout,
enable_offline_queue: false,
db: clientargs.index,
retry_strategy: undefined
};
logger.debug('db getClient: Connecting to redis client.');
logger.debug('db getClient params', params);
const client = redis.createClient(clientargs.host, clientargs.port, params);
await client.connect();
// do stuff once it's connected
}
clientargs.host stores the Elasticache Redis endpoint and I'm using the default port 6379. I've checked that the host and port are being passed in correctly to the getClient function, and that they are being passed into redis.createClient properly as well. I was originally debugging in a local environment and noticed that it was trying to connect to 127.0.0.1 instead of the Elasticache endpoint, so I ran a few tests within the AWS Console but it's still trying on 127.0.0.1:6379.
I have also tried passing the host and port arguments into the client.connect() function, but had no luck with that either.
The Lambda function and the Elasticache Redis cluster are in the same VPC instance and security group, so I don't think that is the issue, but I am fairly new to AWS as a whole so any help/thoughts would be appreciated!

Connect Testcafe to AWS Devicefarm

We recently decided to include E2E to our front-end pipeline and we are using testCafe for that and since we use AWS as our SaaS we're being asked to use Devicefarm for remote testing and I'm facing the problem to connect them. I'm based on the selenium implementations and the testCafe documentation but it seems not to be able to establish connection between them, anyone has any idea why?
My code:
const AWS = require('aws-sdk');
const createTestCafe = require('testcafe');
const PROJECT_ARN = "arn:aws:devicefarm:us-west-2:XXXXXXXX:testgrid-project:XXXXXX-XXXX-XXXX-XXXX-XXXXXXXX";
const devicefarm = new AWS.DeviceFarm({ region: "us-west-2", credentials: AWS.config.credentials });
(async () => {
const testGridUrlResult = await devicefarm.createTestGridUrl({
projectArn: PROJECT_ARN,
expiresInSeconds: 6000
}).promise();
const url = new URL(testGridUrlResult.url || '');
const testCafe = await createTestCafe(url.host, 443);
const runner = testCafe.createRunner();
const remoteConnection = await testCafe.createBrowserConnection();
remoteConnection.once('ready', async () => {
await runner
.src(['./load-mfc.ts'])
.browsers('devicefarm:firefox')
.run();
await testCafe.close();
});
})().catch((e) => console.error(e));
According to the AWS Device Farm documentation about the CreateTestGridUrl method:
Creates a signed, short-term URL that can be passed to a Selenium RemoteWebDriver constructor.
This URL can't be passed to the hostname property of the createTestCafe method because TestCafe doesn't implement the WebDriver protocol. TestCafe works differently then Selenium. Once you created a remote browser connection using TestCafe, you should navigate your remote browser to the browserConnection.url URL to connect to a TestCafe server instance and start test execution in this browser.
As the AWS Device Farm service uses the WebDriver protocol to operate remote browsers, you would need to write a custom Browser Provider Plugin for TestCafe. It may be helpful to read more in the TestCafe documentation about this topic and see how a similar thing is implemented in the BrowserStack provider.

error passing empty credentials to firestore emulator

I am trying to seed some sample data into my local firestore emulator database. I adapted the example from this github issue
My code looks like this:
const {Firestore} = require('#google-cloud/firestore');
const {credentials} = require('grpc');
const db = new Firestore({
projectId: 'my-project-id',
servicePath: 'localhost',
port: 8100,
sslCreds: credentials.createInsecure(),
customHeaders: {
"Authorization": "Bearer owner"
}
});
async function load_data() {
await db.collection("mycollection").doc("myid").set({ foo: "test" })
}
load_data();
But I receive the error
this.credentials._getCallCredentials is not a function
Tested on node 10 and 12 with same error.
Library versions:
#google-cloud/firestore 3.5.1
grpc 1.24.2
Is there a better approach to writing to local emulated firestore? Or is there something wrong with my code?
The problem here is that you're trying to use two different implementations of gRPC together. Internally firestore uses #grpc/grpc-js, so that is what you should be using. You should only need to change the second line to const {credentials} = require('#grpc/grpc-js'); and switch the dependency to that library.

AWS lambda with mongoose to Atlas - MongoNetworkError

I am trying to connect MongoDB Atlas with mongoose and aws lambda but i get error MongoNetworkError
AWS Lambda
Mongoose
MongoDB Atlas
The same code was tested with serverless-offline and works perfect, the problem is when i deploy it to AWS Lambda.
This is the code snipet
'use strict';
const mongoose = require('mongoose');
const MongoClient = require('mongodb').MongoClient;
let dbuser = process.env.DB_USER;
let dbpass = process.env.DB_PASSWORD;
let opts = {
bufferCommands: false,
bufferMaxEntries: 0,
socketTimeoutMS: 2000000,
keepAlive: true,
reconnectTries: 30,
reconnectInterval: 500,
poolSize: 10,
ssl: true,
};
const uri = `mongodb+srv://${dbuser}:${dbpass}#carpoolingcluster0-bw91o.mongodb.net/awsmongotest?retryWrites=true&w=majority`;
// simple hello test
module.exports.hello = async (event, context, callback) => {
const response = {
body: JSON.stringify({message:'AWS Testing :: '+ `${dbuser} and ${dbpass}`}),
};
return response;
};
// connect using mongoose
module.exports.cn1 = async (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
let conn = await mongoose.createConnection(uri, opts);
const M = conn.models.Test || conn.model('Test', new mongoose.Schema({ name: String }));
const doc = await M.find();
const response = {
body: JSON.stringify({data:doc}),
};
return response;
};
// connect using mongodb
module.exports.cn2 = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
console.log("Connec to mongo using connectmongo ");
MongoClient.connect(uri).then(client => {
console.log("Success connect to mongo DB::::");
client.db('awsmongotest').collection('tests').find({}).toArray()
.then((result)=>{
let response = {
body: JSON.stringify({data:result}),
}
callback(null, response)
})
}).catch(err => {
console.log('=> an error occurred: ', err);
callback(err);
});
};
In the CloudWatch logs i see this error
{
"errorType": "MongoNetworkError",
"errorMessage": "failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
"stack": [
"MongoNetworkError: failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
" at Pool.<anonymous> (/var/task/node_modules/mongodb-core/lib/topologies/server.js:431:11)",
" at Pool.emit (events.js:189:13)",
" at connect (/var/task/node_modules/mongodb-core/lib/connection/pool.js:557:14)",
" at callback (/var/task/node_modules/mongodb-core/lib/connection/connect.js:109:5)",
" at runCommand (/var/task/node_modules/mongodb-core/lib/connection/connect.js:129:7)",
" at Connection.errorHandler (/var/task/node_modules/mongodb-core/lib/connection/connect.js:321:5)",
" at Object.onceWrapper (events.js:277:13)",
" at Connection.emit (events.js:189:13)",
" at TLSSocket.<anonymous> (/var/task/node_modules/mongodb-core/lib/connection/connection.js:350:12)",
" at Object.onceWrapper (events.js:277:13)",
" at TLSSocket.emit (events.js:189:13)",
" at _handle.close (net.js:597:12)",
" at TCP.done (_tls_wrap.js:388:7)"
],
"name": "MongoNetworkError",
"errorLabels": [
"TransientTransactionError"
]
}
Here is example on github to reproduce the error.
https://github.com/rollrodrig/error-aws-mongo-atlas
Just clone it, npm install, add your mongo atlas user, password and push to AWS.
Thanks.
Some extra steps are required to let lambda call external endpoint
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
Your atlas should also whitelist IP address of the servers, from which lambda will be connected.
Another option to consider - VPC peering between your lambda VPC and Atlas.
I have some questions concerning your configuration:
Did you whitelist the AWS Lambda function's IP address in Atlas?
Several posts on SO indicate that users get a MongoNetworkError like this if the IP is not whitelisted. [1][4]
Did you read the best-practices guide by Atlas which states that mongodb connections should be initiated outside the lambda handler? [2][3]
Do you use a public lambda function or a lambda function inside a VPC? There is a substantial difference between them and the latter one is more error-prone since the VPC configuration (e.g. NAT) must be taken into account.
I was able to ping the instances in the Atlas cluster and was able to establish a connection on port 27017. However, when connecting via the mongo shell, I get the following error:
Unable to reach primary for set CarpoolingCluster0-shard-0.
Cannot reach any nodes for set CarpoolingCluster0-shard-0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
When I use your GitHub sample from AWS lambda I get the exact same error message as described in the question.
As the error messages are not authentication-related but network-related, I assume that something is blocking the connection... Please double-check the three config questions above.
[1] What is a TransientTransactionError in Mongoose (or MongoDB)?
[2] https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/
[3] https://blog.cloudboost.io/i-wish-i-knew-how-to-use-mongodb-connection-in-aws-lambda-f91cd2694ae5
[4] https://github.com/Automattic/mongoose/issues/5237
Well, thanks everyone. Finally i found the solution with the help of the mongo support.
Here is the solution for anyone who needs
When you create a Mongo Altas cluster they ask you add your local ip and it is automatically added to the WhiteList. You can see it in
Your cluster > Network Access > IP Whitelist there in the list you will see your IP. It mean that only people from YOUR network will be able to connect to your MongoAtlas. AWS Lambda is NOT in your network, soo aws lambda will never connect to your Mongo Atlas. That is why i get the error MongoNetworkError.
Fix
You need to add the AWS Lambda IP to the Mongo Atlas WhiteListIP
go to your Your cluster > Network Access > IP Whitelist
click in the button ADD IP ADDRESS
click on ALLOW ACCESS FROM ANYWHERE it will add the ip 0.0.0.0/0 to the list, click confirm
Test your call from AWS Lambda and i will work.
FINALLY !
What you did is tell to Mongo Atlas that ANYONE from ANYWHERE can connect to your mongo Atlas.
Of course this is not a good practice. What you need is add only the AWS Lambda IP, here is when VPC comes to scene.
Create a VPC is little complex and it have many steeps, there are good tutorials in the other comments.
But for sure this small guide tacle the MongoNetworkError

Node.js + Express + simpledb; "TypeError: Cannot read property 'Errors' of null" when trying to list domains

I'm trying to get a very simple test of Amazon SimpleDB running with Node.js/Express. This is the code I'm using (AWS key/secret sanitized, of course):
var express = require('express');
var simpledb = require('simpledb');
var app = express.createServer();
var sdb = new simpledb.SimpleDB(
{keyed:'MYKEY', secret:'MYSECRET'}, simpledb.debuglogger);
app.get('/', function(req, res) {
console.log("about to list domains...");
sdb.listDomains(function(error, result, meta) {
console.log("listing domains, I think?");
});
});
app.listen(8888);
This is the error I'm getting:
DEBUG: simpledb: 2012-04-06T01:34:24.856Z create {"keyid":"MYKEY","secret":"MYSECRET","secure":false,"consistent":true,"test":false,"maxtry":null,"expbase":null,"delaymin":null,"delayscale":null,"randomdelay":null} {"secure":false,"host":"sdb.amazonaws.com","path":"/","version":"2009-04-15","port":80}
about to list domains...
DEBUG: simpledb: 2012-04-06T01:34:29.253Z request 1333676069253 ListDomains {}
DEBUG: simpledb: 2012-04-06T01:34:29.387Z handle 1333676069253 ListDomains {"Action":"ListDomains","Version":"2009-04-15","SignatureMethod":"HmacSHA256","SignatureVersion":"2","Timestamp":"2012-04-06T01:34:29.253Z","AWSAccessKeyId":"MYKEY","Signature":"AWSSIGNATURE"} 1 false null
/home/rob/node_modules/simpledb/lib/simpledb.js:136
if( res.Errors ) {
^
TypeError: Cannot read property 'Errors' of null
at [object Object].handle (/home/rob/node_modules/simpledb/lib/simpledb.js:136:12)
at /home/rob/node_modules/simpledb/lib/simpledb.js:188:18
at Parser.<anonymous> (/home/rob/node_modules/simpledb/node_modules/aws-lib/lib/aws.js:81:13)
at Parser.emit (events.js:67:17)
at Object.onclosetag (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/xml2js/lib/xml2js.js:120:24)
at emit (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/sax/lib/sax.js:148:32)
at emitNode (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/sax/lib/sax.js:152:3)
at closeTag (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/sax/lib/sax.js:226:5)
at Object.write (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/sax/lib/sax.js:567:29)
at Parser.<anonymous> (/home/rob/node_modules/simpledb/node_modules/aws-lib/node_modules/xml2js/lib/xml2js.js:145:29)
I'm pretty new to Node.js, the simpledb module and SimpleDB itself, but this to me seems like a bug in the simpledb module. I can't figure out what I'm doing wrong otherwise - I'm confident my key/secret are valid (as I've tested with both set invalid, separately and together, and I get back actual errors from Amazon that indicate the key/secret are invalid).
This error, though, has me stumped. Any ideas?
This turned out to be a dependency issue in the simpledb node module:
Hi - this seems to have been caused by the latest version of a
dependent lib - I've rebuilt and publish a new version 0.0.8 - please
let me know if this works - thanks!
Source. It has since been fixed.
FYI - Since your original post, AWS has published their official SDK for Node.js. http://docs.aws.amazon.com/nodejs/latest/dg/nodejs-dg-examples.html
var AWS = require('aws-sdk');
AWS.config.update({region: 'us-east-1'});
var db = new AWS.SimpleDB();
db.client.listDomains(function(error, data) {
if (error) {
console.log(error);
} else {
console.log(data);
}
});

Resources