AWS Lambda and Redis - node.js

I'm trying to write an AWS Lambda function which uses redis(on amazon elasticcache). The problem – I can't connect to redis. I use code like this
'use strict'
function handler (data, context, cb) {
const redis = require("redis")
console.log('before client initialization')
const client = redis.createClient({
url: 'redis://propper-url-cache.some.0001.euw1.cache.amazonaws.com:6379',
retry_strategy: function(options) {
console.log(options)
if (options.total_retry_time > 1000) {
throw new Error('can`t connect to redis')
}
}
})
console.log('after client initialization')
client.on("error", function (err) {
console.log('in error')
cb({error: err})
});
client.get("counter", function (err, counter) {
console.log('counter', counter)
if(_.isNull(counter)) {
counter = 0
}
client.set('counter', counter + 1, function(err) {
console.log(err)
cb(null, {counter: counter})
})
});
}
exports.handler = handler
as a result I see something like this in logs:

15:33:41
START RequestId: d8024ec2-7f36-11e6-996c-1bfcb60572c6 Version: $LATEST

15:33:42
2016-09-20T13:33:42.632Z d8024ec2-7f36-11e6-996c-1bfcb60572c6 before client initialization

15:33:42
2016-09-20T13:33:42.813Z d8024ec2-7f36-11e6-996c-1bfcb60572c6 after client initialization

15:33:44
END RequestId: d8024ec2-7f36-11e6-996c-1bfcb60572c6

15:33:44
REPORT RequestId: d8024ec2-7f36-11e6-996c-1bfcb60572c6 Duration: 3002.67 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 19 MB

15:33:44
2016-09-20T13:33:44.620Z d8024ec2-7f36-11e6-996c-1bfcb60572c6 Task timed out after 3.00 seconds
when I change redis url for something which definitely makes no sense I have an additional row:
2016-09-20T13:29:42.953Z 48fcb071-7f36-11e6-bc52-c5ac58c12843 { attempt: 1, error: { [Error: Redis connection to some-url.euw1.cache.amazonaws.com:6379 failed - getaddrinfo ENOTFOUND some-url.euw1.cache.amazonaws.com some-url.euw1.cache.amazonaws.com:6379] code: 'ENOTFOUND', errno: 'ENOTFOUND', syscall: 'getaddrinfo', hostna
Any ideas?

You need to have Redis in same VPC as Lambda. Check your security group settings. And then if you have EC2 access, install redis-cli and try to connect Redis. If this get connected your Lambda Redis will also get connected. As said earlier, you need to have your lambda in same VPC.
The following is boilerplate code demonstrating connecting to Lambda:
console.log('before client initialization')
const redisOptions = {
host: 'xxxx.xxx.xxx.xxx.xxx.amazonaws.com',
port: 6379,
}
var client = redis.createClient(redisOptions);
console.log('after client initialization');
client.on('connect', function(result) {
console.log('connected');
}

Related

Lambda timeout after 10s both in serverless offline and online

I am new to lambda, mongodb and nodejs and I'm trying to access a mongodb database follwoing this tutorial: [pretty nice tutorial][1] . This is my second attempt at connecting to mongodb and after spending hours setting things up I run into an unsurmountable wall and none of all the answers on stack overflow have helped. Any help will be greatly appreciated.
I have inserted console.log instructions and they both are executed so the timout happens after connecting to the database, but after that I don't know what is executed. I have also set up VPC properly in AWS Lambda so I have no idea why this is timing out. ( See edit below) It might be worth mentioning that I have never been able to get past this stage: of all the things I have tried there is always a timeout error.
I have a handler.js file which, among other functions, has this one:
'use strict';
require('dotenv').config({ path: './variables.env' });
const connectToDatabase = require('./db');
const Note = require('./models/Note');
module.exports.create = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
console.log("here1");
connectToDatabase()
.then(() => {
Note.create(JSON.parse(event.body))
.then(note => callback(null, {
statusCode: 200,
body: JSON.stringify(note)
}))
.catch(err => callback(null, {
statusCode: err.statusCode || 500,
headers: { 'Content-Type': 'text/plain' },
body: 'Could not create the note.'
}));
});
console.log("here2");
};
I also have a db.js file and a Note.js file as instructed in the tutorial ( here posted respectively )
//file db.js
const mongoose = require('mongoose');
mongoose.Promise = global.Promise;
let isConnected;
module.exports = connectToDatabase = () => {
if (isConnected) {
console.log('=> using existing database connection');
return Promise.resolve();
}
console.log('=> using new database connection');
return mongoose.connect(process.env.DB)
.then(db => {
isConnected = db.connections[0].readyState;
});
};
//file Note.js
const mongoose = require('mongoose');
const NoteSchema = new mongoose.Schema({
title: String,
description: String
});
module.exports = mongoose.model('Note', NoteSchema);
Both serverless offline and serverless online give me a timeout error:
Serverless offline:
here1
=> using new database connection
here2
✖ Lambda timeout.
Serverless online:
START RequestId: f75c6e23-84e2-485c-8fb4-065a95bbbb9b Version: $LATEST
2022-10-25T11:53:44.171Z f75c6e23-84e2-485c-8fb4-065a95bbbb9b INFO here1
2022-10-25T11:53:44.171Z f75c6e23-84e2-485c-8fb4-065a95bbbb9b INFO => using new database connection
2022-10-25T11:53:44.311Z f75c6e23-84e2-485c-8fb4-065a95bbbb9b INFO here2
END RequestId: f75c6e23-84e2-485c-8fb4-065a95bbbb9b
REPORT RequestId: f75c6e23-84e2-485c-8fb4-065a95bbbb9b Duration: 10013.48 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 100 MB Init Duration: 864.37 ms
2022-10-25T11:53:54.174Z f75c6e23-84e2-485c-8fb4-065a95bbbb9b Task timed out after 10.01 seconds
I am sending this JSON POST request through Insomnia:
{
"title":"some title",
"description": "some description"
}
After changing the timeout time to 5 mins, I am getting an error related to the connection to mongo saying I should whitelist my IP. It is whitelisted but I will investigate.
[1]: https://medium.com/hackernoon/building-a-serverless-rest-api-with-node-js-and-mongodb-2e0ed0638f47

Making POST call outside of the GCP environment to create workload identity

I need to access GCP resources outside of the GCP environment from AWS using a AWS lambda. So, I found this document [accessing GCP resources from AWS][1] which provides a way to access the GCP resources and asks to create a workload identity pool.
I need to create a Workload identity pool in GCP using a REST API call. The REST API call has to run outside of the GCP environment, that is, in this case from the AWS environment. My GCP's IAM user doesn't have privileges to create a workload identity pool (due to org policy reasons). But, I've a service account which has admin privileges to create a workload identity pool and all the required permissions to access the required resources once the pool is created.
I'm a newbie to GCP and figuring out ways of calling a POST REST API call using my service account credentials. Any help is much appreciated.
Edited
Pasting the sample code I've been trying to make the REST call.
const {google} = require('googleapis');
const util = require('util');
const https = require('https');
const aws4 = require('aws4');
const auth = new google.auth.GoogleAuth({
keyFile: 'serviceAccountCreds.json',
scopes: ['https://www.googleapis.com/auth/cloud-platform'],
});
async function createSignedRequestParams(jsonBodyParams) {
const getAccessToken = await auth.getAccessToken();
console.log(`createSignedRequestParams() - this.credentials:${getAccessToken !== null}`);
// Set up the request params object that we'll sign
const requestParams = {
path: '/v1beta/projects/serviceAccountdev/locations/global/workloadIdentityPools?workloadIdentityPoolId=12345',
method: 'POST',
host: 'iam.googleapis.com',
headers: { 'Content-Type': 'application/json' },
body: jsonBodyParams
};
console.log(`createSignedRequestParams() - (signed) requestParams:${util.inspect(requestParams)}`);
return requestParams;
}
const jsonBodyParams = {
"description": "createWorkloadIdentityPool",
"display-name": "devAccount"
};
async function request(requestParams, jsonBodyParams) {
console.log(`request() requestParams:${util.inspect(requestParams)} jsonBodyParams:${jsonBodyParams}`);
// return new pending promise
return new Promise((resolve, reject) => {
const req = https.request(requestParams);
if (['POST', 'PATCH', 'PUT'].includes(requestParams.method)) {
req.write(jsonBodyParams);
}
req.end();
// Stream handlers for the request
req.on('error', (err) => {
console.log(`request() req.on('error') err:${util.inspect(err)}`);
return reject(err);
});
req.on('response', (res) => {
let dataJson = '';
res.on('data', chunk => {
dataJson += chunk;
});
res.on('end', () => {
const statusCode = res.statusCode;
const statusMessage = res.statusMessage;
const data = JSON.parse(dataJson);
console.log(`request() res.on('end')`, { statusCode, statusMessage, data });
resolve({ statusMessage, statusCode, data });
});
});
});
}
async function postTheRequest(reqParams, jsonBodyParams) {
try {
const response = await request(reqParams, jsonBodyParams);
return response;
} catch (error) {
console.log(error);
}
}
reqParams = createSignedRequestParams(jsonBodyParams);
postTheRequest(reqParams, jsonBodyParams);
output of the above code
[Running] node "c:\Users\av250044\.aws\GCP_Code_examples\registerTheWorkloadIdentifier.js"
request() requestParams:Promise { <pending> } jsonBodyParams:[object Object]
request() req.on('error') err:{ Error: connect ECONNREFUSED 127.0.0.1:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 443 }
{ Error: connect ECONNREFUSED 127.0.0.1:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 443 }
Wondering if I'm passing the PATH and host are correct. Please let me know your thoughts on my code sample.
[1]: https://cloud.google.com/iam/docs/access-resources-aws#iam-workload-pools-add-aws-rest

Mongoose Unexpected token u in JSON at position 0

I'm creating an AWS Lambda API with Node JS. There's a template for the db connection itself, but when I run a test in Lambda I get an error and I believe the problem is the connection itself.
Here's the error message I get with Lambda test:
START RequestId: 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 Version: $LATEST
2020-05-23T01:46:23.685Z 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 INFO => using new database connection
2020-05-23T01:46:23.689Z 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 ERROR (node:8) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
2020-05-23T01:46:23.726Z 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 ERROR (node:8) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
2020-05-23T01:46:23.828Z 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 ERROR Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"SyntaxError: Unexpected token u in JSON at position 0","reason":{"errorType":"SyntaxError","errorMessage":"Unexpected token u in JSON at position 0","stack":["SyntaxError: Unexpected token u in JSON at position 0"," at JSON.parse ()"," at /var/task/handler.js:17:22"," at processTicksAndRejections (internal/process/task_queues.js:97:5)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: SyntaxError: Unexpected token u in JSON at position 0"," at process. (/var/runtime/index.js:35:15)"," at process.emit (events.js:310:20)"," at processPromiseRejections (internal/process/promises.js:209:33)"," at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}
END RequestId: 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5
REPORT RequestId: 6b9eef97-9c54-44bc-8ecc-dfbec7e200e5 Duration: 186.61 ms Billed Duration: 200 ms Memory Size: 1024 MB Max Memory Used: 34 MB
Unknown application error occurred
DB.js
const mongoose = require("mongoose");
mongoose.Promise = global.Promise;
let isConnected;
module.exports = connectToDatabase = () => {
if (isConnected) {
console.log("=> using existing database connection");
return Promise.resolve();
}
console.log("=> using new database connection");
return mongoose.connect(process.env.DB).then((db) => {
isConnected = db.connections[0].readyState;
});
};
Handler.js
"use strict";
const connectToDatabase = require("./db");
const Lead = require("./leads.model.js");
require("dotenv").config({ path: "./variables.env" });
module.exports.hello = (event, context, callback) => {
console.log("Hello World");
callback(null, "Hello World");
};
module.exports.create = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase().then(() => {
Lead.create(JSON.parse(event.body))
.then((lead) =>
callback(null, {
statusCode: 200,
body: JSON.stringify(lead),
})
)
.catch((err) =>
callback(null, {
statusCode: err.statusCode || 500,
headers: { "Content-Type": "text/plain" },
body: "Could not create the lead.",
})
);
});
};
variables.env
DB=mongodb+srv://dbUser:password#api-jodqc.mongodb.net/test?retryWrites=true&w=majority
Whenever I see Unexpected token u in JSON at position 0, it usually means JSON.parse is trying to parse undefined. Looking at the code, I only see you executing parse in one spot
Lead.create(JSON.parse(event.body))
I'm guessing when you execute the Lambda, you're passing in an object that looks like a lead directly, so maybe something like below.
{
isALead: true
}
Instead, you need to test with an object that looks like
{
body: "{ \"isALead\": true }"
}

getting Error: getaddrinfo ENOTFOUND while performing rest api call in node.js using http.request

i have created api in node.js which consume set of api hosted at http://dev.abc.co.in:20081
not every time but randomly sometimes it throws the error
Error: getaddrinfo ENOTFOUND dev.abc.co.in
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:60:26) {
errno: 'ENOTFOUND',
code: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: 'dev.abc.co.in'
}
to call those api i have used request node module because i started getting this error i switched to fetch-node npm module and finally replace the code with internal node module http but getting same error
here is the code i have written using http.request
try{
const options = {
hostname: "dev.abc.co.in",
port : 20081,
path: "/api/entity/workorder",
method: Config.method
};
if(Config.headers){
options.headers = Config.headers
}
const req = http.request(options, (res) => {
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('end', () => {
callback(res, data);
});
req.socket.destroy();
}).on("error", (err) => {
console.log("===Error: ", err);
callback(null, err);
});
if(Config.method!="GET" && Config.body){
Config.headers["Content-Length"] = Config.body.length;
req.write(Config.body);
}
req.end();
}catch(e){
console.log("Exception=====",e);
}
as shown in error message issue related to DNS so i try to resolve this DNS using
node -pe 'require("dns").lookup("dev-vsg.dovertech.co.in",function(){console.dir(arguments)})
but still not resolved.
1) Omit 'http://' from the beginning of your demain and all slashes from the end or any path after the actual domain.
2) Try to resolve your hostname:
const dns = require('dns');
dns.resolve("testdomain.com", 'ANY', (err, records) => {
if (err) {
console.log("Error: ", err);
} else {
console.log(records);
}
});
If dns records has been returned, then you will know it's a node js problem and after that we can investigate further. If not, then it's a domain configuration issue.

Error: Timed out while waiting for handshake in Lambda logs

The function of SFTP is working locally and it has the correct configuration.
The problem is with the logs of Lambda that complains about the configuration I believe by giving me an Error: Timed out while waiting for a handshake.
const config = {
host: 'ftp.xxxx.at',
port: '22',
username: 'SFTP_xxx',
password: 'xxx9fBcS45',
MD5_Fingerprint: 'xxx:8f:5b:1a',
protocol: "sftp",
algorithms: {
serverHostKey: ['ssh-rsa', 'ssh-dss']
}
};
// get file
// get sftp file
// here ....
const get = () => {
sftp.connect(config).then(() => {
return sftp.get('./EMRFID_201811210903.csv', null, 'utf8', null);
}).then((stream) => {
let body = stream.on('data', (chunk) => {
body += chunk;
})
stream.on('end', () => {
uploadRFIDsToS3(body)
// close connection
sftp.end()
});
}).catch((err) => {
console.log('catch err:', err)
})
};
-
vpc:
securityGroupIds:
- sg-01c29be1d8fbxx59
subnetdIds:
- subnet-007a88d9xxea434d
-
2019-02-18T13:53:51.121Z e688c7bd-24fc-45a1-a565-f2a4c313f846 catch err: { Error: Timed out while waiting for handshake
at Timeout._onTimeout (/var/task/node_modules/ssh2-sftp-client/node_modules/ssh2/lib/client.js:695:19)
at ontimeout (timers.js:482:11)
at tryOnTimeout (timers.js:317:5)
at Timer.listOnTimeout (timers.js:277:5) level: 'client-timeout' }
I added VPC and Security Group in AWS and I still get the same error.
I ran out of ideas of how to fix it.
Increase the time-out value for the lambda function. The option to allocate more memory and time-out are some of the settings for any lambda function in aws
I figure it out.
What was wrong is that Lambda was actually going to another function without getting the connection established.
So what I did is that I added await to the connection and other functions that should not interfere with each other to make to work.
To understand more about await go to this link:
https://javascript.info/async-await

Resources