Share code between AWS lambda functions in node.js - node.js

It seems it is not possible to pass around some code (containing data and functions) that is invoked as a AWS lambda function within another AWS lambda function.
For example, take this customConfigLambda:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
exports.handler = (event, context, callback) => {
callback(null, callbackPayload);
};
When I call the previous AWS lambda function in another AWS lambda function like here:
var AWS = require('aws-sdk');
AWS.config.update({accessKey: '123', secretAccessKey: 'abc', region: 'us-east-1' });
var lambda = new AWS.Lambda({region: 'us-east-1'});
exports.handler = (event, context, callback) => {
var params = {FunctionName: 'customConfigLambda'};
lambda.invoke(params, function(err, callbackPayload) {
if (err) {
// do nothing
}
else {
console.log('callbackPayload:', JSON.stringify(callbackPayload, null, 2));
}
});
};
Then I can see only callbackPayload.fooData but not callbackPayload.fooFunction.
How can I have some callbackPayload.fooFunction(s) shared between multiple other AWS lambda functions?

As of AWS Reinvent 2018, Amazon has introduced Lambda Layers.
Lambda Layers, a way to centrally manage code and data that is shared across multiple functions.
The idea is that now you can put common components in a ZIP file and upload it as a Lambda Layer. Your function code doesn’t need to be changed and can reference the libraries in the layer as it would normally do instead of packaging them separately.

See the docs on Using the Callback Parameter at:
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html#nodejs-prog-model-handler-callback
It says this about the result (the callbackPayload in your code):
result – is an optional parameter that you can use to provide the
result of a successful function execution. The result provided must be
JSON.stringify compatible. If an error is provided, this parameter is
ignored.
To be JSON.stringify compatible you cannot have any functions there. See the http://json.org/ to see what is valid JSON (only strings, numbers, objects, arrays, true, false and null).
If you want to share code between your AWS Lambda functions in a broad sense, you have to require the same Node module in both of them, so that you can make a common set of functions available to all of your AWS Lamda handlers. But you cannot pass around arbitrary code between them because those will not survive the JSON.stringify.
As a test you can try running this in the browser:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
alert(JSON.stringify(callbackPayload));
(see DEMO)
or this in Node:
var callbackPayload = {};
callbackPayload.fooData = 'fooFromData';
callbackPayload.fooFunction = function() {return 'fooFromFunction'; };
console.log(JSON.stringify(callbackPayload));
and see the result:
{"fooData":"fooFromData"}
The functions is stripped out during the serialization process.
Of course you could do something like this:
callbackPayload.fooFunction
= function() {return 'fooFromFunction'; }.toString();
and get a JSON result:
{"fooData":"fooFromData","fooFunction":"function () {return 'fooFromFunction'; }"}
which you could theoretically eval on the other end but I wouldn't really recommend it.

Related

Very lazy speed of scanning in Lambda and DAX for DynamoDB

I'm getting a problem using Lambda and DAX.
In lambda, Without Nodejs DAX clien the scan average time is 900ms but If I use DAX, it's 4500ms. It's weird because if I use DAX, estimated time will be short than before.
This is latest code. In here I'm only getting one record but still it's same. )
const AWS = require('aws-sdk');
const AmazonDaxClient = require('amazon-dax-client');
const config = require('../config.json');
AWS.config.update({
region: config.region,
accessKeyId: config.accessKeyId,
secretAccessKey: config.secretAccessKey
});
var dax = null;
var daxClient=null;
const daxConfig = {
endpoints:[config.daxEndpoints],
region:config.region
}
if(dax == null & daxClient == null) {
console.log('initialized');
dax = new AmazonDaxClient(daxConfig);
daxClient = new AWS.DynamoDB.DocumentClient({service: dax });
}
exports.main = function(event, context, callback) {
context.callbackWaitsForEmptyEventLoop = false
const params = {
TableName: "game_dev"
};
daxClient.scan(params, function(err, data) {
if (err) {
console.log(JSON.stringify(err));
} else {
console.log("Query succeeded.");
}
});
}
AWS Lambda uses reusable containers. On first load it is an "initialized container". 2nd load on out till it refreshes (4hrs?) it reuses the initialized container. It may even scale out depending on the workload.
The trick you need to master in your code is to not reinitialize the DynamoDb client on each call. This AWS link goes into more detail on it: AWS Best Practices on AWS Service Client Initialization
So for your case, set up your client variables, dax and daxclient, outside the scope of the function handler. In C# I declare them outside the scope of the handler then initialize them in the handler if they are null. If not null I skip initialization and reuse. Not sure what best practice is for nodejs though.

Is there a way to invoke AWS Lambda synchronously from node.js?

I'm trying to run a specific function from an existing app via AWS Lambda, using the JS SDK to invoke Lambda from my node.js app. Since I'm overwriting the existing function, I'll have to keep its basic structure, which is this:
overwrittenFunction = function(params) {
//get some data
return dataArray;
}
..so I need to end up with an array that I can return, if I'm looking to keep the underlying structure of the lib I use the same. Now as far as I know, Lambda invocations are asynchronous, and it's therefore not possible to do something like this:
overwrittenFunction = function(params) {
lambda.invoke(params, callback);
function callback(err,data) {
var dataArray = data;
}
return dataArray;
}
(I've also tried similar things with promises and async/await).
afaik I have two options now: somehow figure out how to do a synchronous Lambda invocation, or modify my library / existing app (which I would rather not do if possible).
Is there any way to do such a thing and somehow return the value I'm expecting?
(I'm using node v8.9.4)
Lambda and async/await are a bit tricky, but the following is working for me (in production):
const lambdaParams = {
FunctionName: 'my-lambda',
// RequestResponse is important here. Without it we won't get the result Payload
InvocationType: 'RequestResponse',
LogType: 'Tail', // other option is 'None'
Payload: {
something: 'anything'
}
};
// Lambda expects the Payload to be stringified JSON
lambdaParams.Payload = JSON.stringify(lambdaParams.Payload);
const lambdaResult = await lambda.invoke(lambdaParams).promise();
logger.debug('Lambda completed, result: ', lambdaResult.Payload);
const resultObject = JSON.parse(lambdaResult.Payload)
Wrap it all up in a try/catch and go to town.
You can use async await but as the AWS SDK uses node callback pattern you'll need to wrap the function with the built-in promisify.
const promisify = require('utils').promisify
const aws = require('aws-sdk');
const lambda = aws.Lambda();
const invoke = promisify(lambda.invoke);
async function invocation(params) {
try {
return await invoke(params);
} catch (err) {
throw new Error('Somethings up');
}
}
const data = invocation(params);

Best Practices in Serverless Framework

I am new beginner in serverless framwork.
When study Best Practices in Serverless.
here
I have a question about "Initialize external services outside of your Lambda code".
How to implement it?
For example: Below code in handler.js
const getOneUser = (event, callback) => {
let response = null;
// validate parameters
if (event.accountid && process.env.SERVERLESS_SURVEYTABLE) {
let docClient = new aws.DynamoDB.DocumentClient();
let params = {
TableName: process.env.SERVERLESS_USERTABLE,
Key: {
accountid: event.accountid,
}
};
docClient.get(params, function(err, data) {
if (err) {
// console.error("Unable to get an item with the request: ", JSON.stringify(params), " along with error: ", JSON.stringify(err));
return callback(getDynamoDBError(err), null);
} else {
if (data.Item) { // got response
// compose response
response = {
accountid: data.Item.accountid,
username: data.Item.username,
email: data.Item.email,
role: data.Item.role,
};
return callback(null, response);
} else {
// console.error("Unable to get an item with the request: ", JSON.stringify(params));
return callback(new Error("404 Not Found: Unable to get an item with the request: " + JSON.stringify(params)), null);
}
}
});
}
// incomplete parameters
else {
return callback(new Error("400 Bad Request: Missing parameters: " + JSON.stringify(event)), null);
}
};
The question is that how to initial DynamoDB outside of my Lambda code.
Update 2:
Is below code optimized?
Handler.js
let survey = require('./survey');
module.exports.handler = (event, context, callback) => {
return survey.getOneSurvey({
accountid: event.accountid,
surveyid: event.surveyid
}, callback);
};
survey.js
let docClient = new aws.DynamoDB.DocumentClient();
module.exports = (() => {
const getOneSurvey = (event, callback) {....
docClient.get(params, function(err, data)...
....
};
return{
getOneSurvey : getOneSurvey,
}
})();
Here's the quote in question:
Initialize external services outside of your Lambda code
When using services (like DynamoDB) make sure to initialize outside of your lambda code. Ex: module initializer (for Node), or to a static constructor (for Java). If you initiate a connection to DDB inside the Lambda function, that code will run on every invoke.
In other words, in the same file, but outside of -- before -- the actual handler code.
let docClient = new aws.DynamoDB...
...
const getOneUser = (event, callback) => {
....
docClient.get(params, ...
When the container starts, the code outside the handler runs. When subsequent function invocations reuse the same container, you save resources and time by not instantiating the external services again. Containers are often reused, but each container only handles one concurrent request at a time, and how often they are reused and for how long is outside your control... Unless you update the function, in which case any existing containers will no longer be reused, because they'd have the old version of the function.
Your code will work as written, but isn't optimized.
The caveat with this approach that arises in current generation Node.js Lambda functions (Node 4.x/6.x) is that some objects -- notably, those that create literal persistent connections to external services -- will prevent the event loop from becoming empty (a common example is a mysql database connection, which is holding a live TCP connection to the server; by contrast, a DynamoDB "connection" is actually connectionless, since it's transport protocol is HTTPS). In this case you need to either take a different approach or allow lambda to not wait for an empty event loop before freezing the container, by setting context.callbackWaitsForEmptyEventLoop to false before calling the callback... but only do this if needed and only if you fully understand what it means. Setting it by default because some guy on the Internet said it was a good idea will potentially bring you mysterious bugs, later.

How do you structure sequential AWS service calls within lambda given all the calls are asynchronous?

I'm coming from a java background so a bit of a newbie on Javascript conventions needed for Lambda.
I've got a lambda function which is meant to do several AWS tasks in a particular order, depending on the result of the previous task.
Given that each task reports its results asynchronously, I'm wondering if the right way make sure they all happen in the right sequence, and the results of one operation are available to the invocation of the next function.
It seems like I have to invoike each function in the callback of the prior function, but seems like that will some kind of deep nesting and wondering if that is the proper way to do this.
For example on of these functions requires a DynamoDB getItem, following by a call to SNS to get an endpoint, followed by a SNS call to send a message, followed by a DynamoDB write.
What's the right way to do that in lambda javascript, accounting for all that asynchronicity?
I like the answer from #jonathanbaraldi but I think it would be better if you manage control flow with Promises. The Q library has some convenience functions like nbind which help convert node style callback API's like the aws-sdk into promises.
So in this example I'll send an email, and then as soon as the email response comes back I'll send a second email. This is essentially what was asked, calling multiple services in sequence. I'm using the then method of promises to manage that in a vertically readable way. Also using catch to handle errors. I think it reads much better just simply nesting callback functions.
var Q = require('q');
var AWS = require('aws-sdk');
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';
// Use a promised version of sendEmail
var ses = new AWS.SES({apiVersion: '2010-12-01'});
var sendEmail = Q.nbind(ses.sendEmail, ses);
exports.handler = function(event, context) {
console.log(event.nome);
console.log(event.email);
console.log(event.mensagem);
var nome = event.nome;
var email = event.email;
var mensagem = event.mensagem;
var to = ['email#company.com.br'];
var from = 'site#company.com.br';
// Send email
mensagem = ""+nome+"||"+email+"||"+mensagem+"";
console.log(mensagem);
var params = {
Source: from,
Destination: { ToAddresses: to },
Message: {
Subject: {
Data: 'Form contact our Site'
},
Body: {
Text: {
Data: mensagem,
}
}
};
// Here is the white-meat of the program right here.
sendEmail(params)
.then(sendAnotherEmail)
.then(success)
.catch(logErrors);
function sendAnotherEmail(data) {
console.log("FIRST EMAIL SENT="+data);
// send a second one.
return sendEmail(params);
}
function logErrors(err) {
console.log("ERROR="+err, err.stack);
context.done();
}
function success(data) {
console.log("SECOND EMAIL SENT="+data);
context.done();
}
}
Short answer:
Use Async / Await — and Call the AWS service (SNS for example) with a .promise() extension to tell aws-sdk to use the promise-ified version of that service function instead of the call back based version.
Since you want to execute them in a specific order you can use Async / Await assuming that the parent function you are calling them from is itself async.
For example:
let snsResult = await sns.publish({
Message: snsPayload,
MessageStructure: 'json',
TargetArn: endPointArn
}, async function (err, data) {
if (err) {
console.log("SNS Push Failed:");
console.log(err.stack);
return;
}
console.log('SNS push suceeded: ' + data);
return data;
}).promise();
The important part is the .promise() on the end there. Full docs on using aws-sdk in an async / promise based manner can be found here: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html
In order to run another aws-sdk task you would similarly add await and the .promise() extension to that function (assuming that is available).
For anyone who runs into this thread and is actually looking to simply push promises to an array and wait for that WHOLE array to finish (without regard to which promise executes first) I ended up with something like this:
let snsPromises = [] // declare array to hold promises
let snsResult = await sns.publish({
Message: snsPayload,
MessageStructure: 'json',
TargetArn: endPointArn
}, async function (err, data) {
if (err) {
console.log("Search Push Failed:");
console.log(err.stack);
return;
}
console.log('Search push suceeded: ' + data);
return data;
}).promise();
snsPromises.push(snsResult)
await Promise.all(snsPromises)
Hope that helps someone that randomly stumbles on this via google like I did!
I don't know Lambda but you should look into the node async library as a way to sequence asynchronous functions.
async has made my life a lot easier and my code much more orderly without the deep nesting issue you mentioned in your question.
Typical async code might look like:
async.waterfall([
function doTheFirstThing(callback) {
db.somecollection.find({}).toArray(callback);
},
function useresult(dbFindResult, callback) {
do some other stuff (could be synch or async)
etc etc etc
callback(null);
],
function (err) {
//this last function runs anytime any callback has an error, or if no error
// then when the last function in the array above invokes callback.
if (err) { sendForTheCodeDoctor(); }
});
Have a look at the async doco at the link above. There are many useful functions for serial, parallel, waterfall, and many more. Async is actively maintained and seems very reliable.
good luck!
A very specific solution that comes to mind is cascading Lambda calls. For example, you could write:
A Lambda function gets something from DynamoDB, then invokes…
…a Lambda function that calls SNS to get an endpoint, then invokes…
…a Lambda function that sends a message through SNS, then invokes…
…a Lambda function that writes to DynamoDB
All of those functions take the output from the previous function as input. This is of course very fine-grained, and you might decide to group certain calls. Doing it this way avoids callback hell in your JS code at least.
(As a side note, I'm not sure how well DynamoDB integrates with Lambda. AWS might emit change events for records that can then be processed through Lambda.)
Just saw this old thread. Note that future versions of JS will improve that. Take a look at the ES2017 async/await syntax that streamlines an async nested callback mess into a clean sync like code.
Now there are some polyfills that can provide you this functionality based on ES2016 syntax.
As a last FYI - AWS Lambda now supports .Net Core which provides this clean async syntax out of the box.
I would like to offer the following solution, which simply creates a nested function structure.
// start with the last action
var next = function() { context.succeed(); };
// for every new function, pass it the old one
next = (function(param1, param2, next) {
return function() { serviceCall(param1, param2, next); };
})("x", "y", next);
What this does is to copy all of the variables for the function call you want to make, then nests them inside the previous call. You'll want to schedule your events backwards. This is really just the same as making a pyramid of callbacks, but works when you don't know ahead of time the structure or quantity of function calls. You have to wrap the function in a closure so that the correct value is copied over.
In this way I am able to sequence AWS service calls such that they go 1-2-3 and end with closing the context. Presumably you could also structure it as a stack instead of this pseudo-recursion.
I found this article which seems to have the answer in native javascript.
Five patterns to help you tame asynchronis javascript.
By default Javascript is asynchronous.
So, everything that you have to do, it's not to use those libraries, you can, but there's simple ways to solve this. In this code, I sent the email, with the data that comes from the event, but if you want, you just need to add more functions inside functions.
What is important is the place where your context.done(); is going to be, he is going to end your Lambda function. You need to put him in the end of the last function.
var AWS = require('aws-sdk');
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';
var ses = new AWS.SES({apiVersion: '2010-12-01'});
exports.handler = function(event, context) {
console.log(event.nome);
console.log(event.email);
console.log(event.mensagem);
nome = event.nome;
email = event.email;
mensagem = event.mensagem;
var to = ['email#company.com.br'];
var from = 'site#company.com.br';
// Send email
mensagem = ""+nome+"||"+email+"||"+mensagem+"";
console.log(mensagem);
ses.sendEmail( {
Source: from,
Destination: { ToAddresses: to },
Message: {
Subject: {
Data: 'Form contact our Site'
},
Body: {
Text: {
Data: mensagem,
}
}
}
},
function(err, data) {
if (err) {
console.log("ERROR="+err, err.stack);
context.done();
} else {
console.log("EMAIL SENT="+data);
context.done();
}
});
}

when/node liftAll not working on s3 object

I'm trying to lift all the functions in an Amazon S3 object using when/node.
var when = require('when');
var nodefn = require('when/node');
var AWS = require('aws-sdk');
var s3 = new AWS.S3();
var promisedS3 = nodefn.liftAll(s3);
when(promisedS3.listBuckets())
.then(function(data) {
console.log(data);
})
However, it looks like a request object is being printed out. I'm kind of at a loss as to what is happening here, I can get the correct results if I individually lift functions like so:
var listBucketsP = nodefn.lift(s3.listBuckets.bind(s3));
Any ideas?
Try this:
nodefn.liftAll(s3.__proto__, undefined, s3);
Then just do
s3.listBuckets().then(function(data) {
console.log(data);
});
This worked for me.
Explanation: the methods you are trying to modify are not part of the s3 object itself, but of its prototype. When's node.liftAll's 3 argument version takes the source object first, an optional transformation function, and finally the destination object (to attach the lifted functions onto).
So we're taking the functions from the prototype and attach the promisified versions to the object we're working with.

Resources