Trying to send several messages (from AWS SQS lambda, if that matters) but it's never waiting for the promises.
function getEndpoint(settings){
return new Promise(function(resolve, reject) {
// [...] more stuff here
}
Which is then called in a loop:
exports.handler = async (event) => {
var messages = [];
event.Records.forEach(function(messageId, body) {
//options object created from some stuff
messages.push(getEndpoint(options).then(function(response){
console.log("anything at all"); //NEVER LOGGED
}));
});
await Promise.all(messages);
};
But the await seems to be flat out skipped. I'm not sure how I'm getting Process exited before completing request with an explicit await. I have similar async await/promise setups in other scripts that work, but cannot spot what I've done wrong with this one.
You forgot to return something to lambda:
exports.handler = async (event) => {
var messages = [];
event.Records.forEach(function(messageId, body) {
//options object created from some stuff
messages.push(getEndpoint(options));
});
await Promise.all(messages);
return 'OK'
};
this should also work:
exports.handler = (event) => { // async is not mandatory here
var messages = [];
event.Records.forEach(function(messageId, body) {
//options object created from some stuff
messages.push(getEndpoint(options));
});
return Promise.all(messages); // returning a promise
};
and you could use map:
exports.handler = (event) => { // async is not mandatory here
const messages = event.Records.map(function(messageId, body) {
//options object created from some stuff
return getEndpoint(options)
});
return Promise.all(messages); // returning a promise
};
To understand why this happens, you must dive a bit into lambda's implementation: it will essentially wait for the function stack to be cleared and since you did NOT return anything at all in there, the function stack got empty right after it queued all the stuff - adding a simple return after the await call makes the fn stack to NOT be empty which means lambda will wait for it to be finished.
If you run this on standard node, your function would also return before the promises were finished BUT your node process would NOT exit until the stack was cleared. This is where lambda diverges from stock node.
Related
I'm trying to use AWS lambda to test a few API calls using axios, however I'm having some trouble. Every post I came across said the best way to handle promises in Lambda was to use async/await rather than .then, so I made the switch. When I run the program using node it works perfectly, but when I invoke the Lambda locally, it seems like everything after the axios call is being skipped. When I invoke the Lambda locally without await, the calls after it run fine, but then I'm forced to use .then which the Lambda doesn't wait for anyway. I've increased the Lambda timeout to 900, and I've run sam build before sam invoke local every time.
function checkServers() {
console.log("Inside checkServer");
console.log("Before apis to test");
// apisToTest has length of 2
apisToTest.forEach(async (apiToTest) => {
console.log("Api to test");
let res = await axios(apiToTest)
console.log("x"); // This gets skipped
console.log(res); // This gets skipped
})
console.log("After api to test")
}
exports.lambdaHandler = async (event, context) => {
console.log("Inside lambda handler");
checkServers();
console.log("After lambda handler");
};
// Used to test app using node command
checkServers()
This yields the following output:
INFO Inside lambda handler
INFO Inside checkServer
INFO Before apis to test
INFO Api to test
INFO Api to test
INFO After api to test
INFO After lambda handler
Thanks for all of your replies, unfortunately those weren't the ideal solutions for my use case, though they were very helpful in me coming up with the solution.
async function checkServers() {
let emailBody = "";
let callResult = "";
let completedCalls = 0;
let promises = [];
for (const apiToTest of apisToTest) {
await axios(apiToTest).then((res) => {
// Do something
}).catch((r) => {
// Handle error
})
}
}
exports.lambdaHandler = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = true;
await checkServers();
};
To summarize, I replaced the forEach call to a for...of call, changed the checkServers to async, and combined await with .then() and .catch to handle the Promise result. I was unaware that both can be used to do so in tandem. Hope this helps anyone that had an issue similar to the one I had.
you got two options:
set callbackWaitsForEmptyEventLoop to true - and then AWS will wait until the eventloop is empty
await your promises. Note that you didn't await your promises on checkServers. Consider using await Promise.all([...]) to wait for all your promises to finish.
Note: If you will run the lambda again, the events from the previous invocations may "leak" to the next invocation.
You can learn about it here: https://lumigo.io/blog/node-js-lambda-execution-leaks-a-practical-guide/
Disclosure: I work in the company that published this blog post
Similar to what saart is saying you could try the below code and see if it works:
async function checkServers() {
const promises = apisToTest.map((apiToTest) => axios(apiToTest))
const resolvedPromises = await Promise.all(promises);
return resolvedPromises;
}
exports.lambdaHandler = async (event, context) => {
try {
const result = await checkServers();
console.log(result);
} catch (error) {
console.log(error)
}
};
// Used to test app using node command
checkServers();
I have created a lambda for dynamodb trigger and it is working ok if I use lambda function with callback but as soon as I change it to async it stops working. It is getting called but call returns before the end of the internal function. I am using await for any async call.
const handler: Handler = async (event: any, context: Context, callback : Callback) => { console.log(event);
try{
// construct request handler
console.log(event.Records);
const createHandler = dummyHandler;
await createHandler.handleRequest(event.Records);
return callback(null, 'Successfully processed ${event.Records.length} records.');
}
catch(err){
console.log(err);
throw err;
}
finally{
console.log('Finally I am here');
}
}
So this code works. Though strangely finally is executed before the handleRequest completes. But still handleRequest does complete as expected.
But since callback is kinda older and unwanted version I am trying to remove it and add promise in my async handler and then it stops working as expected.
const handler: Handler = async (event: any, context: Context): Promise<boolean> =>
{
console.log(event);
try{
// construct request handler
console.log(event.Records);
const createHandler = dummyHandler;
const result = await createHandler.handleRequest(event.Records);
return result;
}
catch(err){
console.log(err);
throw err;
}
finally{
console.log('Finally I am here');
}
}
In my internal function, I am doing few db queries and updates. I am using await in all async calls. But still after my first db query, finally block is being executed. In handler with callback, other db calls still execute but in async version with no callback, after finally is called, nothing else happens and request is ended.
I have checked memory size (256MB) and timeout (300seconds) for lambda function and neither of them are exceeding.
So in case anyone else experiences this issue. I ended up making following changes to my async handler without callback.
const handler: Handler = async (event: any, context: Context):Promise<boolean> => {
console.log(event);
try{
// construct request handler
console.log(event.Records);
const createHandler = new ...();
return await createHandler.handleRequest(event.Records);
}
catch(err){
console.log(err);
throw err;
}
finally{
console.log('Event Record');
}
This seems to be working as expected. All my db queries and updates are completed (basically this call createHandler.handleRequest is ended properly) before the request ends. Though needs extensive testing. Still welcome any feedback.
Basically it was working with promise but jarmod was right, I just needed to return the await call.
Simple call to ec2 Describing Security groups and returning the security group ID. Using Async / await, but when logging the return value, I get undefined. I fully admit I'm coming from Python and I've tried my hardest to wrap my brain around async calls. I thought I had it nailed, but I'm obviously missing something.
'use strict';
// Load Modules
const AWS = require('aws-sdk')
//Set the region
AWS.config.update({region: 'us-west-2'});
// Call AWS Resources
const ec2 = new AWS.EC2();
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
var ec2params = { Filters: [{Name: 'tag:t_whitelist',Values[event['site']]}]};
await ec2.describeSecurityGroups(ec2params, function (err, response) {
if (err) {return console.error(err.message)}
else {
var sgId = response.SecurityGroups[0].GroupId;
return sgId;
};
});
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => {console.log(sgId)});
}
"sgId" should return the security group ID. It does print out fine in the original function before the return.
Typically if it is an async call you want you handle it similar to this way without using a callback
// Load Modules
const AWS = require('aws-sdk')
//Set the region
AWS.config.update({ region: 'us-west-2' });
// Call AWS Resources
const ec2 = new AWS.EC2();
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
var ec2params = { Filters: [{ Name: 'tag:t_whitelist', Values[event['site']]}] };
try {
const securityGroupsDesc = await ec2.describeSecurityGroups(ec2params).promise();
const sgId = securityGroupsDesc.SecurityGroups[0].GroupId;
//do something with the returned result
return sgId;
}
catch (error) {
console.log('handle error');
// throw error;
}
});
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => { console.log(sgId) });
}
however if it doesn't support async you just use the callback to handle the returned data or error without using async function.However Reading into AWS docs you can find that the function ec2.describeSecurityGroups() returns an AWS Request
which has a method promise() that needs to be invoked to send the request and get a promise returned.Note that the try catch here is not needed but good to have in case error occurs during the process.
As I said in the comment, chance are that describeSecurityGroups doesn't return a Promise. Try transforming it explictly in a Promise instead:
const promiseResponse = await new Promise((res, rej) => {
ec2.describeSecurityGroups(ec2params, function (err, response) {
if (err) {return rej(err.message)}
else {
var sgId = response.SecurityGroups[0].GroupId;
res(sgId);
};
})
});
// promiseResponse is now equal to sgId inside the callback
return promiseResponse; // this will work because the function is async
Note: You can drop the else keyword
Here is the code that worked using async / await. Thanks to #Cristian Traina I realized ec2.describeSecurityGroups wasn't returning a promise, it was returning an AWS.Event.
// Get Security Group ID From Event
const getSgIdFromEvent = async (event) => {
console.log('Getting Security Group ID')
var params = { Filters: [{Name: 'tag:t_whitelist', Values
[event['site']]}]};
const describeSG = await ec2.describeSecurityGroups(params).promise();
return describeSG.SecurityGroups[0].GroupId;
};
// Get Ingress Rules from Security Group
const getSgIngressRules = async (sgId) => {
console.log(`Getting SG Ingress rules for ${sgId}`)
var params = { GroupIds: [ sgId]};
try{
const ingressRules = await ec2.describeSecurityGroups(params).promise();
return ingressRules;
}
catch (error) {
console.log("Something went wrong getting Ingress Ruls");
}
};
// MAIN FUNCTION
exports.handler = (event, context) => {
getSgIdFromEvent(event)
.then(sgId => {return getSgIngressRules(sgId);})
.then(ingressRules => {console.log(ingressRules);});
}
I submitted this as the answer now since the getSgIdFromEvent function I have, is only 8 lines and still using the async/await like I was desiring.
What I was missing was the .promise() on the end of the function and returning that promise.
Thanks for all the responses!
I'm having some issues with how my functions flow in lambda. I'm trying to grab value stored in S3, increment it, and put it back. However, my program doesn't flow how I feel it should be. I'm using async waterfall to run the flow of my functions.
Here's my code:
let AWS = require('aws-sdk');
let async = require('async');
let bucket = "MY_BUCKET";
let key = "MY_FILE.txt";
exports.handler = async (event) => {
let s3 = new AWS.S3();
async.waterfall([
download,
increment,
upload
], function (err) {
if (err) {
console.error(err);
} else {
console.log("Increment successful");
}
console.log("test4");
return null;
}
);
console.log("test5");
function download(next) {
console.log("test");
s3.getObject({
Bucket: bucket,
Key: key
},
next);
}
function increment(response, next) {
console.log("test2");
console.log(response.Body);
let newID = parseInt(response.Body, 10) + 1;
next(response.ContentType, newID);
}
function upload(contentType, data, next) {
console.log("test3");
s3.putObject({
Bucket: bucket,
Key: key,
Body: data,
ContentType: contentType
},
next);
}
};
I'm only getting test and test5 on my log. I was under the impression that after the download function, increment should run if it was okay or the callback function at the end of the waterfall should run if there was an error. The program doesn't give an error on execution and it doesn't appear to go to either function.
Could someone guide me to what I'm missing in my understanding?
EDIT: So it seems my issue was related to my function declaration. The default template declared it as async(event). I thought this was odd as usually they are declared as (event, context, callback). Switching to the later (or even just (event) without the async) fixed this. It looks like my issue is with calling the function as asynchronous. This blocked the waterfall async calls?? Can anyone elaborate on this?
Your problem is that your handler is declared as an async function, which will create a promise for you automatically, but since you are not awaiting at all your function is essentially ending synchronously.
There are a couple of ways to solve this, all of which we'll go over.
Do not use promises, use callbacks as the async library is designed to use.
Do not use the async library or callbacks and instead use async/await.
Mix both together and make your own promise and resolve/reject it manually.
1. Do not use promises
In this solution, you would remove the async keyword and add the callback parameter lambda is passing to you. Simply calling it will end the lambda, passing it an error will signal that the function failed.
// Include the callback parameter ────┐
exports.handler = (event, context, callback) => {
const params =[
download,
increment,
upload
]
async.waterfall(params, (err) => {
// To end the lambda call the callback here ──────┐
if (err) return callback(err); // error case ──┤
callback({ ok: true }); // success case ──┘
});
};
2. Use async/await
The idea here is to not use callback style but to instead use the Promise based async/await keywords. If you return a promise lambda will use that promise to handle lambda completion instead of the callback.
If you have a function with the async keyword it will automatically return a promise that is transparent to your code.
To do this we need to modify your code to no longer use the async library and to make your other functions async as well.
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
const Bucket = "MY_BUCKET";
const Key = "MY_FILE.txt";
async function download() {
const params = {
Bucket,
Key
}
return s3.getObject(params).promise(); // You can await or return a promise
}
function increment(response) {
// This function is synchronous, no need for promises or callbacks
const { ContentType: contentType, Body } = response;
const newId = parseInt(Body, 10) + 1;
return { contentType, newId };
}
async function upload({ contentType: ContentType, newId: Body }) {
const params = {
Bucket,
Key,
Body,
ContentType
};
return s3.putObject(params).promise();
}
exports.handler = async (event) => {
const obj = await download(); // await the promise completion
const data = increment(obj); // call synchronously without await
await upload(data)
// The handlers promise will be resolved after the above are
// all completed, the return result will be the lambdas return value.
return { ok: true };
};
3. Mix promises and callbacks
In this approach we are still using the async library which is callback based but our outer function is promised based. This is fine but in this scenario we need to make our own promise manually and resolve or reject it in the waterfall handler.
exports.handler = async (event) => {
// In an async function you can either use one or more `await`'s or
// return a promise, or both.
return new Promise((resolve, reject) => {
const steps = [
download,
increment,
upload
];
async.waterfall(steps, function (err) {
// Instead of a callback we are calling resolve or reject
// given to us by the promise we are running in.
if (err) return reject(err);
resolve({ ok: true });
});
});
};
Misc
In addition to the main problem of callbacks vs. promises you are encountering you have a few minor issues I noticed:
Misc 1
You should be using const rather than let most of the time. The only time you should use let is if you intend to reassign the variable, and most of the time you shouldn't do that. I would challenge you with ways to write code that never requires let, it will help improve your code in general.
Misc 2
You have an issue in one of your waterfall steps where you are returning response.ContentType as the first argument to next, this is a bug because it will interpret that as an error. The signature for the callback is next(err, result) so you should be doing this in your increment and upload functions:
function increment(response, next) {
const { ContentType: contentType, Body: body } = response;
const newId = parseInt(body, 10) + 1;
next(null, { contentType, newId }); // pass null for err
}
function upload(result, next) {
const { contentType, newId } = result;
s3.putObject({
Bucket: bucket,
Key: key,
Body: newId,
ContentType: contentType
},
next);
}
If you don't pass null or undefined for err when calling next async will interpret that as an error and will skip the rest of the waterfall and go right to the completion handler passing in that error.
Misc 3
What you need to know about context.callbackWaitsForEmptyEventLoop is that even if you complete the function correctly, in one of the ways discussed above your lambda may still hang open and eventually timeout rather than successfully complete. Based on your code sample here you won't need to worry about that probably but the reason why this can happen is if you happen to have something that isn't closed properly such as a persistent connection to a database or websocket or something like that. Setting this flag to false at the beginning of your lambda execution will cause the process to exit regardless of anything keeping the event loop alive, and will force them to close ungracefully.
In the case below your lambda can do the work successfully and even return a success result but it will hang open until it timesout and be reported as an error. It can even be re-invoked over and over depending on how it's triggered.
exports.handler = async (event) => {
const db = await connect()
await db.write(data)
// await db.close() // Whoops forgot to close my connection!
return { ok: true }
}
In that case simply calling db.close() would solve the issue but sometimes its not obvious what is hanging around in the event loop and you just need a sledge hammer type solution to close the lambda, which is what context.callbackWaitsForEmptyEventLoop = false is for!
exports.handler = async (event) => {
context.callbackWaitsForEmptyEventLoop = false
const db = await connect()
await db.write(data)
return { ok: true }
}
The above will complete the lambda as soon as the function returns, killing all connections or anything else living in the event loop still.
Your function terminates before the waterfall is resolved. That is, the asynchronous calls aren't executed at all. That is why you don't see any of the console.log calls you have within the waterfall functions, and only see the one that is called synchronously immediately after the call to async.waterfall.
Not sure how well async.waterfall is supported by AWS Lambda, but since promises are natively supported and perform the same functionality (with fewer loc), you could use promises instead. Your code would look something like this:
module.exports.handler = (event,context) =>
s3.getObject({
Bucket: bucket,
Key: key
}).promise()
.then(response => ({
Body: parseInt(response.Body, 10) + 1,
ContentType: response.contentType,
}))
.then(modifiedResponse => s3.putObject({
Bucket: bucket,
Key: key,
Body: modifiedResponse.data,
ContentType: modifiedResponse.contentType}).promise())
.catch(err => console.error(err));
I am trying to run a small snippet of lambda code where i am pushing data to S3 using firehose. Here is my snippet
const AWS = require( 'aws-sdk' );
var FIREhose = new AWS.Firehose();
exports.handler = async (event,context,callback) => {
// TODO implement
const response = {
statusCode:200,
Name:event.Name,
Value:event.Value
};
const params = {
DeliveryStreamName: 'kinesis-firehose',
Record: { Data: new Buffer(JSON.stringify(response)) }
};
FIREhose.putRecord(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
};
Here are my events
{
"Name": "Mike",
"Value": "66"
}
When i run this lambda all i am getting response as null . Since i am not passing any callback lambda will default run the implicit callback and returns null. I see that no data is pushed to S3 bucket.
But when i add callback(null,"success") line at the end like this
FIREhose.putRecord(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
callback(null,"success")
};
I see the data is pushed to S3. Why is that ?
Does async functions always need a callback with some text appended to it ?
Any help is appreciated ?
Thanks
The problem here is that you're mixing your node.js lambda patterns.
Either you use an asynchronous function and return or throw:
exports.handler = async (event,context,callback) => {
// code goes here.
await FIREhose.putRecord(params).promise();
return null; // or whatever result.
};
Or you use the callback approach:
exports.handler = (event,context,callback) => {
// code goes here.
FIREhose.putRecord(params)
.promise();
.then((data) => {
// do stuff with data.
// n.b. you could have used the cb instead of a promise here too.
callback(null, null); // or whatever result.
});
};
(There's a third way using context. but that's a very legacy way).
This is all due to how lambda works and detects when there's been a response.
In your first example (no callback), lambda is expecting your handler to return a promise that it has to wait to resolve/reject, which, in turn, will be the response. However, you're not returning a promise (undefined) and so there's nothing to wait for and it immediately returns- quite probably before the putRecord call has completed.
When you used callback though, you explicitly told lambda that you're using the "old" way. And the interesting thing about the callback approach is that it waits for node's event loop to complete (by default). Which means that .putRecord will probably complete.