I have an AWS lambda that does a very expensive job and sometimes it takes 14m to finish (lambda function has a 15m timeout).
The call to this lambda is done using the lambda.invoke method from the aws-sdk with the following settings:
const lambda = new AWS.Lambda({
maxRetries: 0,
httpOptions: {
timeout: (15 * 60 * 1000) + (20 * 1000) // time to wait for a response (lambda timeout + 20s)
}
});
In my nodejs logs I'm able to see:
Feb 07 14:29:43.825 invoking lambda
Feb 07 14:45:04.396
Closing job 151218 due to some error: "TimeoutError: Connection timed out after 920000ms"
While on my AWS lambda logs I see
Feb 07 14:29:44.329 INIT_START Runtime Version: java:8.v14
...
Feb 07 14:40:22.293 REPORT RequestId: xxx Duration: 635736.88 ms
As you can see, the invoke + lambda start is consistent, but the lambda ends at 14:40 but the nodejs invoke method is stuck waiting for the response and it times out at 14:45.
I've read the AWS documentation regarding timeouts (https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/) and everything seems correct, so I do not know why my invoke method does not return once the lambda finished successfully.
Any ideas?
Related
I added below block in serverless.yml file to schedule a CRON:
testCron1:
handler: handler.testCron1
events:
- schedule: cron(15 14 22 MAY FRI 2020)
Then I create a testCron1 lambda function in handle.js file.
const testCron1 = (event, context) => {
return new Promise(async (resolve, reject) => {
console.log("*********** NEW est CRON ************");
console.log("Current Time ==> ", new Date(moment().utc().format()));
resolve(true);
});
}
When I try to deploy using command serverless deploy, it gives me below error:
An error occurred: TestCron1EventsRuleSchedule1 - Parameter ScheduleExpression is not valid. (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code:ValidationException; Request ID: c5b3178b-348a-4ffd-a205-d5255dcca2ab)
If you are using lambda use cloud watch as a cron trigger.
it would be easy to handle the lambda function trigger at a particular time with cloud watch.
you can find trigger rules expression here : https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
Supported value list you can check here : https://docs.aws.amazon.com/systems-manager/latest/userguide/reference-cron-and-rate-expressions.html
Example cron : 0 11 17 4 ? 2020 min hours day month dayOfWeek year
This will execute once on FRI, 17 APR 2020 11:00:00 GMT as an example.
I am calling a lambda via aws-sdk with explicit RequestResponse
const Lambda = require("aws-sdk/clients/lambda");
const lambda = new Lambda({region: "region"});
const params = {
FunctionName: "myFunctionArn",
InvocationType: "RequestResponse",
Payload: JSON.stringify({
//... my payload
})
};
lambda.invoke(params).promise().then(data => {
console.log(data);
});
It is a long running task >~5 minutes and my timeout is set to 10 minutes. I download an mp3, compress it, save it to S3 and then return the url to the client.
There are no errors in cloudwatch and the process goes smoothly, the mp3 is store with lower quality to S3, however the function gets executed two times.
If the mp3 file is small enough (~8 MB) there is only one execution, however if it is a big file (~100MB) it will get executed two times and of course the function will timeout. I am using the /tmp folder to store the file temporarily, I will as well remove them after the mp3 has been safely stored in S3
I have scattered my function with logging and there is absolutely no sign of errors and this is happening every single time, not sporadically.
Those are my cloudwatch logs
Thanks
EDIT_1: I have tried by adding some options to the lambda client
const Lambda = require("aws-sdk/clients/lambda");
const lambda = new Lambda({
region: "ap-southeast-1",
httpOptions: { timeout: 10 * 60 * 1000, connectTimeout: 10 * 60 * 1000 }
});
But looks like nothing has changed
EDIT_2: seems like now this is happening also for short running tasks and is completely random. I am at loss here I really don't know what to do
Setting maxRetries to 0 in the SDK config params will turn off retries being invoked by the SDK itself.
const Lambda = require("aws-sdk/clients/lambda");
const lambda = new Lambda({
maxRetries: 0,
region: "ap-southeast-1",
httpOptions: { timeout: 10 * 60 * 1000, connectTimeout: 10 * 60 * 1000 }
});
After spending countless hours through debugging, I just discovered that the cause of this weird behavior was because the function was going out of memory.
Incredibly enough nothing showed up on Cloudwatch.
When writing a Lambda function which utilizes the aws-sdk NodeJS package for a Shopify webhook, I noticed that the require statement for the package is taking over 3 seconds to load. This is causing issues because Shopify requires a response from its webhooks within 5 seconds. I abstracted the code out of my function to test it by itself and received the same results:
exports.handler = (event, context, callback) => {
const start = new Date().getTime();
console.log('start: '+ new Date())
require('aws-sdk');
console.log('end:' + new Date())
const end = new Date().getTime();
console.log('length: ' + (end - start) + 'ms');
callback();
}
Here was the output:
START RequestId: Version: $LATEST
2017-11-30T13:23:57.506Z start: Thu Nov 30 2017 13:23:57 GMT+0000 (UTC)
END RequestId:
REPORT RequestId: Duration: 3001.29 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 31 MB
2017-11-30T13:24:00.499Z Task timed out after 3.00 seconds
The issue appears to be that I was not giving the Lambda function enough memory to load in the package. I increased the memory with the following results:
512MB brought it down to 1000ms
1GB brought it down to 500ms
We have created a NodeJS based Lambda function named - execInspector which gets triggered everyday once. This function is created based on AWS Lambda blueprint --> "inspector-scheduled-run" in NodeJS.
The problem we see is the scheduled job fails randomly one day or the other. We are getting only the below logs from the cloudwatch log stream.
In a week, it randomly runs =~ 4/5 times & fails remaining days. Based on the log, it consumes only very little amount of memory/time for its execution but not sure why it fails randomly. It also retries itself 3 times before getting killed.
From the below log we could also find that the job only takes 35 MB avg. & takes only 60 sec to complete on an avg. We tried modifying the NodeJS run time, increasing memory, timeouts well beyond this limit but nothing worked out.
Can you please help with some alternate approaches to handle these failures automatically & if anyone has insights on why its happening?
Additional Inputs:
We have already given 5 mins of maximum timeout also, but it fails saying "timed out after 300 secs.".
What i mean here is the task of just triggering the inspector takes only less than 30 secs on avg. Since, its a PaaS based solution, I cannot expect always this to be completed within 30 secs. But 60 secs should be more than enough for this to handle a job which it was able to complete within 30 secs.
Sample CloudWatch Successful log:
18:01:00
START RequestId: 12eb468a-4174-11e7-be7b-6d0faaa584aa Version: $LATEST
18:01:03
2017-05-25T18:01:02.935Z 12eb468a-4174-11e7-be7b-6d0faaa584aa { assessmentRunArn: 'arn:aws:inspector:us-east-1:102461617910:target/0-Ly60lmEP/template/0-POpZxSLA/run/0-MMx30fLl' }
2017-05-25T18:01:02.935Z 12eb468a-4174-11e7-be7b-6d0faaa584aa { assessmentRunArn: 'arn:aws:inspector:us-east-1:102461617910:target/0-Ly60lmEP/template/0-POpZxSLA/run/0-MMx30fLl' }
18:01:03
END RequestId: 12eb468a-4174-11e7-be7b-6d0faaa584aa
END RequestId: 12eb468a-4174-11e7-be7b-6d0faaa584aa
18:01:03
REPORT RequestId: 12eb468a-4174-11e7-be7b-6d0faaa584aa Duration: 2346.37 ms Billed Duration: 2400 ms Memory Size: 128 MB Max Memory Used: 33 MB
REPORT RequestId: 12eb468a-4174-11e7-be7b-6d0faaa584aa Duration: 2346.37 ms Billed Duration: 2400 ms Memory Size: 128 MB Max Memory Used: 33 MB
Cloudwatch log:
Similar log below is repeated 3 times which seems to be a retry attempt
06:32:52
START RequestId: 80190395-404a-11e7-845d-1f88a00ed4f3 Version: $LATEST
06:32:56
2017-05-24T06:32:55.942Z 80190395-404a-11e7-845d-1f88a00ed4f3 Execution Started...
06:33:52
END RequestId: 80190395-404a-11e7-845d-1f88a00ed4f3
06:33:52
REPORT RequestId: 80190395-404a-11e7-845d-1f88a00ed4f3 Duration: 60000.88 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 32 MB
06:33:52
2017-05-24T06:33:52.437Z 80190395-404a-11e7-845d-1f88a00ed4f3 Task timed out after 60.00 seconds
2017-05-24T06:33:52.437Z 80190395-404a-11e7-845d-1f88a00ed4f3 Task timed out after 60.00 seconds
Lambda code:
'use strict';
/**
* A blueprint to schedule a recurring assessment run for an Amazon Inspector assessment template.
*
* This blueprint assumes that you've already done the following:
* 1. onboarded with the Amazon Inspector service https://aws.amazon.com/inspector
* 2. created an assessment target - what hosts you want to assess
* 3. created an assessment template - how you want to assess your target
*
* Then, all you need to do to use this blueprint is to define an environment variable in the Lambda console called
* `assessmentTemplateArn` and provide the template arn you want to run on a schedule.
*/
const AWS = require('aws-sdk');
const inspector = new AWS.Inspector();
const params = {
assessmentTemplateArn: process.env.assessmentTemplateArn,
};
exports.handler = (event, context, callback) => {
try {
// Inspector.StartAssessmentRun response will look something like:
// {"assessmentRunArn":"arn:aws:inspector:us-west-2:123456789012:target/0-wJ0KWygn/template/0-jRPJqnQh/run/0-Ga1lDjhP"
inspector.startAssessmentRun(params, (error, data) => {
if (error) {
console.log(error, error.stack);
return callback(error);
}
console.log(data);
return callback(null, data);
});
} catch (error) {
console.log('Caught Error: ', error);
callback(error);
}
};
The log says your request is timing out after 60 seconds. You can set it as high as 5 minutes according to this https://aws.amazon.com/blogs/aws/aws-lambda-update-python-vpc-increased-function-duration-scheduling-and-more/ If your task takes about 60 seconds and the timeout is 60 secs then maybe some are timing out. Thats what the log suggests to me. Otherwise, post some code from the function
I try to develop a little browser game based on NodeJS and Angular 4.
I have an API server running on NodeJS which is connected to a MongoDB and a second server running Angular 4.
I want to execute recurring standard functions (like every 15 minutes) in the background.
Do I need a third server which runs that functions? Or can I run that functions independently on my API server - no matter which route is open?
You might want to have a look to this library node-cron. You can set it up to work with your services. You will need to initialise the job right after your sever is initialised. An example:
var CronJob = require('cron').CronJob;
var job = new CronJob({
cronTime: '00 30 11 * * 1-5',
onTick: function() {
/*
* Runs every weekday (Monday through Friday)
* at 11:30:00 AM. It does not run on Saturday
* or Sunday.
*/
},
start: false,
timeZone: 'America/Los_Angeles'
});
job.start();
You can use setTimeout() and setInterval() in Node just like in the browser:
setInterval(() => {
// this runs every 15 minutes
}, 15 * 60 * 1000);