I have a Python 3.6 - Flask application deployed onto AWS Lambda using Zappa, in which I have an asynchronous task execution function defined using #Task as discussed here
However, I find that the function call still times out at 30 seconds as against the 5 minute timeout that AWS Lambda enforces for non-API calls. I even checked the timeout in my Lambda settings and it is set to 5 minutes.
The way I discovered this is when the lambda's debug output started repeating without a request - something that happens because the lamba is called 2 more times because of either an error or timeout (as per the AWS Lambda documentation).
Can anyone help me with getting this resolved?
[EDIT : The lambda function is also not part of any VPC and is set to be accessible from the internet.]
Here are the logs below. Basically, the countdown is a sleep timer counting to 20 seconds, followed by a #task call to application.reviv_assign_responder, but as we see, there is no outpust past 'NEAREST RESPONDER' and the countdown starts again, indicating that the function has timed out and has been called again by (AWS') design.
Log output in Pastebin : https://pastebin.com/VEbdCALg
Second incident - https://pastebin.com/ScNhbMcn
As we can see in the second log, it clearly states:
[1515842321866] wait_one_and_notify : 30 : 26 [1515842322867]
wait_one_and_notify : 30 : 27 [1515842323868] wait_one_and_notify : 30
: 28 [1515842324865] 2018-01-13T11:18:44.865Z
72a8d34a-f853-11e7-ac2f-dd12a3d35bcb Task timed out after 30.03
seconds
You can check the default settings that Zappa applies to all your lambda functions here, and you will see that by default timeout_seconds is set-up to 30 seconds, This will apply over the default Lambda setup in AWS Console, because by default this is 3 seconds (you can check this limit in AWS Lambda FAQ.
For your #Task you must increase/setup your timeout_seconds in your zappa_settings.(json|yaml) file and redeploy this, You can put 5 mins (5*60==300 seconds) but this increase will be for all your functions defined in your virtualenv deployed with zappa.
You can check more details exposed in this issue in Zappa repo.
The timeout_seconds parameter in Zappa is misleading. That is, it does limit the timeout of the Lambda function, but the requests are served through CloudFront, which has a default timeout of 30 seconds. To verify that, try lowering the timeout_seconds to 20 - it will correctly timeout in 20 seconds. However past 30 there is no effect because of CloudFront limitation.
The default timeout is 30 seconds. You can change the value to be from 4 to 60 seconds. If you need a timeout value outside that range, request a change to the limit.
In other words, there is nothing you can do in either Zappa or Lambda to fix this, because the problem lies elsewhere (CloudFront).
I haven't tried it myself, but you might be able to up the limit by creating the cloudfront distribution in front of Lambda, though it seems you are still limited by max. 60s (unless you request more through AWS support, as indicated in the previous link).
Related
We have a ServiceBusTrigger Azure Function. During the execution of an event, it will make an API call to different service. However, before the API call returns with result, if it takes longer time, the Azure Function would throw an exception with message like the following:
Exception while executing function: FunctionName One or more errors occurred. (A task was canceled.) A task was canceled.
From the multiple occurrences, this happened after the event is triggered by 100 seconds.
My question is: Is this due to timeout? If yes, why it's timing out/cancelling the task only after 100 seconds? Shouldn't the Azure Function default timeout be 5-minute?
Thanks for any answers in advance.
After more tests, it was determined the timeout/cancellation was due to the API call as a default timeout of 100 seconds. After changing to a longer timeout, the issue was resolved.
I trigger a Post Function Import (Action in CDS), this would typically take about 2 minutes for processing. The POST operation was successfully completed in JAVA, however I get a Gateway Timeout.
How to increase the timeout of incoming requests? I had tried to set the property INCOMING_CONNECTION_TIMEOUT: 0 in mta.yaml of service project as well as using the command
cf set-env x-service-name-blue INCOMING_CONNECTION_TIMEOUT 0
cf restage x-service-name-blue
It did not work either.
Could you assist?
Update: I think the correct environment variable on the approuter is called SESSION_TIMEOUT. Can you try this one instead?
This is for the XS Advanced approuter, though I'm not sure if it still applies to the one used for CF apps, this documentation suggest that it's a property of the approuter, so you can try setting it there.
I'm building an API that works with AWS route53 to add, delete and update records for a specific zone.
When I'm using change_resource_record_set(), I'm getting a response with status = 'PENDING'. So my idea is to run a while loop with get_change() until the status will be INSYNC. in Boto3 route53 documentation they say it takes up to 60 seconds, but I think it is ridiculous to wait 60 seconds for each update. Do you have any ideas on how I can make sure the changes took place at the AWS servers?
i have the following code in my azure function with 5 minutes manual timeout.
when i run the above function in azure, i see the function creates a new instance after 3 minutes.(check the below image)
both the instances completes successfully ,but returns Status: 504 Gateway Timeout which in turn fails my function execution.
i have hosted the function in App Service Plan, and also increased the timeout in host.json file to 10 minutes
"functionTimeout": "00:10:00"
Several questions in here:
Timeouts - The function timeout in host.json applies to the underlying function runtime; not the http pipeline. You should not have an http function running longer than a minute. The http calls will timeout independently of the runtime (as you see with the 504). However, you could use that timeout for a long-running (ie, 60 minute on appservice plan) queue trigger. If you need a long-running function, the http call could queue a message for a queue trigger, or you could use the Durable Function support.
Why is it invoking again? The simplest interpretation here is that your function is just receiving a second http request message. Do you have evidence that's not the case? You could bind to the HttpRequestMessage and log additional http request properties to track this down.
Like the title describes - I have an Azure Function on the App Service Plan, configured for Always On and no functionTimeout set in my host.json, and it appears to timeout / not finish anytime after 30 minutes to 1 hour.(...but I feel this may be a false positive...)
The HTTP Triggered function can sometimes take over 1-2 hours to complete. I understand that this probably isn't the best design and according to the Azure Function Best Practices I should break this out into smaller / more manageable pieces - I get that. However, I expect the Function on the App Service plan to work as advertised - no hard limit on execution time. Perhaps this is the same question as Unexpected azure-function timeouts on app-service-plan, but that has no answer and I am using an HTTP Trigger instead.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
According to the Kudu Function Invocation Logs, this case reports "Never Finished", and when I click on the Toggle Output button to view the logs, they never come in.
When I viewed this function's run in the Logs section of that trigger, it seems like the function just stopped, and the log stream just reports no new trace:
2017-07-26T16:36:43.116 [INFO] [Class1] Update operation started processing 790 sales records ...
2017-07-26T16:36:43.116 [DBUG] [Class2] Matching and updating ids from the map...
2017-07-26T16:38:07 No new trace in the past 1 min(s).
2017-07-26T16:39:07 No new trace in the past 2 min(s).
2017-07-26T16:40:07 No new trace in the past 3 min(s).
2017-07-26T16:41:07 No new trace in the past 4 min(s).
So not sure why this function just seemed to stop - or perhaps it stopped collecting log statements (there are many), and for some reason, the function never completed.
Any ideas?
Approx time: 2017-07-26T16:00:00 UTC
InvocationID: d856c107-f1ee-455a-892b-ed970dcad128 (I think?)
If it is indeed being timed out, is there any way for us to know, (Exception? App Insights? etc.)
Based on my test, I found azure function will not stop your function if you don't set the timeout.
Here is my test, I create a ManualTrigger function which will log the message every 10 minutes.
The codes like below:
public static void Run(string input, TraceWriter log)
{
for (int i = 0; i < 100; i++)
{
log.Info( "Worked " + i*10 + " minutes ");
Thread.Sleep(600000);
}
}
The log details:
In the log, you could find my function executed 70 minutes.It still works well.
The no trace means there are no new requests send to the azure function.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
As Jesse Carter says, you couldn't execute long time function when you used HTTP Triggered method.
Since your client-side(send request) will have a timeout value. It will wait for the function's response.
Normally, if we want to execute long time function, I suggest you could use http trigger to get the request. In the http trigger function you could add a queue message to the azure storage queue.
Then you could write a queue trigger function which will execute the long time work.
If your HTTP method takes more than a minute, you should be offloading it to a Queue. Period. (I know the other answers have said this, but it's worth repeating).
Http connections are a limited resource.
While Azure Functions as an execution engine can handle long running
operations (as demonstrated by queue / service bus support), the
http pipeline may cut off / timeout long running requests.
Queue triggers can easily run for 30+ minutes. If your job is longer than that, you really should split it into multiple queue messages.
Also check out Durable Function support: https://github.com/Azure/azure-functions-durable-extension/
Regardless of the function app timeout setting, 230 seconds is the maximum amount of time that an HTTP triggered function can take to respond to a request. This is because of the default idle timeout of Azure Load Balancer. For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.
Function app timeout duration: Check Notes