I'm using a Logic App where the workflow calls at a certain point an Azure Function using the Webhook URL (as a workaround to Azure Functions Durable).
The goal of this function is to insert/update data into an Azure SQL Database with a SQL request
"MERGE INTO...USING...WHEN NOT MATCHED...WHEN MATCHED AND...".
In the logs of the Azure Function, i could see it failed and it seems to run 4 times (maybe due to the supposed Timeout, I don't know), but I don't understand since I increased the CommandTimeout to 50minutes and I set 1Hour to the Timeout of the action "Launch Webhook" in the LogicApp :S Here's the sample of the exception logged in the Azure Function :
Exception while executing function: XmlImport_DoWork
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: XmlImport_DoWork ---> System.Data.SqlClient.SqlException : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception : The wait operation timed out
The table actually have around 250,000 lines and it seems to be good when I launch the LogicApp (and so the Azure Function) to a table which is almost empty !
Any ideas about what's going on and how to fix it ? I tried to look at the "Query Performance Insight" in Azure SQL database component but there are nothing in "Recommendations" section
The Function App where are stored my Azure Functions is using an App Service Plan.
BTW the XML file I was trying to import in DB has a size of 20M but I tried with a lighter XML (9M) but it didn't work either
Azure Durable Function: V2 and .Net Core 2.2 - Timeout expired issue RESOLOVED
The activity function 'A_ValidateAndImportData' failed: "Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.". See the function execution logs for additional details.
Using DAPPER to call SQL Server stored procedure: Dapper not honoring "Connection Timeout" Property in the connection string
Solution: Use a connection timeout parameter to provide "0"(ZERO or increase timeout according to your need) to solve this problem
Example Code:
public async Task<int> ValidateAndImportData(string connectionString, int param1,
int databaseTimeOut = 0)
{
using (var connection = new SqlConnection(connectionString))
{
var param = new DynamicParameters();
param.Add("#param1", param1);
param.Add("#returnStatus", dbType: DbType.Int32, direction: ParameterDirection.Output);
await connection.ExecuteAsync("[dbo].[ValidateAndImportData]", param,
commandType: CommandType.StoredProcedure, commandTimeout: databaseTimeOut).ConfigureAwait(false);
return param.Get<int>("returnStatus");
}
}
Related
I have an Azure Function App and Azure SQL DB.
In my Azure Function I query data (Using EF Core 3) and the following code:
var stuffIWant = dbContext.UsageCounts
.Where(a=> a.Elo > 0 && a.Elo < 1300)
.GroupBy(a => a.TheHash)
.Select(b => new stuff { Hash = (long)b.Key, Count = b.Count() })
.OrderByDescending(a => a.Count).Take(10).ToList();
I am getting a very high failure rate with an error that looks like:
[Error] Executed 'FunctionName' (Failed, Id=456456-0040-4349-81e3-54646546, Duration=30220ms)The wait operation timed out.
The exception:
[Error] Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
When I execute the function it sometimes (rarely) works fine and is able to make 8 queries like this (with the bounds for Elo changing) and each takes <200ms to complete.
I can also run this in a Sandbox project on my local machine, connecting to the same Azure DB using EF with the same model and can run it hundreds of times without ever timing out, each query taking <200ms to complete.
When the Azure function does work it always goes through each of the 8 queries and returns the data, when it doesn't work it always fails at the first query.
I added a "test" query to my function:
var test = dbContext.UsageCounts.Where(a => a.Elo > 2200).Take(10).ToList();
This happens before my failing query, and always succeeds.
Why is my function timing out most of the time when this query is nowhere near the execution time limit?
My database is not set to auto-pause.
My compute and IO utilization is under 20%
To resolve this [Error] Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding., you can try following ways:
In Tools->Options->Query Execution->SQL Server->General->Execution time-out. Set the time-out to more than 30 seconds or 0 to wait for the indefinite time.
Check connection string, e.g. connectionString="Data Source=.;Initial Catalog=LMS;User ID=test;Password=test#123"
You can refer to The wait operation timed out establishing a connection to SQL Server FROM Azure Function App, Azure Function Execution Timeout Expired and Troubleshooting connectivity issues and other errors with Azure SQL Database and Azure SQL Managed Instance
I'm using an Azure function that sends an array of around 200 documents to a CosmosDB via the Output Binding. That function gets triggered about 1000 at the same time by queue messages.
In some cases I get the "Request rate is large" error and the function execution fails. The documentation says when this error occurs, I can retry the execution in some milliseconds, but I suspect the azure function runtime is doing that for me. I couldn't find any documentation explicitly saying that when the output binding throws that exception it will retry automatically (like with the .NET Linq library).
Can someone point me out to see if this is the case?
The Output binding uses SDK 1.13.2 which already has the retry mechanism in place.
Assuming you are using Azure Functions v1, if you are using the IAsyncCollection the Function will do an UpsertDocumentAsync for each AddAsync, if you are using a single document output, then the UpsertDocumentAsync should be happening once.
In any case, the SDK retries by default 9 times on a throttled result, after that, the exception is bubbled and you Function will error; the document should go back to the queue for retrying as per the QueueTrigger design and after a couple of iterations, it goes to the deadletter queue..
If you want more granular control of the flow, you could obtain the DocumentClient and do the UpsertDocumentAsync yourself with a try/catch, if it fails more than 9 times, you can opt to send to another Queue or retry another set of times. Something like:
using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
using Microsoft.Azure.Documents.Linq;
[FunctionName("CosmosDBSample")]
public static async Task<HttpResponseMessage> Run(
[QueueTrigger("my-queue")] MyPOCOClass myMessage,
[DocumentDB("test", "test", ConnectionStringSetting = "CosmosDB"] DocumentClient client,
TraceWriter log)
{
try
{
await client.UpsertDocumentAsync(myMessage);
}
catch(DocumentClientException ex)
{
// retry / queue somewhere else?
log.Warning($"DocumentClientException {ex.Message} in document {myMessage.Id}.");
}
}
Like the title describes - I have an Azure Function on the App Service Plan, configured for Always On and no functionTimeout set in my host.json, and it appears to timeout / not finish anytime after 30 minutes to 1 hour.(...but I feel this may be a false positive...)
The HTTP Triggered function can sometimes take over 1-2 hours to complete. I understand that this probably isn't the best design and according to the Azure Function Best Practices I should break this out into smaller / more manageable pieces - I get that. However, I expect the Function on the App Service plan to work as advertised - no hard limit on execution time. Perhaps this is the same question as Unexpected azure-function timeouts on app-service-plan, but that has no answer and I am using an HTTP Trigger instead.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
According to the Kudu Function Invocation Logs, this case reports "Never Finished", and when I click on the Toggle Output button to view the logs, they never come in.
When I viewed this function's run in the Logs section of that trigger, it seems like the function just stopped, and the log stream just reports no new trace:
2017-07-26T16:36:43.116 [INFO] [Class1] Update operation started processing 790 sales records ...
2017-07-26T16:36:43.116 [DBUG] [Class2] Matching and updating ids from the map...
2017-07-26T16:38:07 No new trace in the past 1 min(s).
2017-07-26T16:39:07 No new trace in the past 2 min(s).
2017-07-26T16:40:07 No new trace in the past 3 min(s).
2017-07-26T16:41:07 No new trace in the past 4 min(s).
So not sure why this function just seemed to stop - or perhaps it stopped collecting log statements (there are many), and for some reason, the function never completed.
Any ideas?
Approx time: 2017-07-26T16:00:00 UTC
InvocationID: d856c107-f1ee-455a-892b-ed970dcad128 (I think?)
If it is indeed being timed out, is there any way for us to know, (Exception? App Insights? etc.)
Based on my test, I found azure function will not stop your function if you don't set the timeout.
Here is my test, I create a ManualTrigger function which will log the message every 10 minutes.
The codes like below:
public static void Run(string input, TraceWriter log)
{
for (int i = 0; i < 100; i++)
{
log.Info( "Worked " + i*10 + " minutes ");
Thread.Sleep(600000);
}
}
The log details:
In the log, you could find my function executed 70 minutes.It still works well.
The no trace means there are no new requests send to the azure function.
Currently, the HTTP Triggered method does not return until the work is complete. (Is this a problem - the HTTP trigger needs to return quicker?)
As Jesse Carter says, you couldn't execute long time function when you used HTTP Triggered method.
Since your client-side(send request) will have a timeout value. It will wait for the function's response.
Normally, if we want to execute long time function, I suggest you could use http trigger to get the request. In the http trigger function you could add a queue message to the azure storage queue.
Then you could write a queue trigger function which will execute the long time work.
If your HTTP method takes more than a minute, you should be offloading it to a Queue. Period. (I know the other answers have said this, but it's worth repeating).
Http connections are a limited resource.
While Azure Functions as an execution engine can handle long running
operations (as demonstrated by queue / service bus support), the
http pipeline may cut off / timeout long running requests.
Queue triggers can easily run for 30+ minutes. If your job is longer than that, you really should split it into multiple queue messages.
Also check out Durable Function support: https://github.com/Azure/azure-functions-durable-extension/
Regardless of the function app timeout setting, 230 seconds is the maximum amount of time that an HTTP triggered function can take to respond to a request. This is because of the default idle timeout of Azure Load Balancer. For longer processing times, consider using the Durable Functions async pattern or defer the actual work and return an immediate response.
Function app timeout duration: Check Notes
I am getting the time out error, when I called Azure Service Bus with the below code.
So I have applied retry logic, however still I am getting the time out error.
var msgFactory = MessagingFactory.CreateFromConnectionString(connection);
var namespaceManager = NamespaceManager.CreateFromConnectionString(connection);
if (!await namespaceManager.QueueExistsAsync(queueName)) //Time out Error
{///Code}
The request has timed out after 60000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. TrackingId:143b4d25-e97c-4270-8714-93a4c6818fea,TimeStamp:1/19/2016 8:30:36 PM
Maybe just because a pair of parentheses missing in:
if (!await namespaceManager.QueueExistsAsync(queueName))
which should be:
if (!(await namespaceManager.QueueExistsAsync(QueueName)))
We need to run the pregame outside of the company firewall, then it will work.
I have the following logic in a WebJob using the new 0.3.0-beta WebJobs SDK. When my code fails processing the message, the Azure dashboard shows an Aggregate Exception (which makes sense since this is async). HOWEVER, it does not retry processing the message.
The very little documentation I've been able to find indicates that the message should be retried within 10 minutes of failure. Is this not the case with the new SDK?
public static Task ProcessMyMessageAsync(
[QueueTrigger(Config.MY_QUEUE)] string msg,
int dequeueCount,
CancellationToken cancellationToken)
{
var processor = Config.Container.GetInstance<IMessageProcessor>();
return processor.HandleJobAsync(msg, dequeueCount, cancellationToken);
}
The exception I get stems from a SQL Timeout exception (its a db query against SQL Azure in my code):
System.AggregateException: System.AggregateException: One or more errors occurred.
---> System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception: The wait operation timed out
You should set MaxDequeueCount.
JobHostConfiguration jobHostConf = new JobHostConfiguration();
jobHostConf.Queues.MaxDequeueCount = 10;
var host = new JobHost(jobHostConf);
host.RunAndBlock();
That will retry 10 times, before the messages is put on a dead/bad letter queue.
You could also use a custom retry policy in the function. I suggest you look at "The Transient Fault Handling Application Block" https://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx
Or you could enable retry in EF with a SqlAzureExecutionStrategy
https://msdn.microsoft.com/en-us/data/dn456835.aspx