I'm trying to run Azure Orchestrations and I sunddenly I started getting the error from the statusQueryGetUri:
Orchestrator function 'UploadDocumentOrchestrator' failed: The function 'UploadDocumentOrchestrator' doesn't exist, is disabled, or is not an orchestrator function.
I've run this functions dozens of times today without any issue. VSCode doesn't flag any issues. I've tried restarting and that hasn't helped either.
This has happened once before. I got the error, troubleshot for like 3 hours, got frustrated, left, and came back to my computer an hour later and the problem was resolved without changing anything. I suspect the issue is related to caching somehow, but I'm not sure where to fix this.
There is a similar issue noted at:
https://github.com/Azure/azure-functions-durable-extension/issues/577
After I run func start, my "missing" orchestration appears in the launch screen indicating that it is valid as shown below.
I start the UploadDocumentOrchestrator with the HttpTrigger StartUploadDocuments. I see the trigger complete its execution UploadDocumentOrchestrator never starts in the logs.
UploadDocumentOrchestrator: orchestrationTrigger
For detailed output, run func with --verbose flag.
[2023-01-11T19:58:10.490Z] Executing 'StartUploadDocuments' (Reason='This function was programmatically called via the host APIs.', Id=f158ac3e-fa39-403c-9671-307ea54d5948)
[2023-01-11T19:58:10.599Z] Started orchestration with ID = '29b9ab7fcffe4f59bd47032ef21c19e8'.
[2023-01-11T19:58:10.620Z] Executed 'StartUploadDocuments' (Succeeded, Id=f158ac3e-fa39-403c-9671-307ea54d5948, Duration=154ms)
[2023-01-11T19:58:13.952Z] Host lock lease acquired by instance ID '0000000000000000000000004906C298'.
As it mentioned in the Same GitHub Ticket, it is solved by clearing the Azure storage emulator data, check here.
It might be due to the Caching issue and also #ConnorMcMahon given another scenario in GitHub Issue #1381 of the same error - renaming the function caused the same issue for a few users:
The function ‘XYZ’ doesn’t exist, is disabled, or is not an orchestrator function.
Related
An error has started popping up in my Azure Data Factory Pipeline. I have a few Azure Function steps in the pipeline, but for some reason, one of the Azure Function steps has started returning an error. In Azure Data Factory, the error is a 3608 code after running for 1 minute 40 seconds:
Failure type: User configuration issue
Details: Call to provided Azure function 'CollateSheetsHTTPTrigger' failed with status-'InternalServerError' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'.
However, in a prior run sub-pipeline, this Azure Function ran successfully on the same data (parameters and worksheet are on the only difference). The subsequent 3 runs of pipelines fail immediately (after 2 seconds) at the first Azure Function (a different AZ function now) step in each, with the same 3608 error code but different details:
Call to provided Azure function '???????????????' failed with status-'NotFound'
and message - '<html> <head><title>404 Not Found</title></head> <body
bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center>
</body> </html> '.
Now it gets even stranger. After these 3 failed pipelines, the next pipeline which is pretty much the same as the previous 4 except for a few parameters, runs successfully, even though it has the same 2 AZ functions that failed before. And then the next 2 pretty similar pipelines also run successfully.
I then went and looked at the monitoring page for the 2 Azure Functions:
The first AZ function that failed, had 2 errors even though it only failed once in AZ Data Factory... the timing is slightly different for the 2 errors but they could only come from the first failed pipeline, so why does it say there are 2 errors? Then if you look at the actual error, all it says is "The operation timed out". The function was not running for more than 150 seconds so this is strange. Additionally, I have a bunch of error catching code and nothing comes up there.
The other failed AZ function steps from the other function do not show up on the monitoring page, it seems as if the first error crashed the AZ function app and then it eventually restarted?
I'm sorry I can't help but I did have a similar problem with an Azure function that executes a SOAP-call to a webservice every minute. Since 4 days this also fails with a timeout. If I run the function within my debugger it runs without problems. But the Azure Function fails every time, after 20 sec.
I'll follow this question and hope someone else can help...
An Azure Support Engineer identified the issue, it was due to a change to the azure-function-host library. The relevant issue is here https://protect-eu.mimecast.com/s/-CT5C3QxrTmREBwhgPxLm?domain=github.com and was fixed last week
I am trying to diagnose this error when my timer function runs. I have not found much help on this on google search
UnscheduledInvocationReason: IsPastDue, OriginalSchedule: 2019-06-13T15:13:00.0000000-07:00
It seems like the process just stops when this error comes.
Anyone have any insight on this?
The IsPastDue flag is passed to your azure function to indicate if the timer was overdue or not. A timer function can run late in some scenarios like the app service was restarted, in this case it is still invoked but the IsPastDue flag will be set to true to give your function a chance to react.
These links are helpful
Timer trigger for Azure Functions
Also, it seems that it was an issue and it is submitted on github but that was on 2017
TimerTrigger can miss IsPastDue
Got this error locally.
[2021-06-16T14:58:22.779Z] Executing 'Functions.TimerTrigger'
(Reason='Timer fired at 2021-06-16T16:58:22.7688953+02:00',
Id=adbaee54-8a3e-4983-a7e4-a73f69153e5e) [2021-06-16T14:58:22.780Z]
Trigger Details: UnscheduledInvocationReason: IsPastDue,
OriginalSchedule: 2021-06-16T16:37:00.0000000+02:00
[2021-06-16T14:59:22.614Z] Starting worker process failed
[2021-06-16T14:59:22.615Z] The operation has timed out.
Solved by clearing Blob Emulator locally:
delete
FROM [AzureStorageEmulatorDb510].[dbo].[Blob]
I have also asked this question in the MSDN Azure forums, but have not received any guidance as to why my function goes idle.
I have an Azure function running on a Consumption plan that goes idle (i.e. does not respond to new messages on the ServiceBus trigger queue) despite following the instructions outlined in this GitHub issue:
The configuration for the function is the following json:
{
"ConnectionStrings": {
"MyConnectionString": "Server=tcp:project.database.windows.net,1433;Database=myDB;User ID=user#project;Password=password;Encrypt=True;Connection Timeout=30;"
},
"Values": {
"serviceBusConnection": "Endpoint=sb://project.servicebus.windows.net/;SharedAccessKeyName=SharedAccessKeyName;SharedAccessKey=KEY_HERE",
}
}
And the function signature is:
public static void ProcessQueue([ServiceBusTrigger("queueName", AccessRights.Listen, Connection = "serviceBusConnection")] ...)
Based on the discussion in the GitHub issue, I believed that having either a serviceBusConnection entry OR an AzureWebJobServiceBus entry should be enough to ensure that the central listener triggers the function when a new message is added to the ServiceBusQueue, but that is proving to not be the case.
Can anyone clarify the difference between how those two settings are used, or notice anything else with the settings I provided that might be causing the function to not properly be triggered after a period of inactivity?
I suggest there are several possible causes for this behavior. I have several Azure subs and only one of them had issues with Storage/Service Bus-based triggers only popping up when app is not idle. So far I have observed that actions listed below will prevent triggers from working correctly:
Creating any Storage-based trigger, deleting (for any reason) the triggering object and re-creating it.
Corrupting azure function input parameters by deleting/altering associated objects without recompiling a function
Restarting functions app when one of the functions fails to compile/bind to trigger OR input parameter and hangs may cause same problems.
It has also been observed that using legacy Connection Strings setting for trigger binding will not work.
Clean deploy of an affected function app will most likely solve the problem if it was caused by any of the actions described above.
EDIT:
It looks like this is also caused by setting Authorization/Authentication on the functions app, but I have not yet figured out if it happens in general or when Auth has specific configuration. Tested on affected Azure sub by disabling auth at all - function going idle after 30-40 mins, queue trigger still initiates an execution, though with a delay as expected. I have found an old bug related to this, but it says issue resolved.
I am working with Azure Queues, Controlling them using WebJob, In the functions file i have one QueueTrigger function, which fires up when AzureQueue receives some message.
Now the problem is that function (QueueTrigger) executes successfully, I have setup proper exception handling inside queue trigger function, everything executed without any errors. But when QueueTrigger function ends, code is throwing exception. Storage Exception 404 Not found.
and that message is not getting deleted from queue. Next time when i run job it is still fetching the old message. I have manually created storage containers
azure-jobs-host-archive
azure-jobs-host-output
azure-webjobs-dashboard
azure-webjobs-hosts
I have seen in one post answer but this does not help at all.
I have no idea how to tackle this exception or why this exception in throwing in the code.
Thank you
Eman
After updating the following packages the problem get solved.
Microsoft.WindowAzure.Storage
Microsoft.Azure.Webjobs
Microsoft.Azure.WebJobs.Host
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.