An error has started popping up in my Azure Data Factory Pipeline. I have a few Azure Function steps in the pipeline, but for some reason, one of the Azure Function steps has started returning an error. In Azure Data Factory, the error is a 3608 code after running for 1 minute 40 seconds:
Failure type: User configuration issue
Details: Call to provided Azure function 'CollateSheetsHTTPTrigger' failed with status-'InternalServerError' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'.
However, in a prior run sub-pipeline, this Azure Function ran successfully on the same data (parameters and worksheet are on the only difference). The subsequent 3 runs of pipelines fail immediately (after 2 seconds) at the first Azure Function (a different AZ function now) step in each, with the same 3608 error code but different details:
Call to provided Azure function '???????????????' failed with status-'NotFound'
and message - '<html> <head><title>404 Not Found</title></head> <body
bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center>
</body> </html> '.
Now it gets even stranger. After these 3 failed pipelines, the next pipeline which is pretty much the same as the previous 4 except for a few parameters, runs successfully, even though it has the same 2 AZ functions that failed before. And then the next 2 pretty similar pipelines also run successfully.
I then went and looked at the monitoring page for the 2 Azure Functions:
The first AZ function that failed, had 2 errors even though it only failed once in AZ Data Factory... the timing is slightly different for the 2 errors but they could only come from the first failed pipeline, so why does it say there are 2 errors? Then if you look at the actual error, all it says is "The operation timed out". The function was not running for more than 150 seconds so this is strange. Additionally, I have a bunch of error catching code and nothing comes up there.
The other failed AZ function steps from the other function do not show up on the monitoring page, it seems as if the first error crashed the AZ function app and then it eventually restarted?
I'm sorry I can't help but I did have a similar problem with an Azure function that executes a SOAP-call to a webservice every minute. Since 4 days this also fails with a timeout. If I run the function within my debugger it runs without problems. But the Azure Function fails every time, after 20 sec.
I'll follow this question and hope someone else can help...
An Azure Support Engineer identified the issue, it was due to a change to the azure-function-host library. The relevant issue is here https://protect-eu.mimecast.com/s/-CT5C3QxrTmREBwhgPxLm?domain=github.com and was fixed last week
Related
I'm trying to run Azure Orchestrations and I sunddenly I started getting the error from the statusQueryGetUri:
Orchestrator function 'UploadDocumentOrchestrator' failed: The function 'UploadDocumentOrchestrator' doesn't exist, is disabled, or is not an orchestrator function.
I've run this functions dozens of times today without any issue. VSCode doesn't flag any issues. I've tried restarting and that hasn't helped either.
This has happened once before. I got the error, troubleshot for like 3 hours, got frustrated, left, and came back to my computer an hour later and the problem was resolved without changing anything. I suspect the issue is related to caching somehow, but I'm not sure where to fix this.
There is a similar issue noted at:
https://github.com/Azure/azure-functions-durable-extension/issues/577
After I run func start, my "missing" orchestration appears in the launch screen indicating that it is valid as shown below.
I start the UploadDocumentOrchestrator with the HttpTrigger StartUploadDocuments. I see the trigger complete its execution UploadDocumentOrchestrator never starts in the logs.
UploadDocumentOrchestrator: orchestrationTrigger
For detailed output, run func with --verbose flag.
[2023-01-11T19:58:10.490Z] Executing 'StartUploadDocuments' (Reason='This function was programmatically called via the host APIs.', Id=f158ac3e-fa39-403c-9671-307ea54d5948)
[2023-01-11T19:58:10.599Z] Started orchestration with ID = '29b9ab7fcffe4f59bd47032ef21c19e8'.
[2023-01-11T19:58:10.620Z] Executed 'StartUploadDocuments' (Succeeded, Id=f158ac3e-fa39-403c-9671-307ea54d5948, Duration=154ms)
[2023-01-11T19:58:13.952Z] Host lock lease acquired by instance ID '0000000000000000000000004906C298'.
As it mentioned in the Same GitHub Ticket, it is solved by clearing the Azure storage emulator data, check here.
It might be due to the Caching issue and also #ConnorMcMahon given another scenario in GitHub Issue #1381 of the same error - renaming the function caused the same issue for a few users:
The function ‘XYZ’ doesn’t exist, is disabled, or is not an orchestrator function.
I'm using Azure's Runs API to get a pipeline run result as described here:
https://learn.microsoft.com/en-us/rest/api/azure/devops/pipelines/runs/get?view=azure-devops-rest-6.0#runresult
I can see in the documentation how to get the state and final result so I can know if the run was a success or a failure. However, in case of a failure, I don't see how I can get the error that occurred in that run as a string.
How can I get the actual error which caused the pipeline run to fail?
You can use the REST API "Timeline - Get" to list the issues (error and warning) associated with a run.
Note:
This API can only list the first 10 issues. If the run has more than 10 issues, the rest will not be listed in the response. To get the complete issues, you can use the API "Builds - Get Build Log" or "Logs - Get" to get the complete logs that contains the complete issues.
[UPDATE]
The buildId is same as the runId, and you can find it from the URL of the pipeline (build) run.
The timelineId is not required in the API request, you can use the request URI like as below.
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline/?api-version=6.0
I have a Firebase project with 29 functions 2 with python and 27 with nodejs.
Modified 2 of them and now I can't deploy properly. I get an error log that send me to the logviewer and one of the errors is:
ERROR: build step 3
"us.gcr.io/fn-img/buildpacks/nodejs10/builder:nodejs10_20201201_20_RC00"
failed: step exited with non-zero status: 46
The functions keep on working, but I can't update/deploy properly. When I try to deploy them individually I get that error for both functions, but when I try to deploy ALL the functions I only get the error with those 2 functions the rest of the functions, that don't have any modification have no problem redeploying.
I checked the source code in the Cloud console and they have a warning icon saying that:
Function is active, but last deployment failed
The source code in the Cloud console is the same as the one I'm trying to deploy but the functions has the same functionality that before when I made the changes, the functions still works but can't update.
These are javascript functions that I deployed using the Firebase Node Sdk.
Any help?
EDIT I:
I reverted the changes on one of the functions that's been there for over 2 years and still have the same issue, can't update/deploy, that function triggers on storage.onFinalize().
The other function on firestore.onCreate()
EDIT II:
The newest function that I created is not in use, is part of a new feature in my android application, so I duplicated it, gave it different name and deployed without issues. In that case I could delete the original function without any issue as is not being used. But I can't do the same for the other function, the other function is constantly in use.
I have created 2 pipeline in Azure Datafactory. We have a custom activity created to run a python script inside the pipeline.When the pipeline is executed manually it successfully run for n number of time.But i have created a scheduled trigger of an interval of 15 minutes in order to run the 2 pipelines.The first execution successfully runs but in the next interval i am getting the error "Operation on target PyScript failed: Hit unexpected exception and execution failed." we are blocked wiht this.any input on this would be really helpful.
from ADF troubleshooting guide, it states...
Custom Activity :
The following table applies to Azure Batch.
Error code: 2500
Message: Hit unexpected exception and execution failed.
Cause: Can't launch command, or the program returned an error code.
Recommendation: Ensure that the executable file exists. If the program started, make sure stdout.txt and stderr.txt were uploaded to the storage account. It's a good practice to emit copious logs in your code for debugging.
Related helpful doc: Tutorial: Run Python scripts through Azure Data Factory using Azure Batch
Hope this helps.
If you are still blocked, please share failed pipeline run ID & failed activity run ID, for further analysis.
I am new to Azure development and developed a function app
I published my function app to Azure portal. It is working fine on my development machine but on portal it's throwing following exception (some times)
The operation 'ScanLogs' with id '2eggec6de-54f5-4t34-5423-afffce5c6a43' did not complete in '00:02:00'.
I couldn't find solution to this error. Can somebody help me to understand what this error is about and why we get this?
following is timeout specified in host.json in prod.
Depending on the version of function you are running check here for the exact syntax in host.json - https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json-v1
The default timeout for a function on the consumption plan is 2 minutes, if you need it to run longer then change/add the functiontimeout value, i.e.
"functionTimeout": "00:05:00",
in host.json.
NB: On a Consumption plan this can't be more than 10 minutes so if you need it to run longer either find a way to break up your function into smaller chunks and maybe use a Durable function fan=out fan-in pattern or change it to run on a dedicated App Service Plan where it can run for as long as you like but obviously you'll have to pay to have the server running 24/7