I'm running an Azure function in Azure, the function gets triggered by a file being uploaded to blob storage container. The function detects the new blob (file) but then outputs the following message - Did not find any initialized language workers.
Setup:
Azure function using Python 3.6.8
Running on linux machine
Built and deployed using azure devops (for ci/cd capability)
Blob Trigger Function
I have run the code locally using the same blob storage container, the same configuration values and the local instance of the azure function works as expected.
The functions core purpose is to read in the .xml file uploaded into blob storage container and parse and transform the data in the xml to be stored as Json in cosmos db.
I expect the process to complete like on my local instance with my documents in cosmos db, but it looks like the function doesn't actually get to process anything due to the following error:
Did not find any initialized language workers
Troy Witthoeft's answer was almost certainly the right one at the time the question was asked, but this error message is very general. I've had this error recently on runtime 3.0.14287.0. I saw the error on many attempted invocations over about 1 hour, but before and after that everything worked fine with no intervention.
I worked with an Azure support engineer who gave some pointers that could be generally useful:
Python versions: if you have function runtime version ~3 set under the Configuration blade, then the platform may choose any of python versions 3.6, 3.7, or 3.8 to run your code. So you should test your code against all three of these versions. Or, as per that link's suggestion, create the function app using the --runtime-version switch to specify a specific python version.
Consumption plans: this error may be related to a consumption-priced app having idled off and taking a little longer to warm back up again. This depends, of course, on the usage pattern of the app. (I infer (but the Engineer didn't say this) that perhaps if the Azure datacenter my app is in happens to be quite busy when my app wants to restart, it might just have to wait for some resources to become available.). You could address this either by paying for an always-on function app, or by rigging some kind of heartbeat process to stop the app idling for too long. (Easiest with a HTTP trigger: probably just ping it?)
The Engineer was able to see a lower-level error message generated by the Azure platform, that wasn't available to me in Application Insights: ARM authentication token validation failed. This was raised in Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication.ArmAuthenticationHandler.HandleAuthenticate() at /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/Authentication/Arm/ArmAuthenticationHandler.cs. There was a long stack trace with innermost exception being: System.Security.Cryptography.CryptographicException : Padding is invalid and cannot be removed.. Neither of us were able to make complete sense of this and I'm not clear whether the responsibility for this error lies within the HandleAuthenticate() call, or outside (invalid input token from... where?).
The last of these points may be some obscure bug within the Azure Functions Host codebase, or some other platform problem, or totally misleading and unrelated.
Same error but different technology, environment, and root cause.
Technology Net 5, target system windows. In my case, I was using dependency injection to add a few services, I was getting one parameter from the environment variables inside the .ConfigureServices() section, but when I deployed I forget to add the variable to the application settings in azure, because of that I was getting this weird error.
This is due to SDK version, I would suggest to deploy fresh function App in Azure and deploy your code there. 2 things to check :
Make sure your local function app SDK version matches with Azure function app.
Check python version both side.
This error is most likely github issue #4384. This bug was identified, and a fix was released mid-june 2020. Apps running on version 3.0.14063 or greater should be fine. List of versions is here.
You can use azure application insights to check your version. KUSTO Query the logs. The exception table, azure SDK column has your version.
If you are on the dedicated App Service plan, you may be able to "pull" the latest version from Microsoft by deleting and redeploying your app. If you are on consumption plan, then you may need to wait for this bugfix to rollout to all servers.
Took me a while to find the cause as well, but it was related to me installing a version of protobuf explicitly which conflicted with what was used by Azure Functions. Fair, there was a warning about that in the docs. How I found it: went to <your app name>.scm.azurewebsites.net/api/logstream and looked for any errors I could find.
Related
I've been working on a small function to automate my certificate renewal in Azure Functions.
The function works in my local emulator (in vscode), running under node v10.15.3.
However, when running it online, an exeption is generated on the syntax of an async iterator when the file containing it is included
Stack: /home/site/wwwroot/node_modules/acme-dns-01-cloudflare/index.js:125
for await(const zone of consumePages(pagination =>
It's my understanding that this syntax has been adopted in node versions 10.x. I therefore added the console output line: console.log(process.versions); and get the response that the function is running node version 8.16.1. I therefore checked the WEBSITE_NODE_DEFAULT_VERSION application setting, and confirmed it is set to 10.14.1. I have also checked it with another recommended setting of ~10 and got the same result.
Unfortunately the documentation is difficult to search for such a specific issue, but I have not yet come across anything that states that Linux functions are limited to node v8.x
As extra information, the FUNCTIONS_WORKER_RUNTIME is set to "node", and the runtime version is 2.0.12733.0 (~2)
At time of writing, this issue on github highlights the problem https://github.com/Azure/azure-functions-host/issues/4948. Different node versions are simply not available on Linux consumption plans regardless of the setting in WEBSITE_NODE_DEFAULT_VERSION.
Hopefully their new arrangements will be in place soon for anyone else who has this issue.
For now you can switch to a windows consumption plan, or potentially switch to a service plan (I haven't checked this, as it sort of defeats the point of functions)
Refer to this issue1 and issue2, looks like WEBSITE_NODE_DEFAULT_VERSION won't work for linux function, have to set the LinuxFxVersion property to select the node version.
Follow is my flow to change it.
1.Go to your Function App in the portal and open the Resource Explorer. You will find LinuxFxVersion is node:2.0-node8-appservice.
2.Select the web under config node, then choose the Edit button. Find the linuxFxVersion and change the value to NODE|10.14, after this click the PUT button to update the setting. Then restart your Function, check the node version you will find it's 10.14.
I am using azure functions with the blob trigger and with the runtime 2.x. After the function being executed the runtime adds blob receipt into the location below (azure-webjob-hosts>blobreceipt>)
Is there any way to make runtime to use the same hostnameid in the path (in this case it's 8a76f42c1c01-2137340777)? Because if run function locally or publish new version of the code it would use different host identifier (like macbook-id).
The main problem is when I run func azure functionapp publish it will requeue all the existing blobs with new version of the code even when the new version has the same func name because it will use different host/func identifier to check for receipts.
I know that in runtime v1 you could add id into host.json. I dont see such option in runtime v2.
Edit:
Another thing. Just this morning I found out that my function run over all the blobs again. When I looked into blobreceipts I see that the hostid has changed and there are new bunch of receipts for the new host id. Host Id has changed even though I didnt do any deployments. How is that possible? I have also reported the bug on coretools about hostid getting changed on publish. https://github.com/Azure/azure-functions-core-tools/issues/1012
App name is altamente-prod-svc-shopify-background. But the host id is d7031c9b43fc-2137340777. I have removed all receipts to do more tests for now but you can see in logs that the func was initiated again this morning by itself over all my 7 blobs. There is just one function for now (and also WarmUp func which appeared automatically). I have created the linux func app via Azure portal with no functions and then just did publish with core tools from my mac.
There is a workaround now in the Azure Functions bug report that you linked to. Basically, set AzureFunctionsWebHost:hostid in the App settings to some explicit value. I'm using a self-hosted functions runtime and Kubernetes doesn't allow colons in environment variables, so I used the work-around of setting AzureFunctionsWebHost__hostid instead, which also works.
I have deployed my web job on production environment and suddenly blob trigger stops working(Looking into App Insight I know that blob trigger is not called).
If I debug the same code from the local machine then it triggers blob trigger.But stopped working in the production environment.
Always On is enabled.
Also, I have these containers are present
azure-jobs-host-archive,
azure-jobs-host-output,
azure-webjobs-dashboard,
Microsoft.Azure.WebJobs.Host,
Installed packages:-
Microsoft.Azure.Webjobs installed version is 2.0.0,
Microsoft.Azure.WebJobs.Host installed version is 2.0.0
According to your description, we couldn't directly find out why the blob trigger doesn't work. I suggest you could follow below way to troubleshoot by yourself.
If I debug the same code from the local machine then it triggers blob trigger.But stopped working in the production environment.
Since you found your webjob could work well in local, I guess maybe there are something wrong with the connection string config in your web app.
I suggest you could try to follow below way to change the production environment appsetting to set the storage connection string in it.
1.Open the portal and edit the appsetting as below.
2.Create another web app and publish the web job to it and run.
If these two ways all doesn't help you solve the error, I suggest you could post the log/error message and the details codes about the web job.
More details about how to find the log/error message, you could follow below way.
In Service Fabric cluster, If application has multiple versions(say 1.0.0,1.0.1,1.0.2), then how can we shift the application to one version to another version(say active is 1.0.0, then I wanted to shift to 1.0.1) with out redeploying the application. Is there a PowerShell command to do this?
You should be able to use the PowerShell command
Start-ServiceFabricApplicationUpgrade
This being said I did hit an issue with my local cluster, telling me I couldn't upgrade / roll back the application if the service description had changed, which it hadn't. Using an Azure hosted cluster this worked as expected, perhaps an inconsistency with how the package is copied into the image store.
Depending on what you are attempting to achieve you could also look at named instance where you are able to deploy multiple versions of an application at once, for A - B testing.
Here are some similar posts:
Post 1
Post 2
EDIT:
Thanks to Aleksey L for the comment below. With a bit of messing around due to types not being the same and as long as you haven't changed any parameters between versions this will work,if you have you will need to manually build up the hash table.
I am trying to deploy a Cloud Service with 1 Web Role to Azure.
When I do so, I get this message:
Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
The project runs just fine locally, and I'm having a hard time figuring out how to start debugging this issue. Are there any common problems that cause this message or steps to figure out what is causing it?
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-paas-compute-diagnostics-data. This will walk through all of the diagnostic data available as well as how to troubleshoot the most common issues.
We also had this annoying problem and in our case:
We use local storage, but it wasn't defined in service definition (or Worker Role's properties)
Our worker role project has reference to a service project which has reference to data layer project. But, the worker role project doesn't have reference to the data layer project. As soon as we added reference to data layer project in worker role project, it deploys successfully.
Problem #1 can be easily noticed if you first run the project in your local machine. Exception will be thrown.
Problem #2, however, is more difficult, mainly because it runs just fine in local machine. After 5 days of trouble shooting, we finally found the problem. So, check all references and try to add sub-reference projects, those that are referenced by other references.
We had similar problem, and it was due to some DDLs failed to load. (due to different version from the one MS have deployed to the VM)
Try to set CopyLocal to "true" for all the References in the project, and re-deploy.
I would either remote desktop to the cloud instance and review the Windows Event Logs for exceptions or redeploy with IntelliTrace Enabled. If you choose the later, you can download the IntelliTrace logs from Visual Studio and debug
http://msdn.microsoft.com/en-us/library/windowsazure/ff683671.aspx
One way to find out the actual error is to click on the " 1 instance" at the top of Dashboard after trying to deploy your web role. It will tell you the status of the role instance. The status should include more information about the type of error which blocks your deployment.
It depends on what your case is. For me, the status claimed that I had an unhandled Security exception. After some investigation, it turned out that under my role's OnStart(), I tried to create a event source. However, Azure service doesn't have the permission to create an event source.
For more possible issues, check http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx
For me, the issue was with my SQL Azure DB firewall rules. My Azure SQL Database servers are not set to "Allow Access to Azure services", so I have to explicitly list IPs that are allowed.
I discovered this after wrapping my code in a try/catch that swallowed all exceptions, refactoring my OnStart() and RunAsync() methods, and setting all my references to Copy Local = True. None of that worked, then I saw that I had this line in my RunAsync() method:
log4net.Config.XmlConfigurator.Configure();
I am using the AdoNetAdapter for log4net and connecting to an Azure SQL DB for logging, so that led me to check the firewall rules.
For me, I had some differing version of nuget packages in my various projects. Once I consolidated everything to the same version(s), it worked fine.
With the release of Windows Azure SDK version 2.2 for Visual Studio 2012 and 2013, Now you can Remote Debug Cloud Resources within Visual Studio.
Once your cloud service is published and running live in the cloud, you can simply set a breakpoint in your local source code. This may help you in digging out what's going wrong!