We have a background worker role that runs a few services for us. A communication service and a notification service. when i run my code locally both these services run and Im able to write to a log and all seems fine. When I point my config files at production connections strings and other production variables and run my local code against it - it all works as expected. When I push the code to Azure my service doesnt seem to run, I no longer see any data being written to my log table.
We have another solution where we do the same thing (communication and notification service) and this works as expected in this other solution.
I have compared all references in our working solution to those references in our non-working solution - all needed dlls have been set to copyLocal=true - We've tried a variety of different tools to see what the issue is but we have not been able to pin point the exact issue.
EDITED:
This is a simple process in the background
I write to the log using a built logging utility - this log table exists on our sql azure database
We verify that this is working - on account that the service spins up with the azure utility and eventually says 'Ready' - in past occurances - we'd get the error messages or it would just continue to spin.
Our service is started up by the WorkerRole.cs file - Run / OnStart.
Related
This is my first nestjs application (or any nodejs application for that matter) and I'm having trouble deploying it to a "production" environment. Steps I've followed:
1) I installed the Azure extension for VS Code
2) I click the blue up arrow icon in VS Code to initiate the deployment
3) The first time I tried, I had manually created my node JS application from portal.azure.com and chose that app service from the list, it didn't work
4) the second time I tried it, I created a new app service from the deployment process in VS Code
5) The application deploys and I get a deployment successful message. If I expand deployments under my new app service in the azure extension in VS Code, I see the deployment and when I select that, I get a log that finishes with this screenshot:
6) I try hitting an endpoint on my nestJS api from postman and I get an application error message that has a link to https://mywebsite.azurewebsites.net/detectors to troubleshoot. When I click that link, it fails to load in the azure portal
7) I read somewhere that I need to include my nodeJS version on the app service so I tried adding that - see screenshot below:
8) I can see the files if I use the SSH tool from the azure portal
a couple things to mention, I've read a few things that suggest I need to do something with "tsc"? in my package.json file. Since this is my first time doing anything with nestjs/nodejs, I have no clue what that means. I have not modified my package.json file (at least the scripts section) at all from how it comes out the box. Is there something I need to adjust there? Is there something I need to change on azure? I'm really liking nestjs a whole lot, but getting it to work in on my "real" server is proving to be a challenge...
any help is greatly appreciated.
TIA
We have set up Application Insights for our Dev & Prod environments, not with the SDK but through the Portal. We're now in the process of installing the SDK so we can have more control over customizing logging, what to measure in performance, etc.
I have found how to separate the environments in code (separate Instrumentation Key in different config files etc), but I have found nothing when it comes to my local environment. Which instrumentation key should I use there, the dev one? Wouldn't then this skew our dev metrics everytime one of the developers runs the app locally?
And also, doesn't it make sense to have a separate App Service slot just for the local environment, so I can test everything and see the logs I'm trying out locally, and not have to deploy to dev everytime I want to see what I'm doing?
I've tried creating a separate slot for local, but it generates a weird url based on the name I give, which I can't change later.
I've googled for a couple of days already and couldn't find any (or very little) helpful advice when it comes to this.
I realise there is a "Just add the SDK to try local only mode" option in Visual Studio, but then I would have to use it exclusively locally. What I want is to use all three - my local, dev & prod.
We're using .net core 2.2 for our backend and Angular 7 for front end.
I'm an idiot.
The url set up automatically basically means nothing. I solved the problem by just adding another App Service slot (created from our App Service production one, just like dev), and added this key to the local settings in our project.
Now we can use this key to get real time results as we debug, and use the other two for dev & production.
I'm running an Azure function in Azure, the function gets triggered by a file being uploaded to blob storage container. The function detects the new blob (file) but then outputs the following message - Did not find any initialized language workers.
Setup:
Azure function using Python 3.6.8
Running on linux machine
Built and deployed using azure devops (for ci/cd capability)
Blob Trigger Function
I have run the code locally using the same blob storage container, the same configuration values and the local instance of the azure function works as expected.
The functions core purpose is to read in the .xml file uploaded into blob storage container and parse and transform the data in the xml to be stored as Json in cosmos db.
I expect the process to complete like on my local instance with my documents in cosmos db, but it looks like the function doesn't actually get to process anything due to the following error:
Did not find any initialized language workers
Troy Witthoeft's answer was almost certainly the right one at the time the question was asked, but this error message is very general. I've had this error recently on runtime 3.0.14287.0. I saw the error on many attempted invocations over about 1 hour, but before and after that everything worked fine with no intervention.
I worked with an Azure support engineer who gave some pointers that could be generally useful:
Python versions: if you have function runtime version ~3 set under the Configuration blade, then the platform may choose any of python versions 3.6, 3.7, or 3.8 to run your code. So you should test your code against all three of these versions. Or, as per that link's suggestion, create the function app using the --runtime-version switch to specify a specific python version.
Consumption plans: this error may be related to a consumption-priced app having idled off and taking a little longer to warm back up again. This depends, of course, on the usage pattern of the app. (I infer (but the Engineer didn't say this) that perhaps if the Azure datacenter my app is in happens to be quite busy when my app wants to restart, it might just have to wait for some resources to become available.). You could address this either by paying for an always-on function app, or by rigging some kind of heartbeat process to stop the app idling for too long. (Easiest with a HTTP trigger: probably just ping it?)
The Engineer was able to see a lower-level error message generated by the Azure platform, that wasn't available to me in Application Insights: ARM authentication token validation failed. This was raised in Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication.ArmAuthenticationHandler.HandleAuthenticate() at /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/Authentication/Arm/ArmAuthenticationHandler.cs. There was a long stack trace with innermost exception being: System.Security.Cryptography.CryptographicException : Padding is invalid and cannot be removed.. Neither of us were able to make complete sense of this and I'm not clear whether the responsibility for this error lies within the HandleAuthenticate() call, or outside (invalid input token from... where?).
The last of these points may be some obscure bug within the Azure Functions Host codebase, or some other platform problem, or totally misleading and unrelated.
Same error but different technology, environment, and root cause.
Technology Net 5, target system windows. In my case, I was using dependency injection to add a few services, I was getting one parameter from the environment variables inside the .ConfigureServices() section, but when I deployed I forget to add the variable to the application settings in azure, because of that I was getting this weird error.
This is due to SDK version, I would suggest to deploy fresh function App in Azure and deploy your code there. 2 things to check :
Make sure your local function app SDK version matches with Azure function app.
Check python version both side.
This error is most likely github issue #4384. This bug was identified, and a fix was released mid-june 2020. Apps running on version 3.0.14063 or greater should be fine. List of versions is here.
You can use azure application insights to check your version. KUSTO Query the logs. The exception table, azure SDK column has your version.
If you are on the dedicated App Service plan, you may be able to "pull" the latest version from Microsoft by deleting and redeploying your app. If you are on consumption plan, then you may need to wait for this bugfix to rollout to all servers.
Took me a while to find the cause as well, but it was related to me installing a version of protobuf explicitly which conflicted with what was used by Azure Functions. Fair, there was a warning about that in the docs. How I found it: went to <your app name>.scm.azurewebsites.net/api/logstream and looked for any errors I could find.
I am trying to deploy a Cloud Service with 1 Web Role to Azure.
When I do so, I get this message:
Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
The project runs just fine locally, and I'm having a hard time figuring out how to start debugging this issue. Are there any common problems that cause this message or steps to figure out what is causing it?
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-paas-compute-diagnostics-data. This will walk through all of the diagnostic data available as well as how to troubleshoot the most common issues.
We also had this annoying problem and in our case:
We use local storage, but it wasn't defined in service definition (or Worker Role's properties)
Our worker role project has reference to a service project which has reference to data layer project. But, the worker role project doesn't have reference to the data layer project. As soon as we added reference to data layer project in worker role project, it deploys successfully.
Problem #1 can be easily noticed if you first run the project in your local machine. Exception will be thrown.
Problem #2, however, is more difficult, mainly because it runs just fine in local machine. After 5 days of trouble shooting, we finally found the problem. So, check all references and try to add sub-reference projects, those that are referenced by other references.
We had similar problem, and it was due to some DDLs failed to load. (due to different version from the one MS have deployed to the VM)
Try to set CopyLocal to "true" for all the References in the project, and re-deploy.
I would either remote desktop to the cloud instance and review the Windows Event Logs for exceptions or redeploy with IntelliTrace Enabled. If you choose the later, you can download the IntelliTrace logs from Visual Studio and debug
http://msdn.microsoft.com/en-us/library/windowsazure/ff683671.aspx
One way to find out the actual error is to click on the " 1 instance" at the top of Dashboard after trying to deploy your web role. It will tell you the status of the role instance. The status should include more information about the type of error which blocks your deployment.
It depends on what your case is. For me, the status claimed that I had an unhandled Security exception. After some investigation, it turned out that under my role's OnStart(), I tried to create a event source. However, Azure service doesn't have the permission to create an event source.
For more possible issues, check http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx
For me, the issue was with my SQL Azure DB firewall rules. My Azure SQL Database servers are not set to "Allow Access to Azure services", so I have to explicitly list IPs that are allowed.
I discovered this after wrapping my code in a try/catch that swallowed all exceptions, refactoring my OnStart() and RunAsync() methods, and setting all my references to Copy Local = True. None of that worked, then I saw that I had this line in my RunAsync() method:
log4net.Config.XmlConfigurator.Configure();
I am using the AdoNetAdapter for log4net and connecting to an Azure SQL DB for logging, so that led me to check the firewall rules.
For me, I had some differing version of nuget packages in my various projects. Once I consolidated everything to the same version(s), it worked fine.
With the release of Windows Azure SDK version 2.2 for Visual Studio 2012 and 2013, Now you can Remote Debug Cloud Resources within Visual Studio.
Once your cloud service is published and running live in the cloud, you can simply set a breakpoint in your local source code. This may help you in digging out what's going wrong!
I have a Worker Role which isn´t working on Cloud environment.
When I run it locally, it runs perfectly, but, when I deploy it to Azure I´m having some troubles. The deploy, itself, occurs seamlessy, after the VM starts, my app doesn´t run. There is nothing on Event Log, and, even after I setted up the app to flush all Trace Messages to Azure Table, nothing is wrote there too.
How can I check if my app is really running on the VM? Why my app isn´t working there as it works locally?
Have tried to implement diagnostics on your webrole? This is the best way to find any errors in your code. An other solution is to install sysinternals during startup. Patriek van Dorp has made a nu get package thad adds the sysinternal suite as a plugin for your cloud project.
The best way is to enable RDP and remote into the machine. Then you can look at the processes running and ensure that things are running as you expect. It is odd that there is nothing in the event log if it is failing to run. Does the portal show the deployment as Ready?
Maybe too late, but i had similar issue.When i run it locally, it was running.After deploy did not run anything.Problem was that, i deploying into website,where worker not available.You should deploy into CloudService(there are available both roles) or make Sheduler, which will do request on your page, where process job you needed.In your custom intreval, ofcourse.
BTW sorry about my english ...
Regards