I have multiple services on GCP app engine that communicate to each other using google tasks. And one of the queues I have this error when I tried to create a new task:
This only appears when I left the services unused for a few hours but the problem disappear when I re-deploy the app engine service witch creates the task. And then, the problem comes back again.
If I tried to create a new task using the gcloud command line tool it works. No connection error.
I have two queues with the same setup and only this one have this problem.
Does anyone have the same problem?
Thanks in advance for your help!
Seems like a general issues relating to Google Client Libraries - github issue
Related
This is my first nestjs application (or any nodejs application for that matter) and I'm having trouble deploying it to a "production" environment. Steps I've followed:
1) I installed the Azure extension for VS Code
2) I click the blue up arrow icon in VS Code to initiate the deployment
3) The first time I tried, I had manually created my node JS application from portal.azure.com and chose that app service from the list, it didn't work
4) the second time I tried it, I created a new app service from the deployment process in VS Code
5) The application deploys and I get a deployment successful message. If I expand deployments under my new app service in the azure extension in VS Code, I see the deployment and when I select that, I get a log that finishes with this screenshot:
6) I try hitting an endpoint on my nestJS api from postman and I get an application error message that has a link to https://mywebsite.azurewebsites.net/detectors to troubleshoot. When I click that link, it fails to load in the azure portal
7) I read somewhere that I need to include my nodeJS version on the app service so I tried adding that - see screenshot below:
8) I can see the files if I use the SSH tool from the azure portal
a couple things to mention, I've read a few things that suggest I need to do something with "tsc"? in my package.json file. Since this is my first time doing anything with nestjs/nodejs, I have no clue what that means. I have not modified my package.json file (at least the scripts section) at all from how it comes out the box. Is there something I need to adjust there? Is there something I need to change on azure? I'm really liking nestjs a whole lot, but getting it to work in on my "real" server is proving to be a challenge...
any help is greatly appreciated.
TIA
I'm running an Azure function in Azure, the function gets triggered by a file being uploaded to blob storage container. The function detects the new blob (file) but then outputs the following message - Did not find any initialized language workers.
Setup:
Azure function using Python 3.6.8
Running on linux machine
Built and deployed using azure devops (for ci/cd capability)
Blob Trigger Function
I have run the code locally using the same blob storage container, the same configuration values and the local instance of the azure function works as expected.
The functions core purpose is to read in the .xml file uploaded into blob storage container and parse and transform the data in the xml to be stored as Json in cosmos db.
I expect the process to complete like on my local instance with my documents in cosmos db, but it looks like the function doesn't actually get to process anything due to the following error:
Did not find any initialized language workers
Troy Witthoeft's answer was almost certainly the right one at the time the question was asked, but this error message is very general. I've had this error recently on runtime 3.0.14287.0. I saw the error on many attempted invocations over about 1 hour, but before and after that everything worked fine with no intervention.
I worked with an Azure support engineer who gave some pointers that could be generally useful:
Python versions: if you have function runtime version ~3 set under the Configuration blade, then the platform may choose any of python versions 3.6, 3.7, or 3.8 to run your code. So you should test your code against all three of these versions. Or, as per that link's suggestion, create the function app using the --runtime-version switch to specify a specific python version.
Consumption plans: this error may be related to a consumption-priced app having idled off and taking a little longer to warm back up again. This depends, of course, on the usage pattern of the app. (I infer (but the Engineer didn't say this) that perhaps if the Azure datacenter my app is in happens to be quite busy when my app wants to restart, it might just have to wait for some resources to become available.). You could address this either by paying for an always-on function app, or by rigging some kind of heartbeat process to stop the app idling for too long. (Easiest with a HTTP trigger: probably just ping it?)
The Engineer was able to see a lower-level error message generated by the Azure platform, that wasn't available to me in Application Insights: ARM authentication token validation failed. This was raised in Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication.ArmAuthenticationHandler.HandleAuthenticate() at /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/Authentication/Arm/ArmAuthenticationHandler.cs. There was a long stack trace with innermost exception being: System.Security.Cryptography.CryptographicException : Padding is invalid and cannot be removed.. Neither of us were able to make complete sense of this and I'm not clear whether the responsibility for this error lies within the HandleAuthenticate() call, or outside (invalid input token from... where?).
The last of these points may be some obscure bug within the Azure Functions Host codebase, or some other platform problem, or totally misleading and unrelated.
Same error but different technology, environment, and root cause.
Technology Net 5, target system windows. In my case, I was using dependency injection to add a few services, I was getting one parameter from the environment variables inside the .ConfigureServices() section, but when I deployed I forget to add the variable to the application settings in azure, because of that I was getting this weird error.
This is due to SDK version, I would suggest to deploy fresh function App in Azure and deploy your code there. 2 things to check :
Make sure your local function app SDK version matches with Azure function app.
Check python version both side.
This error is most likely github issue #4384. This bug was identified, and a fix was released mid-june 2020. Apps running on version 3.0.14063 or greater should be fine. List of versions is here.
You can use azure application insights to check your version. KUSTO Query the logs. The exception table, azure SDK column has your version.
If you are on the dedicated App Service plan, you may be able to "pull" the latest version from Microsoft by deleting and redeploying your app. If you are on consumption plan, then you may need to wait for this bugfix to rollout to all servers.
Took me a while to find the cause as well, but it was related to me installing a version of protobuf explicitly which conflicted with what was used by Azure Functions. Fair, there was a warning about that in the docs. How I found it: went to <your app name>.scm.azurewebsites.net/api/logstream and looked for any errors I could find.
I'm writing this simple answer to help others that would otherwise have had to spend hours getting confused by every tutorial out there.
My issue:
Newbie to Azure web apps and have been trying to learn how to deploy via GitHub. Every tutorial and video I checked out showed that as soon as you create a Web App then go to the Deployment options screen it lets you choose how you want to deploy it.
However I was not seeing this in mine - it was as if it was pre-configured for me but there was no ftp or GitHub option showing.
Solution:
The problem may be because I had used the node.js empty template to create my web app and it preconfigured something for me (though I don't see why this should be the case); but in any case I went to the Deployment options screen, pressed the Disconnect button. Waited then hit settings, and hey presto I finally got to where all the tutorials were talking about.
I am trying to deploy an application to the Azure Service Fabric using the release definition task. When it gets to the task to deploy the server is returning the following error:
The type initializer for 'Microsoft.ServiceFabric.Powershell.Constants' threw an exception
I checked the Endpoint configuration and it appears to be set up as it is supposed to be:
No Authentication (this is an internal text box)
Cluster endpoint: tcp://[service fabric server]:19000
It downloads the artifacts without a problem, but in deploy it searches for the paths for publish profile and application package and finds them. After it finds them it throws the error. I have tried replacing TCP in the endpoint with http, added and removed the :19000 as well and all I get is this error. I have been searching online with little success. Any help to this end is much appreciated.
John
After lots of researching trying every suggestion I could find, we decided try and connect to the machine via Powershell on the box and it too was returning this error. So we uninstalled the SDK and re-installed it and the connection could be made and the builds started to work. I don't know exactly why it failed, but apparently a re-install did the trick. It may have been a bad install, or it could have been a versioning problem. Either way, try a re-install first.
I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.
Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".
I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.
After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)
In case of missing Reference you can troubleshoot this issue by:
Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.
IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.
Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)
Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:
Debugging Azure Role Recycles
Enable Remote Access to your role - RDP login
Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
Review the Azure text file logs across both C:\logs and c:\resources
Review custom logs in the Volume E: or F: for any custom role startup logging
Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.
Installing Azure Debugging Tools via Powershell
PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe
If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!
WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)
The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.
If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.
From Avkash above:
Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.
Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.
Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.
If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.