We have an Azure Container instance that seems to be freezing under heavy load. And we are able to cause this through a load test. I am not looking for the exact solution to that right now, but what I am confused about is that I can't seem to get any logs from the Container Instance when this happens that would tell me exactly what is going.
My instance is Docker container that runs a NodeJS application. I added Application Insights to the application, and have been successful in getting any exceptions that arise from the application itself. But when we experience the freezing behavior, it is not actually getting to the application inside the container, so Application Insights doesn't help me in this case.
Also, if I go to my Container Instance in Azure, and look under the Events tab, I am not seeing any kind of error, or anything really that would tell me my container instance is in a "not-working" state, even though we are not able to reach it.
What do you see in the "Logs" and "Connect" tab in the Azure portal?
Can you also check the overview page in the Azure portal to see the CPU/memory/network usage?
You can use the Azure CLI command az container attach to check the container instance state and also the logs. There are three ways to get the different logs, see Retrieve container logs and events in Azure Container Instances. The restart policy will also help when the container instance gets some problems on it.
Related
I've pulled in our development project and other developers have no issue.
I am running Azure Storage Emulation.
In my case, when I run it, I get a 404 error saying can't find the container.
Drilling into it, the container is 'azure-webjobs-hosts', and googling this shows this to be a standard container name, that stores webjob information.
I cannot find how this is first created though, and the code I have pulled in, which is based on a default new project, does not appear to create it.
I would like to how 'azure-webjobs-hosts' container is usually created, as I can't find anything online. Perhaps I need to install some kind of tool, library or SDK?
I would assume it is supposed to be created automatically if it is missing, but it would appear that I've missed a step somehow.
If I manually create the container, it then complains about a missing blob, and rather than try to patch this together myself I thought it would be better if I found out the root issue and resolved it.
Any suggestions?
As per my experience,
Perhaps I need to install some kind of tool, library or SDK?
Azure Web Jobs SDK isa NuGet Package and can be installed through the cmdlets given in the Official Site.
I would assume it is supposed to be created automatically if it is missing, but it would appear that I've missed a step somehow.
Yes, it would be created/added automatically as a namespace, Package References when the Azure Functions is created as this package is linked with Storage account and essential to store the data i.e., processing in the background. You can get more info about the usage of it from the GitHub Official Article of Azure Web Jobs SDK Integration.
I would like to how 'azure-webjobs-hosts' container is usually created, as I can't find anything online.
In Local System:
azure-webjobs-hosts is a folder created in the blob container locally as soon as this storage account is used by any application moves to running state.
A Folder named Locks will be created inside the azure-webjobs-hostscontainer/folder.
timers folder also created during the Timer Trigger Function run and the log files also created with the block blob type inside the locks folder and also timers folder.
Few more folder created in the blob container automatically based on type of trigger/application integrated with Azure Web Jobs SDK. Those folders are part of that local storage account, can be deleted manually and can also be recreated when the application starts running.
In Azure Portal (cloud):
When you create the Azure Function App in the Portal, Storage account is required. After Creation, Functions will be in running state so the containers such as azure-webjobs-hosts, azure-webjobs-secrets can be created that stores some data such as host.json file (that contains Authorization keys), available in azure-webjobs-secrets.
You can also host multiple function apps to the same storage account so a folder can be created and named with Function App Name inside the containers to show the logs related to that specific Application.
After publishing the local function project (.Net 6) having Http & Timer Triggers from VS2022 IDE to Azure Portal function app, below folder can be created in the associated Storage Account Container:
These are the functionalities of the azure-webjobs-hosts and Azure Web Jobs SDK in an Azure Function App and more information on its usage can be given in the above mentioned references.
I've solved this problem and it was very simple. I had break on all exceptions turned on.
For some reason they decided that even though having no container is expected the first time, they would throw an exception. This exception is rethrown a few times, and eventually handled by something that creates the container.
IMO this is bad design, considering the program expects there to be no container the first time you run it and will create it if needed, it shouldn't be an exception.
Anyway, this was the reason. Hammering F5 or setting your exception settings to default so it doesn't catch runtime errors will fix the problem.
I'm trying to deploy my solution to Azure so my website can be accessed through the Internet.
When I deploy, Azure DevOps tells me Release/Deploy succeeds, but when I open the azurewebsties.net-URL, I get a 404.
After running some diagnostics, it tells me:
Currently no worker is assigned to the app service plan. The site cannot serve any requests.
When I research on how to Add a worker I find this link
https://learn.microsoft.com/en-us/azure-stack/operator/azure-stack-app-service-add-worker-roles?view=azs-2008&tabs=azurerm
But when I try and follow it, my azure interface is too different from theirs, and things don't add up. It says:
Azure App Service on Azure Stack Hub supports free and shared worker tiers by default.
I don't see any worker tiers at all, the tutorial-mentioned tab Roles, deals with userPermissions.
JSON resource seem to show there are no workers indeed.
But diagnostics also says following information is logged by... a worker.
Though I cannot exclude problems elsewhere for a 100%, It really seems to be a worker-problem.
Build- & ReleasePipeline succeed, Artifact get uploaded & deployed, Authentication & KeyVault is good, I get Request&Response Activity in AzurePortal-Graphs. But no website.
So my questions are:
What are these workers, what do they do high-level? Operate like some kind of ServiceBus between WebApp & Azure? What language/format are they in?
How and where can I add one? Is it in code (angular/c#), JSON or in azure portal?
Where can I find up to date documentation
Anybody know how to tackle this?
Thanks in advance!
I have an Azure App Service that runs a Linux container which is pulled from a tag in an Azure Container Registry. If I enable continuous deployment in the App Service and overwrite the tag associated with it, the App Service will automatically update itself by pulling the image from the associated tag. Since this process of deployment from the tag happens automatically, I want to know when and how many times did the App Service pull from the tag and updated itself, all the way from the very first instance that it happened. The deployment and Activity Logs of the App Service does not show what I've been looking for, nor the Azure Monitor logs and metrics.
Have you looked at this article ?
https://learn.microsoft.com/en-us/azure/container-registry/container-registry-diagnostics-audit-logs
You should be able to configure diagnostic logs on the ACR and store those in Log Analytic to view the different pull counts.
How do I start a container in ACI Container Group which is already deployed and in terminated state. Can it be done either through some automation or from logic app? CLI commands show az container restart but not start. The logic app connectors seem to pull the image every time and start it.Is there no means to just start an existing terminated container?
Once a container is terminated there is not a start action you can do. You can deploy a new container which will pull the image you have setup and run whatever app is setup.
If the container is in a failed state you could issue a restart command but the results are basically the same. It re-initializes the container, pulls the image and deploys the app.
This is how containers work. They are not a persistent item such as a Virtual Machine. They are designed to be removed, added, and upgraded as needed.
This is a pain point on Azure. While container instances are not meant to be restarted after being terminated, the container groups they sit inside are meant to be flexible enough to be restarted. And they have that capability - Microsoft Azure's docs just don't really talk about it. It's left for you to figure out.
The main problem is that when you deploy an updated container instance to an existing container group in Azure, and the matching container instance in Azure is terminated, the deployment will not automatically trigger the new version of that container instance to run again.
In my case, I'm scheduling a container deployment to occur from inside an Logic App using the "Create or update a container group" connector (because Microsoft removed the cloud scheduler earlier this year, forcing people to use Logic Apps for simple tasks like this). If I delete the existing container group and run the Logic App, then it will automatically start the Docker container instance as expected. When I re-run it subsequent times, it fails to start the container, because the previous instance inside the container group is in a terminated state. The solution is to add an additional "Start containers in a container group" action, which will start the container instances anew and allow them to run to completion (at which point my containers terminate again, because they're one-off ETL jobs).
I have deployed a Web API written with .net Core to a local dev Azure Service Fabric cluster on my machine. I have plenty of disk space, memory, etc, and the app gets deployed there. However, success is intermittent. Sometimes it doesn't deploy to all the nodes, and now I have it deployed to all the nodes, but within the Azure Service Fabric Manager page, I see each application in each node has an Error status with the message: "The ServiceType was not registered within the configured timeout." I don't THINK I should have to remove and redeploy everything. Is there some way I can force it to 're-register' the installed service type? Microsoft docs are really really thin on troubleshooting these clusters.
Is there some way I can force it to 're-register' the installed service type?
On your local machine you can set the deployment to always remove the application when you're done debugging. However, if it's not completing in the first place I'm not sure if this workflow would still work.
Since we're on the topic, in the cloud I think you'd just have to use the Powershell scripts to first compare the existing app types and version and remove them before "updating". Since the orchestration of this is complicated I like to use tools to manage it.
In VSTS for instance there is an overwrite SameAppTypeAndVersion option.
And finally, if you're just tired of using the Service Fabric UI to remove the Application over and over while you troubleshoot it might be faster to use the icon in the system tray to reset the cluster to a fresh state.