I need to gather different kinds of logs:
Logs of applications running into docker machines orchestrated by Kubernetes.
Logs differents from metrics.
Which tool is more appropriate to do that?
My infrastructure is composed by Kubernetes orchestring docker machines installed in Microsoft azure (using ACS engine and RPM).
You can use Application Insights. App Insights can be used to store logging/monitoring for Docker Apps easily
See this blog for understanding how Application insights support microservices and containers.
You can easily pull the microsoft/applicationinsigthts image from docker hub and start using it.
Here the pricing calculator for application insights (Microsoft will give 5GB of Storage for logs free for every month)
Related
I am facing the following problem: I need to execute on-demand long running workers on Azure VMs. These workers are wrapped in a docker image.
So I looked at what Azure is offering and I seem to have the following two options:
Use a VM with docker-compose. This means I need to be able to programatically start a VM, run the docker image on it, and then shutdown the VM (the specs we use are quite expensive and we can't let it run indefinitely). However this means writing orchestration logic ourselves. Is there a service that maybe we could use to make life easier?
Setting up a k8s cluster. However, I am not sure how pricing works here. Would I be able to add the type of the VMs we use to the cluster, and then use the k8s API to start on-demand containers? How would I get priced in this case?
If the only thing you need are workers, there are a few more options you have. Which service suits best depends on the requirements you have. Based on what's in your question, I would think one of the following two might fit best:
Azure Container Instances
Azure Container Instances offers the fastest and simplest way to run a container in Azure, without having to manage any virtual machines and without having to adopt a higher-level service.
Azure Container Instances is a great solution for any scenario that can operate in isolated containers, including simple applications, task automation, and build jobs.
Azure Container Apps (preview)
Azure Container Apps enables you to run microservices and containerized applications on a serverless platform. Common uses of Azure Container Apps include:
Deploying API endpoints
Hosting background processing applications
Handling event-driven processing
Running microservices
According to Azure's Container services page, here are your options:
IF YOU WANT TO
USE THIS
Deploy and scale containers on managed Kubernetes
Azure Kubernetes Service (AKS)
Deploy and scale containers on managed Red Hat OpenShift
Azure Red Hat OpenShift
Build and deploy modern apps and microservices using serverless containers
Azure Container Apps
Execute event-driven, serverless code with an end-to-end development experience
Azure Functions
Run containerized web apps on Windows and Linux
Web App for Containers
Launch containers with hypervisor isolation
Azure Container Instances
Deploy and operate always-on, scalable, distributed apps
Azure Service Fabric
Build, store, secure, and replicate container images and artifacts Azure
Container Registry
EDIT:
Based on the comment
Let's say the only requirement is that I am able to use the resources on-demand, so I only end up spending the amount of money that would take for a certain job to finish execution. What would you use?
the answer would most probably be Container Apps, if the code you have available is not easily migrated to an Azure Function. The most important reason: they are Serverless, which means they scale to zero and you only pay for actual consumption. Next to that, you have to write limited to no orchestration logic, since the container apps can scale based on event sources.
Enables event-driven application architectures by supporting scale based on traffic and pulling from event sources like queues, including scale to zero.
Another great resource is Comparing Container Apps with other Azure container options.
I have an ASP.NET core 3.1 based web api ready to deploy to Azure for production use. For test / development, I have been deploying it to a traditional app service on Azure which I believe is a shared Windows VM under the hood. I have been on F1 tier and it suits my needs for test and dev.
But for production, even the cheapest plan costs me $93.44 per month which I would like to avoid if I can.
In order to lower the cost, I have decided to containerize my app and deploy it using "web app for containers" or "azure container instances". My question is, based on your experience, which method will give me reasonable production-scale performance while minimize my monthly cost? Or would containerize my app save me any money at all?
Please note that I have evaluated Azure Functions and decided it is not what I would like to use.
For your requirements, first of all, you need to know that the Azure Container Instance benefits for its quick start and running. See this:
Azure Container Instances is a great solution for any scenario that
can operate in isolated containers, including simple applications,
task automation, and build jobs.
It's good for the simple application, but not good for scenarios where you need full container orchestration, including service discovery across multiple containers, automatic scaling, and coordinated application upgrades. And I think it's also not stable for the production use, it's more appropriate for the test.
And the Azure App Service is cost according to the service plan, the service plan billed on a per second basis. You can plan the use with time as you need and the App Service has more features than Container Instance. Or if you do not satisfied with App Service, maybe you can take a look at the Azure Kubernetes Service, it has more control and feature then the Container Instance.
As of beginning of 2022 looks like Container Instances and Web Apps for containers will be the same ~32eur which is a bit better than the app service ~50eur.
I'm working in a hybrid cloud environment where we have the majority of our infrastructure in Azure, but we have some on premises machines that we also want to monitor and aggregate their log data into the same log analytics workspace.
I've successfully installed the OMS Agent for Linux and I can see that at least performance metrics and heartbeat is being collected from our local machine after following this guide. However, I can't find any guidance on how to get logs from Docker containers from on-prem to cloud.
On the on-prem machines we are running Docker, no clusters. We have some IoT Edge modules that are being managed and deployed through the IoT Hub, although I guess it is the same procedure to collect these logs since it is still Docker.
How should I got about getting my Docker logs to log analytics using the OMS agent? Is the OMS agent what I need?
It was my own error. When I reinstalled the agent I could see in the logs that the Docker version was too old for the agent to handle, so it was skipped. Updated Docker and it works.
I have some experience in using azure app services without docker. I did worked on k8 with docker. Now I am seeing an option to deploy containers in azure app services. As per my understanding app services internally use container and orchestration engine. Now someone can explain why this new docker option in azure app services? How it is going to help? When to use this option? Why I should bundle it as docker(extra effort eventhough it trivial)?
Azure App Service on Linux (Web App with built-in images)
The built-in image option running on Linux is an extension of a current Azure App Service offering, catering to developers who want to use FTP or GIT, deploy .NET Core, Node, PHP or Ruby applications to Azure App Service running on Linux
All of the built-in Docker images are open sourced on GitHub and available on DockerHub
Now someone can explain why this new docker option in the azure app
services?
Web App for Containers is catered more towards developers who want to have more control over, not just the code, but also the different packages, runtime framework, tooling etc. that are installed on their containers.
Customers of this offering prefer to package their code and dependencies into containers using various CI/CD systems like Jenkins, Maven, Travis CI or VSTS, alongside setting up continuous deployment webhooks with App Service.
This way you can easily deploy and run containerized applications that scale with your business.
How it is going to help?
This will make sure that the environment that you use locally, is
exactly the same as the one in the cloud.
Just pull container images from Docker Hub or a private Azure
Container Registry and Web App for Containers will deploy the containerized app with your preferred dependencies to production in seconds.
Automate and simplify your container image deployments through
continuous integration/continuous deployment (CI/CD) capabilities
with Docker Hub, Azure Container Registry, and Visual Studio Team
Services
Automatically scale vertically and horizontally based on application
needs. Granular scaling rules are available to handle peaks in
workload automatically while minimizing costs during off-peak times
When to use this option?
If you are so passionate/familiar with Docker/container then you can
use the Azure App service with the container.
If you are planning to host all your container in ACS(Azure Container
Service)/GitHub Repository then this service might be useful
You can refer to this blog for more details
I am beginner in elastic search.I have implemented elastic search for single node with NEST Client in my C# Web Application successfully.Now I want to scale my data horizontally and want to deploy this application on Azure cloud storage.so if any article or documentation(step by step) is available for how to effectively deploy Elasticsearch with proper configuration on Azure with multi node then inform me.
Any suggestion is appreciated.
Take a look at Elastic's Azure Marketplace offering. Within a few screens and minutes, you'll have a cluster deployed on Azure.
If you need more control over the deployed cluster such as
number and size of disks to attach to each data node
install Azure repository plugin for snapshot/restore
Java heap size for each node
take a look at the ARM template on GitHub. This is the same underlying template that powers the Marketplace offering, but deploying directly provides more features than are exposed in the Marketplace UI.