There are many docker containers which are running on a given server. I have configured each application instance running inside the container to send the logs to the standard output. Now, I am interested in the following:
Manage the logs coming out of all these containers.
Store all of these logs and also rotate them as needed.
Only store the logs locally on the server or on a shared location but NOT sending them to any cloud/log management service.
Probably coming up with a container which can take up the task of log management independently.
Any suggestions on how I can achieve this?
You can create data container which will gather all logs from all of your containers.
Then you can process those logs for example with ELK stack (Elastic search, logstash, kibana) to aggregate and transoform and visualise your logs for your need.
also rotate them as needed.
Docker will have logrotate in a future version, but you can already use logrotate for docker container http://blog.sandro-keil.de/2015/03/11/logrotate-for-docker-container/
Related
I am running a couple of VM's inside Azure portal and I have my own private besu nodes running on them. I have my metrics set up inside the Prometheus but I was hoping to hook it up securely to Grafana but I tried everything and I can't. So the next thing is to see can I get the metrics available through prometheus into azure monitor, specifically into log analytics?
the aim is to get the sync status, and the highest block number on each node, into log analytics so we can see what each is doing. That way we know, on a quick look, the status of each node and by extension, the condition of the private chain. What worries me is that although I have alerts if blocks stop being created or nodes lose peers we cannot see it quickly.
Prometheus is one option to give us those stats. If we can get data from prometheus into log anaytics that would solve the problem.
Can anyone help me to how I can go about it or any links. All I am seeing is for containers but I want for my VM's
I am experiencing a sporadic issue running containers on ACI that seems to cause Azure to "lose track" of my container instance and result in an orphaned container. My containers always run successfully, but every now and then I get this weird issue. Some peculiarities:
the container instance will still succeed internally (the code in it runs successfully), and the parent container group even says "Succeeded", but Azure never tells me the container instance itself has been created. It just says "Started". Typically the events you see are Pulling-->Pulled-->Created-->Started. Why is "Created" missing?
I can't view logs of the container without hooking up Azure Log Analytics. The "Logs" tab on the container blade in the Azure portal just says No logs available. Normally you can see the logs of a successful container
in cases of this issue occurring, it tries to pull the image twice (and appears to succeed twice - see image below).
sometimes there will be a 4th event displayed in the portal, "Killed"
I am creating a single-container container group via Logic Apps' Azure Container Instance connector - I do this reliably for many automated workflows. The logic app monitors the container group's state, and pulls the instance's logs and then deletes the group when done. All of my images are hosted on Azure Container Registry. The Python code inside the container pulls data from SQL, generates a PDF report, and posts it on an Azure Blob. I know the code is running/succeeding because I can see the report being posted! I have also hooked up Log Analytics to the container, so I can see my internal python logging. There are NO other errors I see reported by Log Analytics. I get a failure in the logic app though when I try to pull container logs and it can't find them (see bullet point 2 above).
Here's output from log analytics on container events (a more detailed version of above screenshot) - so bizarre that the container REPULLS 10 seconds after the first one successfully pulled. You can then see my first container actually runs successfully and exits with 0, and we then have this orphan container left over that is killed.
I have noticed one thing VERY consistent when this issue occurs. Typically when I look at a successful container creation event in Azure, the event message specifies that it is pulling my image via its tag: myregistry.azurecr.io/riptuskimage:1.2.5. When this issue occurs, the event messages specifies that the image is being pulled by its digest instead: myregistry.azurecr.io/riptuskimage#shah256:d98fja.... EVERY time the issue has occurred, I've noticed this. I have no idea why Azure is doing this. I most certainly specify the tag in my creation request.
I have viewed this post and this post and neither really help.
I've been scratching my head for a while on this one. The fact that it's sporadic (doesn't always happen), and when it does the images pull twice gives me the suspicion it has something to do with my container registry. The image I'm pulling is large - about 1.6GB. I checked the container registry's throttle limits and I don't think a single pull of a 1.6GB image should end up throttling - but the ACI container creation doesn't really give me a way to see if the registry is returning a 429 HTTP error. I'm not pulling anything else at that time.
Anyone have any ideas? Thanks!
Edit: This is a recent phenomenon! I have logic apps in place that have been creating containers for over a year, and this issue only starting occurring in the last few weeks (as of this posting 9/24/2021)
When your container is not working properly in Azure Container Instances, start by viewing its logs with az container logs, and stream its standard out and standard error with az container attach.
The az container attach command provides diagnostic information during container startup.
Also view the diagnostic information provided by the Azure Container Instances resource provider. To view the events for your container, run the az container show command.
With this your first problem will be solved. Check this document for more information.
Azure is pulling the container image twice from the Azure Container Registry because the container is taking a long time to start. Because Azure Container Instances pulls your container image on demand, the startup time you see is directly related to its size. One factor that contributes to container startup time in Azure Container Instances is Image Size.
Check this document for more information.
You can solve this problem by adding a delay after pulling your image from the registry in your logic app.
I have an AKS cluster on Dev Env that has some containers running on it. I have also enabled the Azure Log Analytics on it. But I can see the logs of the current container that are running not the old ones which has been killed or stopped.
I'm wondering how I can keep the logs permanently even the container that has been deleted ot stopped. What is the best solution for it? I hoped that Azure logs can do it but it seems it doesn't.
Your best option is to use Azure Monitor for Containers (can be enabled post-deployment of the cluster): https://learn.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview
It collects logs from pods and stores them in a Log Analytics workspace, where they can be queried using the Kusto Query Language.
problem statement.
as per my understanding, we can run an elastic search, kibana and logstash etc as a pod in kubernates cluster for log management. but it is also memory heavy intensive application. AWS provides various managed services like Cloudwatch, cloud trail and ELK stack for log management.
do we have a similar substitute in Azure as well i.e. some managed service?
you can use AKS with Azure Monitor (reading). I'm not sure you can apply this to not AKS cluster (at least not in a straight forward fashion).
Onboarding (for AKS clusters) is really simple and can be done using various methods (portal included).
You can read more on the docs I've linked (for example, about capabilities).
Azure Monitor for Containers is available now and once integrated some cluster metrics as well as the logs will be automatically collected and made available through log analytics.
Does Azure Container Service integrate with Azure Monitor?
Wondering what the best way is to do logging/monitoring of kubernetes cluster?
If you are looking for monitoring tools on Azure, you may want to use Azure OMS (Opertation Management Suite). This gives you the ability to monitor the container inventory, performance, and logs in a single location. To my understanding, the stats of the container is only available for Linux nodes now, if you are deploying your k8s cluster on the Azure Portal.
To do this, you need to first create an OMS account. By this time, you should have the Workspace ID and the key available. The next step would
be to create the oms pod on each node using a DaemonSet.
For the detailed setup, take a look at https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-oms.
For third party tools, Grafana+influxdb is one of the ways I have tried before. Basically it provides you with the metrics on two levels: POD and NODE respectively. The displayed metrics included CPU Usage, Memory Usage, Network Usage and Filesystem Usage, etc. Of course, you can always alter your query to add extra metrics.
For the implementation of this approach, you can refer to https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/monitoring.md.
Hope this helps :)
you can use this CLI command to browse through kubernetes cluster deployed using azure container service.
az acs kubernetes browse -g -n
This way you can see kubernetes webui
also you can use kubectl proxy command.