I have an AKS cluster running on which I enabled Container Insights.
The Log Analytics workspace has a decent amount of logs in there.
Now I do have my applications running on a separate namespace, and one namespace which has some Grafana containers running (which I also don't want in my captured logs).
So, I searched on how I could reduce the amount of captured logs and came across this Microsoft docs article.
I deployed the template ConfigMap to my cluster and for [log_collection_settings.stdout] and [log_collection_settings.stderr] I excluded the namespaces which I don't want to capture.
When calling kubectl edit configmap container-azm-ms-agentconfig -n kube-system I get the following:
Which means that my config is actually in there.
Now when I open a query window in Log Analytics workspace and execute the following query:
KubePodInventory
| where Namespace == "kube-system"
I get plenty of results with a TimeGenerated column that contains values that are like 5 minutes ago, while I setup the ConfigMap a week ago.
In the logs of one of the pods omsagent-... I see logs like the following:
Both stdout & stderr log collection are turned off for namespaces: '*.csv2,*_kube-system_*.log,*_grafana-namespace_*.log'
****************End Config Processing********************
****************Start Config Processing********************
config::configmap container-azm-ms-agentconfig for agent settings mounted, parsing values
config::Successfully parsed mounted config map
While looking here at StackOverflow, I found the following answers which make me believe that this is the right thing that I did:
https://stackoverflow.com/a/63838009
https://stackoverflow.com/a/63058387
https://stackoverflow.com/a/72288551
So, not sure what I am doing wrong here. Anyone an idea?
Since I hate it myself that some people don't post an answer even if they already have one, here it is (although not the answer you want, at least for now).
I posted the issue on GitHub where the repository is maintained for Container Insights.
The issue can be seen here on GitHub.
If you don't want to click the link, here is the answer from Microsoft:
We are working on adding support for namespace filtering for inventory and perf metrics tables and will update you as soon this feature available.
So, currently we are not able to exclude more data than the ContainerLog table with this ConfigMap.
Related
I'm deploying a container to a Container Optimized OS or COS on Google Compute.
I want to specify Logging and Monitoring for the VM. There are 2 ways to do this:
Specify metadata flags:
Mark the checkboxes
But when I then click on "Equivalent command line", there's no indication of these options.
Am I just misinterpreting something here or am I not allowed to specify these flags in the command?
I tried with the non-COS VM instance and the expected metadata flag showed up to indicate the metadata. But this does not show up in the COS command.
gcloud compute instances create instance-1 \
...
--metadata=MY_TEST_FLAG=test_value
Yes. When using container optimized OS images while creating a VM this issue is coming, But this is for command line code only. REST equivalent is generated properly, As a work around for this you can add the metadata flag to the generated command as mentioned below.
--metadata=google-logging-enabled=true,google-monitoring-enabled=true
I have raised a request on this issue. Please monitor the Google Public Issue Tracker for further updates on the fix of the issue.
In case you find any such issues in future you can report to Google using Report issues and request features with issue trackers.
I have deployed a pod in kubernetes cluster that run a python script.
The problem is i want to force the k8s to stop the container after the script complete his job and not to re-create another pod.
To be aware that i have tried to use kind:job but it doesn't fulfill my need.
I tried two types of kind, job and deployments.
With the deployment the pod always show status first completed after that crush with crashloopbackof error.
With the job the pod always show the status completed but i don't have the possibility to re-excute it with an automated way
Do you have any suggestions about that?
I have posted community wiki answer to summarise the topic.
User Jonas has posted great suggestions:
A kind Job does exactly this. Use Job and your problem is solved.
If you deploy with kubectl create -f job.yaml and your job has a generateName: instead of name:, a new Job will be created each time.
For more information look at the documentation about Jobs. See also information about Gnerated values.
Well, for the last 2 days I battled this documentation:
https://learn.microsoft.com/en-au/azure/aks/static-ip
and
https://learn.microsoft.com/en-au/azure/aks/ingress-own-tls
First of all I ensured that I had my aks k8s cluster upgraded to 1.11.5, so there is no question about having the static IP in a different resource group.
Overall, I could not get the static IP really working. With dynamic everything sounds fine, but I cannot add a A record for a dynamic IP.
I managed to deploy everything successfully, but any curl ip.. does not work. I did run exec -ti locally, and locally everything is fine.
Could someone please point me to a GitHub config or article that has this configuration running? As a disclaimer I know azure very well, so well the service principal assignments are well done, etc. However, I am new, only a few months on k8s.
Thanks in advance for any suggestion.
I can share logs if needed but believe I did check everything from dns to ingress routes. I am worried that this doc is not good and I am just loosing my time.
Answering myself this question, after quite a journey, for when I get older and I forget what I've done, and maybe my nephew will save some hours someday.
First, it's important:
In the values provided to nginx-ingress chart template, there are 2 annotations that are important:
service.beta.kubernetes.io/azure-load-balancer-resource-group: "your IP's resource group"
externalTrafficPolicy: "Local"
Here are all the values documented: https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml
The chart can be deployed near your service's namespace, it should not be in kube-system (with my current knowledge I don't find a reason to have it in system).
Second, could be misleading
There is a delay of ~30+ seconds (in my case) from the moment when IP appeared in the kubectl get services --watch and till the moment curl -i IP was able to answer the call. So, if you have automation or health probes then ensure that you have 1 - 2 mins added to wait. Or maybe take better nodes, bare metal machines.
Look at GCE and DO for the same setup as might help:
https://cloud.google.com/community/tutorials/nginx-ingress-gke
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
The guys at DO, are good writes as well.
Good luck!
Based on your comments, it seems that you are trying to override the externalIPs but use the default value of the helm chart for controller.service.type which is LoadBalancer. What you might want to do is to keep controller.service.type to LoadBalancer and set controller.service.loadBalancerIP with your static IP instead of overriding externalIPs.
Here some documentation from microsoft.
How this can be achieve? I have a catalina.out log in a prod server which is growing fast in space. 6.7 GB in couple of days . I had the Idea at the begging to create a cronjob to be executed 2 or 3 days a week to run a script that copy catalina log to Azure blob storage and then wipe it out with just a command "echo "" > file". But moving 2 GB to azure every day that cron job executes don´t know if is the best idea either. way too big file.
Is there a way that the logs is in another server/azure storage? Where should I configuer that?
I read something about implementing log4j with tomcat, is this possible also? that catalina.out using log4j move it to other server? Howcan I achieve this?. I know that development team should check also why is growing and logging so fast this file, but in the meantime I need a solution to implement.
thanks!!
I read something about implementing log4j with tomcat, is this
possible also?
I think what you want to describe is Log Rotation, if you want to use this way, here is a blog about how to configure it.
I had the Idea at the begging to create a cronjob to be executed 2 or
3 days a week to run a script that copy catalina log to Azure blob
storage
Yes, you could choose this way to manage log, however I still have something to say. If you want to upload the log file to Azure Blob, I think you may get error for the large file . You need split large file into multiple small file. In this article, under the title Upload a file in blocks programmatically, there is detailed description.
From you description, you are not using Azure Web, so if you choose Azure Web , you could also use Azure Functions or WebJobs to do the cronjob.
If you still have other questions, please let me know.
I'm learning containers and how they work and started learning Kubernetes. I was wondering if there is a way I can pull or extract data from a Kubernetes cluster? What I want to do is to pull data from a cluster and monitor the information that is coming from it. I googled some and came up with some clients that work on nodejs and just wanted to know if I'm heading in the right direction.
what I'm trying to do is pull data from a cluster and put it on a front end website for me to see. (Thinking of making a compare CPU usage of different clusters site)
the link in question: https://github.com/tenxcloud/node-kubernetes-client
Look into heapster (https://github.com/kubernetes/heapster) with InfluxDB and Grafana as backends. Grafana is great for the type of monitoring and comparison that you seem to be looking for. This is sometimes included. With AKS in Azure heapster is there out of the box, but InfluxDB and Grafana must be added. I think that the entire stack is present in Google's offering.