We are hosting Confluent Platform services(broker, zookeeper, schema-registry etc.) in Azure VMs. What is the best option to scrape metrics from these and visualize in Azure monitor or application insights?
I am thinking to scrape JMX metrics and visualize in azure monitor/app insights. Monitoring will include things like health status, message flow in topics, produce/consume rate, VM health etc.
What are some good options to implement this? Any better ideas?
PS- I only want to use Azure native services for monitoring/dashboards.
You can add java agents by exporting the environment variable KAFKA_OPTS. Through that, you can add any additional JVM monitoring
Azure docs - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent
Related
I am new to GCP and come from an Azure background. Is there an equivalent of "Azure Application Insights" on the GCP side for Monitoring Applications?
Let me explain my use case more clearly with an example: If I have a .NET based web application running on a Windows VM on GCP can Google Cloud Monitoring help detect Exceptions raised by the running application and send out alerts.
Any pointers/links to further explore this type of monitoring capability would be helpful.
Cloud Monitoring will provide you with many statisctics - most probably with what you need. And if there aren't any metrics to suit you need you may create ones based on the logs collected from the VM.
By default there is a number of logs being ingested but if you want to have full range and experiment with various ones you may want to install a monitoring agent. Go through the documentation and have a look.
You can then use the metrics to create charts and have a live view on a number of things such as cpu utilisation, disk IO/s, dropped/sent/received packets etc. Here's the Cloud Monitoring documentation.
And finally - you can create alerts based on the metrics (set thresholds, time periods etc). They can be simple e-mail alerts for example but they can be sent via pub-sub and trigger some functions or apps too.
Since you're new to GCP it's a lot of reading ahead of you but you will easily find documentation for most of GCP's services.
If you provide more details I can update my answer and give you more precise answer.
We are designing an application which will be hosted on AKS(Azure kubernetes service). The application will consist of a set of services written in asp .net core running in docker containers. I want to monitor the services as well as the containers/nodes and have the observability across the cluster. Azure monitor for containers seems to be a good solution for monitoring containers, nodes and the cluster as a whole however I want the advanced monitoring capabilities of the application insights for the asp .net core services for example application maps, live metrics streams, transaction tracing and such features. Moreover, I don't want to have overlapping solutions. Is the Azure monitor for containers able to provide all or most of these application insights features or do I have to have both solutions in order to get proper cluster monitoring and also the advanced application monitoring?
Azure Monitor for containers provide infrastructure level monitoring and basic application logs with stdout and stderr, Kubernetes events captured out of the box.
It does not provide instrumentation for your apps or distributed tracing capabilities today, which is possible with Application Insights.
If you are looking for application map & instrumentation for events metrics and logs for your app, you can use both together and it's possible to correlate data from both and create dashboards and views.
The long term road map has Azure Monitor for containers & Application insights combined offering
In this (https://learn.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview) guide, you can find all the features of AKS monitoring. If these covers all your use cases then you don't need to install any other tool. If not, then you cover only those features which are missing.
problem statement.
as per my understanding, we can run an elastic search, kibana and logstash etc as a pod in kubernates cluster for log management. but it is also memory heavy intensive application. AWS provides various managed services like Cloudwatch, cloud trail and ELK stack for log management.
do we have a similar substitute in Azure as well i.e. some managed service?
you can use AKS with Azure Monitor (reading). I'm not sure you can apply this to not AKS cluster (at least not in a straight forward fashion).
Onboarding (for AKS clusters) is really simple and can be done using various methods (portal included).
You can read more on the docs I've linked (for example, about capabilities).
Azure Monitor for Containers is available now and once integrated some cluster metrics as well as the logs will be automatically collected and made available through log analytics.
We are planning to host our NServiceBus application in Azure web and worker roles (using Azure Service Bus as transport). Existing documentation is a bit unclear when it comes to monitoring and logging of Azure based endpoints.
What would be recommended practices to set up logging and monitoring of such setup?
Would you recommend creating a custom logging module inheriting from ILog interface?
Does ServiceControl work well for monitoring endpoints in Azure? If yes, where do we deploy it?
EDIT:
Igorek's answer helped me to solve logging issue. Can anyone comment on ServiceControl part of the question?
NServiceBus can hook into and utilize log4net. log4net can send its data to Windows Event Logs thru standard log4net appender. Azure Diagnostics can move the data from Event Logs to WADLogs table for inspection by on-demand tools such as Cerebrata's Diagnostic Manager or 24x7 monitoring services such as AzureWatch. http://www.paraleap.com
Alternatively, log4net can also send the logs directly to a custom Azure Table if you want to build your own monitoring tools.
Also, consider monitoring queue/topic depths of your Service Bus, to make sure that stuff is moving thru correctly. Also, consider possibly scaling by the number of messages in SB queues or topics if your NSB setup supports dynamically adding/removing instances.
Disclaimer: I work for company that makes AzureWatch and we happen to have a number of customers who utilize AzureWatch to monitor their NSB-enabled apps.
ServiceControl can indeed be used to monitor your Azure endpoints, just set up ServiceControl for the transport you use, drop the ServiceControl plugins into your endpoint bin folder, and you're good to go.
ServiceControl is best deployed locally, or on an azure virtual machine.
There is no Management API for reading azure monitoring data. Is there any way other than getting the monitoring data directly from a VM through remote desktop connection? We basically want to display a chart for "CPU usage percentage Vs. Time”. Actually, Azure site displays monitoring data for each VM-is there any way I can read that data and display it in a report? Any PowerShell cmdlets?
Suggested good related articles:
How to monitor Cloud Services
Enabling Diagnostics in Windows Azure
If you, however refer to Windows Azure Virtual Machines then you have to organize monitoring of these a similar way you would organize a monitoring over machines in private cloud (hyper-v)
You can fetch the CPU data for cloud services (Web/workers and VMs) through the Azure Metrics API, see: https://convective.wordpress.com/2014/06/22/using-azure-monitoring-service-with-azure-virtual-machines/
You can find example code in https://github.com/WadGraphEs/AzurePlot