I have a spark cluster running on HDInsights. Ambari shows some general metrics in its dashboard, such as Namenode uptime. Where/How may I find the raw data related to such metric?
Thanks
You should "Enable HDInsight Azure Monitor logs integration" to find the raw data related to any metrics that are available in Ambari UI.
Using Ambari Dashboard:
Ambari dashboard, which contains widgets that show a handful of metrics to give you a quick overview of your HDInsight cluster's health. These widgets show metrics such as the number of live DataNodes (worker nodes) and JournalNodes (zookeeper node), NameNodes (head nodes) uptime, as well metrics specific to certain cluster types, like YARN ResourceManager uptime for Spark and Hadoop clusters.
Using Azure Monitor:
Azure Monitor logs enable data generated by multiple resources, such as HDInsight clusters, to be collected and aggregated in one place to achieve a unified monitoring experience.
As a prerequisite, you'll need a Log Analytics Workspace to store the collected data. If you haven't already created one, you can follow instructions here: Create a Log Analytics Workspace.
HDInsight clusters include Azure Monitor logs integration, which provides queryable metrics and logs, as well as configurable alerts. This article shows how to use Azure Monitor to monitor your cluster.
As an example, run the Availability rate sample query by selecting Run on that query, as shown in the screenshot above. This will show the availability rate of each node in your cluster as a percentage. If you have enabled multiple HDInsight clusters to send metrics to the same Log Analytics workspace, you'll see the availability rate for all nodes in those clusters displayed.
The Ambari agent is likely using os.system() python calls. You can do the same with
ssh user#node -c "uptime"
Related
I am analyzing azure kubernetes service data ingestion to the log analytics and cost optimization.
When i analyze the logs using queries i found that container inventory produces large data ingestion to log analytics which costs more.
When I analyzed about container inventory for AKS, kubernetes i couldn't find proper answer.
Can anyone explain what is container inventory? and how it generates so much of data ingestion in aks?how can i optimize container inventory data ingestion?
I analyzed more about this concept but i couldn't get proper explanation about it as i fresh to azure AKS and kubernetes. can anyone guide me on this?
Thank you!
The ContainerIventory is an inventory of all the containers running in the cluster and their properties) such as state, ports, environment variables etc).
The inventory is collected periodically (once every minute). This means that all this data is collected every minute for every container.
To reduce data ingestion and cost, there are a few things you can do, which are described in the docs, for example disable the environment variable collection, which can be done by either disabling log collection globally in the oms agent config map:
[log_collection_settings.env_var]
enabled = false
Or by disabling it per container by setting the environment variable AZMON_COLLECT_ENV to false
I have a query on AZURE HDInsights. How do I need to design AZURE HDInsights Cluster according to my on-premises infrastructure ?
What are the major parameters which I need to consider before designing the cluster ?
(For Example) If I have 100 servers running on-premises, how many nodes I need to select in my Cloud Cluster like that. ?!! In AWS we have EMR sizing calculator and Cluster Planner/Advisor. Do we have anything similar planning mechanism in AZURE apart from Pricing Calculator ? Please clarify and provide your inputs. With Any example will be really great. Thanks.
Before deploying an HDInsight cluster, plan for the desired cluster capacity by determining the needed performance and scale. This planning helps optimize both usability and costs. Some cluster capacity decisions cannot be changed after deployment. If the performance parameters change, a cluster can be dismantled and re-created without losing stored data.
The key questions to ask for capacity planning are:
In which geographic region should you deploy your cluster?
How much storage do you need?
What cluster type should you deploy?
What size and type of virtual machine (VM) should your cluster nodes use?
How many worker nodes should your cluster have?
Each and every question is addressed here: "Capacity planning for HDInsight Clusters".
We use Spark 2.2 on Azure HDInsight for ad hoc exploration and batch jobs.
The jobs should run ok on a 5x medium VM cluster. They are
1. notebooks (Zeppelin with Livy.spark2 magics)
2. compiled jars being run with Livy.
I have to remember to scale this cluster down to 1 worker when not using it, to save money. (0 workers would be nice, if that were possible).
I'd like Spark to manage this for me... When a Job starts, scale the cluster up to a minimum size first, then pause ~10 mins while that completes. After an idle period without Jobs, scale down again.
You can use PowerShell or Azure classic CLI to scale up/down the cluster. But you might need to write a script to track the cluster resource usage and scale down automatically.
Here is a powershell syntax
Set-AzureRmHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>
Here is a PowerShell workflow runbook that will help you automate the process of scaling in or out your HDInsight clusters depending on your needs
https://gallery.technet.microsoft.com/scriptcenter/Scale-your-HDInsight-f57bb4d8
or
You can use the below option to scale it manually (even though your question is how to scale up/down automatically, I thought it would be useful to someone who wants to scale up/down manually)
Below is the link for an article explaining different methods to scale the cluster using PowerShell or Classic CLI (remember: the latest CLI does n't support scaling feature)
https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-scaling-best-practices
If you want Spark to handle it dynamically, Azure Databricks is the best choice (but it is only Spark cluster, no Hadoop components (except Hive)). As HDInsight - Spark is not a Azure managed service, and will not solve your use case.
Below is the image of a new cluster (in Azure Data bricks) - I highlighted an "enable auto scaling option" which will allow you to scale dynamically when job is executed.
I'm told that Azure Databricks may be a better solution for this use case.
Does Azure Container Service integrate with Azure Monitor?
Wondering what the best way is to do logging/monitoring of kubernetes cluster?
If you are looking for monitoring tools on Azure, you may want to use Azure OMS (Opertation Management Suite). This gives you the ability to monitor the container inventory, performance, and logs in a single location. To my understanding, the stats of the container is only available for Linux nodes now, if you are deploying your k8s cluster on the Azure Portal.
To do this, you need to first create an OMS account. By this time, you should have the Workspace ID and the key available. The next step would
be to create the oms pod on each node using a DaemonSet.
For the detailed setup, take a look at https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-oms.
For third party tools, Grafana+influxdb is one of the ways I have tried before. Basically it provides you with the metrics on two levels: POD and NODE respectively. The displayed metrics included CPU Usage, Memory Usage, Network Usage and Filesystem Usage, etc. Of course, you can always alter your query to add extra metrics.
For the implementation of this approach, you can refer to https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/monitoring.md.
Hope this helps :)
you can use this CLI command to browse through kubernetes cluster deployed using azure container service.
az acs kubernetes browse -g -n
This way you can see kubernetes webui
also you can use kubectl proxy command.
I have created a windows based HDInsight Cluster (v3.2), and I want to monitor my cluster's system metrics such as CPU usage, memory and Disk I/O, but seems the Ambari API is only partially supported, do we have a way to do this kind of work?
It basically all depends on whether you have a Windows-based HDInsight cluster or a Linux-based HDInsight cluster.
On Linux-based clusters, you can point your browser to https://.azurehdinsight.net to access the Ambari web UI. It gives you access to monitoring (status information and alerts related to different processes), metrics (e.g. CPU, Disk, Memory, Network usage), heat maps and lots of details about the different services. You can also start, stop and restart components and services.
On Windows-based clusters, you're much more limited. First, you do not have access to the Ambari web UI. You can only use the Ambari REST API (meaning, you'll need to write your own application using this API). And the API currently only supports monitoring. So there is currently no way to get access to metrics like CPU usage.
Which usage data are you trying to monitor and is not available.
Take a look at Amabri web on HDINSIGHT Linux clusters which shows graphs for CPU, memory and disk usage. Its powered by the Ambari API which is available on both Windows and Linux clusters.