I am new to hadoop/yarn, and need to get container resources consumption during task execution.
When I look at doc in apache hadoop website, it says "nodemanager... Is responsible for container,monitoring their resource usage(cpu,memory,disk,network) and report the same to the resourcemanager".
My understanding is that node manager will periodically report resource usage together with heartbeat.
When I look at source code. In NodeStatusUpdaterImpl, totalResource is included in RegisterNodeManagerRequest.I think it is called when init nodemanager and tell RM about the configured resource. But in NodeHeartbeatRequest, the nodestatus only has container id, but no cpu memory etc.
So could you please help me clarify whether cpu memory used by container will be reported to RM? How I can get such data?
Many thanks!
This is the implementation of Container Monitor:
hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
there are methods to check if a container is over the limitation, and this one isProcessTreeOverLimit will show you how yarn get the memory usage of certain container(process). I am not sure if there is a API we can use to get these info. But you can see this file
hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
It shows you how Yarn gets memory usage: tracking process file in/proc. This answer will give you the command. I think it's possible to add certain code to get memory usage without Yarn API (I hope it has these APIs too).
Related
I am using Kured to perform safe reboots of our nodes to upgrade the OS and kernel versions.
In my understanding, it works by cordoning and draining the node, and the pods are scheduled on a new node with the older version. After the reboot, the nodes are uncordoned and back to the ready state and the temporary worker nodes get deleted.
It was perfectly fine until yesterday when one of the nodes failed to upgrade to the latest kernel version. It was on 5.4.0-1058-azure last week after a successful upgrade and it should be on 5.4.0-1059-azure yesterday after the latest patch, but it is using the old version 5.4.0-1047-azure (which I think is the version of the temporary node that got created).
Upon checking the log analytics on azure, it says that it failed to scale down.
Reason: ScaleDownFailed
Message: failed to drain the node, aborting ScaleDown
Error message
Any idea on why this is happening?
Firstly, there is a little misunderstanding of the OS and Kernel patching process.
In my understanding, it works by cordoning and draining the node, and the pods are scheduled on a new node with the older version.
The new node that is/are added should come with the latest node image version with latest security patches (which usually does not fall back to an older kernel version) available for the node pool. You can check out the AKS node image releases here. Reference
However, it is not necessary that the pod(s) evicted by the drain operation from the node that is being rebooted at any point during the process has to land on the surge node. Evicted pod(S) might very well be scheduled on an existing node should the node fit the bill for scheduling these pods.
For every Pod that the scheduler discovers, the scheduler becomes responsible for finding the best Node for that Pod to run on. The scheduler reaches this placement decision taking into account the scheduling principles described here.
The documentation, at the time of writing, might be a little misleading on this.
About the error:
Reason: ScaleDownFailed
Message: failed to drain the node, aborting ScaleDown
This might happen due to a number of reasons. Common ones might be:
The scheduler could not find a suitable node to place evicted pods and the node pool could not scale up due to insufficient compute quota available. [Reference]
The scheduler could not find a suitable node to place evicted pods and the cluster could not scale up due to insufficient IP addresses in the node pool's subnet. [Reference]
PodDisruptionBudgets (PDBs) did not allow for at least 1 pod replica to be moved at a time causing the drain/evict operation to fail. [Reference]
In general,
The Eviction API can respond in one of three ways:
If the eviction is granted, then the Pod is deleted as if you sent a DELETE request to the Pod's URL and received back 200 OK.
If the current state of affairs wouldn't allow an eviction by the rules set forth in the budget, you get back 429 Too Many Requests. This is typically used for generic rate limiting of any requests, but here we mean that this request isn't allowed right now but it may be allowed later.
If there is some kind of misconfiguration; for example multiple PodDisruptionBudgets that refer the same Pod, you get a 500 Internal Server Error response.
For a given eviction request, there are two cases:
There is no budget that matches this pod. In this case, the server always returns 200 OK.
There is at least one budget. In this case, any of the three above responses may apply.
Stuck evictions
In some cases, an application may reach a broken state, one where unless you intervene the eviction API will never return anything other than 429 or 500.
For example: this can happen if ReplicaSet is creating Pods for your application but the replacement Pods do not become Ready. You can also see similar symptoms if the last Pod evicted has a very long termination grace period.
How to investigate further?
On the Azure Portal navigate to your AKS cluster
Go to Resource Health on the left hand menu as shown below and click on Diagnose and solve problems
You should see something like the following
If you click on each of the options, you should see a number of checks loading. You can set the time frame of impact on the top right hand corner of the screen as shown below (Please press the Enter key after you have set the correct timeframe). You can click on the More Info link on the right hand side of each entry for detailed information and recommended action.
How to mitigate the issue?
Once you have identified the issue and followed the recommendations to fix the same, please perform an az aks upgrade on the AKS cluster to the same Kubernetes version it is currently running. This should initiate a reconcile operation wherever required under the hood.
I have set up an Azure WebApp (Linux) to run a WordPress and an other handmade PHP app on it. All works fine but I get this weird CPU usage graph (see below).
Both apps are PHP7.0 containers.
SSHing in to the two containers and using top I see no unusual CPU hogging processes.
When I reset both apps the CPU goes back to normal and then starts to raise slowly as shown below.
The amount of HTTP requests to the apps has not relation to the CPU usage at all.
I tried to use apache2ctl to see if there are any pending requests but that seems not possible to do inside a docker container.
Anybody got an idea how to track down the cause of this?
This is the top output. The instance has 2 cores. Lots of idle time but still over 100% load and none of the processes use the CPU ...
After handling with MS Support on that issue it seems to have boiled down to the WordPress theme being to slow or inefficient. Each request took very long and hogged CPU resources. All following requests started queuing up and thus increasing the CPU load.
Why that would not show as %CPU in top I was not explained.
They proposed to use a different theme or upscale to a multi core instance.
I am unsatisfied with that solution and will monitor further and try to find the real culprit.
I had almost exactly the same CPU Percentage graph as you did, although a Node.JS app instead of PHP. Disabling Diagnostic Logs > Docker Container Logging seems to have solved the problem for me.
I do not need those logs because I am logging to application insights.
But, in your case you might need more of those logs. I have no solution for that, but I am guessing that heavier log rotation or reducing the sizes of the logs by other means might help
Am facing issue in my application servers. Assume that - there are two nodes in the Load-balancer.
Suddenly one of the node from them becomes unhealthy.
When I logged in that instance. There were no logs coming in pm2.
then I check its CPU it was very high.
So please guide me how can I fix this issue. Or any way to debug it.
Check out flame graphs to see where your Node app is CPU bound.
You can also use the new debugging system in Node 6.3 (--inspect) to debug with the full power of Chrome DevTools.
PM2 has some limited protection for runaway issues like this via the max-memory-restart option. Typically, high CPU will also correlate with high memory usage and this option can be used to restart your app when it begins consuming large amounts of memory (which in your case may or may not be the correct moment but it should help).
--max-memory-restart <memory> specify max memory amount used to autorestart (in octet or use syntax like 100M)
Whenever we are making a API call to our script it gets completed successfully, but after
the end of the script script, memory doesn't get released. Lets say if there was 10MB memory was
used up during execution then after execution memory usage should have come done atleast by 5 MB
but it is not happening.
So after a certain amount of time memory usage go beyond the 75% usage and we start getting alerts.
Docker version 1.11.2, build b9f10c9/1.11.2
Python3.4.2
flask
We motoring usage with docker stats command
Found this solution and it really helped.
This issue was due to linux and python. Python was releasing the memory but
linux thought flask is still running(caller of process) so it should not release that memory and due to this memory was not getting released.
http://www.paulsprogrammingnotes.com/2014/10/large-dictionaries-not-released-from.html?showComment=1483516233443#c3352147816385844344
I'm using Redis cache on the Azure.The Pricing tier of it as Standard 2.5 GB.So my question is, can you tell me how to see the current usage of memory on the cache ? In other words how much of more cache storage remaining for using in the future ? I have tried to find out it on the dash board. But unable to find out it.
You can configure redis cache diagnostics to get this information. Please refer to How to monitor Azure Redis Cache - Available metrics and reporting intervals for more details. From this link, one of the available metrics is Used Memory which I believe you're looking for.
Used Memory The amount of cache memory used for key/value pairs in the
cache in MB during the specified reporting interval. This value maps
to used_memory from the Redis INFO command. This does not include
metadata or fragmentation.
I have not used REDIS Cache personally but if my memory serves me right, I read somewhere that you can find this information by executing REDIS commands through REDIS Console available in the portal as well. For more information about this, please see this link: https://azure.microsoft.com/en-in/documentation/articles/cache-configure/#redis-console.
Run INFO memory command in Redis Console and look for used_memory_human parameter in output.