I'm using Redis cache on the Azure.The Pricing tier of it as Standard 2.5 GB.So my question is, can you tell me how to see the current usage of memory on the cache ? In other words how much of more cache storage remaining for using in the future ? I have tried to find out it on the dash board. But unable to find out it.
You can configure redis cache diagnostics to get this information. Please refer to How to monitor Azure Redis Cache - Available metrics and reporting intervals for more details. From this link, one of the available metrics is Used Memory which I believe you're looking for.
Used Memory The amount of cache memory used for key/value pairs in the
cache in MB during the specified reporting interval. This value maps
to used_memory from the Redis INFO command. This does not include
metadata or fragmentation.
I have not used REDIS Cache personally but if my memory serves me right, I read somewhere that you can find this information by executing REDIS commands through REDIS Console available in the portal as well. For more information about this, please see this link: https://azure.microsoft.com/en-in/documentation/articles/cache-configure/#redis-console.
Run INFO memory command in Redis Console and look for used_memory_human parameter in output.
Related
Inside Azure's Redis Cache Advanced Settings is the maxmemory-reserved setting, which Microsoft documents as:
The maxmemory-reserved setting configures the amount of memory in MB
that is reserved for non-cache operations such as replication during
failover. It can also be used when you have a high fragmentation
ratio. Setting this value allows you to have a more consistent Redis
server experience when your load varies. This value should be set
higher for workloads which are write heavy. When memory is reserved
for such operations it is unavailable for storage of cached data.
I cannot find any documentation on Redis.io about maxmemory-reserved, or within the server configuration file. The command redis-cli config get maxmemory-reserved also does not return anything, so I'm guessing that this is a setting exclusive to Azure.
Is maxmemory-reserved a configurable setting on a Redis server that is not hosted by Azure? If not, is there a way to emulate maxmemory-reserved?
Yes, this is a custom setting added to Redis by Microsoft. You should be able to just use a different maxmemory setting in your Redis configuration file and it will accomplish more or less the same thing.
So I have implemented ELK using REDIS as a caching layer.
I am using REDIS 3.0.4 from an RPM I found for Redhat EL6.
I am also running jemalloc 3.6.0.
I believe the configuration for REDIS is largely vanilla with the exception of a max memory cap and also non-default eviction policy.
maxmemory 500mb
and
maxmemory-policy allkeys-random
though I feel that the eviction policy is probably not required.
Now I have verified that the store is generally empty, i.e. my logstash indexer is doing its job well and the data is making it into Elastic Search.
What concerns me is the used memory for REDIS continues to trend upwards and from what I have seen, if used memory hits max, which it has done, then REDIS stop working, i.e. no more log entries flow through.
So, what am I missing:
Am I being paranoid and can dismiss what I am seeing.
Should I avoid the pre-packaged RPMs.
Is there some additional settings I need to change.
Everything I have read to date about REDIS and ELK suggests that out of the box should be fine.
Be aware that this is a lightweight implementation which I hope will provide impetus for a more broader, bullet proof implementation, which is the reason for the 500MB limit.
I am new to hadoop/yarn, and need to get container resources consumption during task execution.
When I look at doc in apache hadoop website, it says "nodemanager... Is responsible for container,monitoring their resource usage(cpu,memory,disk,network) and report the same to the resourcemanager".
My understanding is that node manager will periodically report resource usage together with heartbeat.
When I look at source code. In NodeStatusUpdaterImpl, totalResource is included in RegisterNodeManagerRequest.I think it is called when init nodemanager and tell RM about the configured resource. But in NodeHeartbeatRequest, the nodestatus only has container id, but no cpu memory etc.
So could you please help me clarify whether cpu memory used by container will be reported to RM? How I can get such data?
Many thanks!
This is the implementation of Container Monitor:
hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
there are methods to check if a container is over the limitation, and this one isProcessTreeOverLimit will show you how yarn get the memory usage of certain container(process). I am not sure if there is a API we can use to get these info. But you can see this file
hadoop-2.6.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
It shows you how Yarn gets memory usage: tracking process file in/proc. This answer will give you the command. I think it's possible to add certain code to get memory usage without Yarn API (I hope it has these APIs too).
I have a Node.js application that receives data via a Websocket connection and pushes each message to an Azure Redis cache. It stores a persistent array of messages in a variable for downstream use, and at regular intervals syncs that array from the cache. Bit convoluted, but at a later point I want to separate out the half of the application that writes to the cache from the half of it that reads from it..
At around 02:00 GMT, based on the Azure portal stats, I appear to have started getting "cache misses" on that sync, which last for a couple of hours before I started getting "cache hits" again sometime around 05:00.
The cache misses correspond to a sudden increase in CPU usage, which peaks at around 05:00. And when I say peaks, I mean it hits 81%, vs a previous max of about 6%.
So sometime around 05:00, the CPU peaks, then drops back to normal, the "cache misses" go away, but looking at the cache memory usage, I drop from about 37.4mb used to about 3.85mb used (which I suspect is the "empty" state), and the list that's being used by this application was emptied.
The only functions that the application is running against the cache are LPUSH and LRANGE, there's nothing that has any capability to remove data, and in case anybody was wondering, when the CPU ramped up the memory usage did not so there's nothing to suggest that rogue additions of data cropped up.
It's only on the Basic plan, so I'm not expecting it to be invulnerable or anything, but even without the replication features of the Standard plan I had expected that it wouldn't be in a position to completely wipe itself - I was under the impression that Redis periodically writes itself to disk and restores from that when it recovers from an error.
All of which is my way of asking:
Does anybody have any idea what might have happened here?
If this is something that others have been able to accidentally trigger themselves, are there any gotchas I should be looking out for that I might have in other applications using the same cache that could have caused it to fail so catastrophically?
I would welcome a chorus of people telling me that the Standard plan won't suffer from this sort of issue, because I've already forked out for it and it would be nice to feel like that was the right call.
Many thanks in advance..
Here my thoughts:
Azure Redis Cache stores information in memory. By default, it won't save a "backup" on disk, so, you had information in memory, for some reason the server got restarted and you lost your data.
PS: See this feedback, there is no option to persist information on disk using azure-redis cache yet http://feedback.azure.com/forums/169382-cache/suggestions/6022838-redis-cache-should-also-support-persistence
Make sure you don't use Basic plan. Basic plan doesn't suppose SLA and from my expirience it lost data quite often
Standard plan provides SLA and utilize 2 instances of Redis Cache. It's quite stable and it didn't lose our data, although such case still possible.
Now, if you're going to use Azure Redis as database, but not as a cache you need to utilize data persistance feature, which is already available in Azure Redis Cache Premium Tier: https://azure.microsoft.com/en-us/documentation/articles/cache-premium-tier-intro (see Redis data persistence)
James, using the Standards instance should give you much improved availability.
With the Basic tier any Azure Fabric update to the Master Node (or hardware failure), will cause you to loose all data.
Azure Redis Cache does not support persistence (writing to disk/blob) yet, even in Standard Tier. But the Standard tier does give you a replicated slave node, that can take over if you Master goes down.
I installed the Azure 1.8 tools/SDK and it upgraded my projects co-located caching from preview to final. However, it also decided to add 20GB to the Role's Local Storage (DiagnosticStore). I manually dialed it down to 500MB but then I get the following message in the Role's Property page (cloud proj => roles => right click role => properties i.e. GUI for ServiceDefinition.csdef):
Caching Diagnostics recommends 20GB of local storage. If you decrease
the size of local storage a full redeployment is required which will
result in a loss of virtual IP addresses for this Cloud Service.
I don't know who signed off on this operating model within MS but it begs a simple Why?. For better understanding, I'm breaking that "Why" into 3 "Why" subquestions for caching in Azure SDK 1.8:
Why is the diagnostics of caching coupled with the caching itself? We just need caching for performance...
Why is the recommendation for a whopping 20Gigs? What happens if I dial it down to 500MB?
Slightly off-topic but still related: why does the decreasing of local storage require a full redeployment? This is especially painful since Azure doesn't provide any strong controls to reserve IP addresses. So if you need to work with 3rd parties that use whitelisted IPs - too bad!?
PS: I did contemplate breaking it into 3 separate questions. But given that they are tightly coupled it seems this would be a more helpful approach for future readers.
Diagnostic store is used for storing cache diagnostic data which includes - server logs, crash dumps, counter data etc. which can be automatically uploaded to Azure Storage by configuring the cache diagnostics (CacheDiagnostics.ConfigureDiagnostics call in OnStart method - without this call, data is generated on local VM but not uplaoded into Azure Storage ). And the amount of data that is collected is controlled by diagnostic level (higher the level, more data is collected) which can be changed dynamically. More details on cache diagnostics is avialble at: http://msdn.microsoft.com/en-us/library/windowsazure/hh914135.aspx
Since you enabled cache, it will come with default diagnostic level that should help in diagnosing cache issues if they happen. This data is stored locally unless you call the ConfigureDiagnostics method in OnStart (which uploads the data to Azure storage).
If a lower storage value is provided (say 2GB), then higher diagnostic levels cannot be used since they need more space (crash dump itself can take upwards 12GB for XL VMs). And if you want higher levels, then you might want to upgrade the deployment with change in the diagnostic store size which defeats the purpose - change diagnostic level without redeployment/upgrade/update/restarts. That is the reason why a limit of 20GB is set to cater to all diagnostic levels (and they can be changed in a running deployment with cscfg change).
is answered above.
Hope this helps.
I'll answer question #3 - local storage decreases are one of the only deployment changes that can't be done in-place (increases are fine, as well as VM size changes and several other changes now possible without redeploy). See this post for details around in-place updates.