Inside Azure's Redis Cache Advanced Settings is the maxmemory-reserved setting, which Microsoft documents as:
The maxmemory-reserved setting configures the amount of memory in MB
that is reserved for non-cache operations such as replication during
failover. It can also be used when you have a high fragmentation
ratio. Setting this value allows you to have a more consistent Redis
server experience when your load varies. This value should be set
higher for workloads which are write heavy. When memory is reserved
for such operations it is unavailable for storage of cached data.
I cannot find any documentation on Redis.io about maxmemory-reserved, or within the server configuration file. The command redis-cli config get maxmemory-reserved also does not return anything, so I'm guessing that this is a setting exclusive to Azure.
Is maxmemory-reserved a configurable setting on a Redis server that is not hosted by Azure? If not, is there a way to emulate maxmemory-reserved?
Yes, this is a custom setting added to Redis by Microsoft. You should be able to just use a different maxmemory setting in your Redis configuration file and it will accomplish more or less the same thing.
Related
Tech Stack
Azure WebApp
.Net core 2.1 WebApi
We have around 4k reference data which is used during auto suggest lookup, so in this i was wondering whether i should cache this data on WebApp or should always get it from database / 3rd party API.
I know i can use RedisCache to solve this issue, but i would like to know how Azure WebApp works when it comes to caching, it will have memory pressure? When? Yes then scale-up is the only solution?
We are using IMemoryCache in .net Core to store reference data and it expires on daily basis or when Azure WebApp is restarted (So 1st user will get delay till it gets all data in cache).
Data size is in range of 500KB - 1MB & sometimes goes till 3MB+.
What is the best approach?
iMemoryCache is not suggested when using WebApps because it is tightly bound to your application instance, so if you try to scale out your app (in case of load surges during the day) your caching mechanism will be broken.
RedisCache is pretty much a dictionary, key-value pairs.
It is very fast on look-ups but it could be very slow in some other operations like a GetAllKeys when it has to run through the whole cache. That will bring your cache server to its knees, so it needs to be handled carefully.
It will not put any significant pressure in the memory consumption of your app, you only need to have a static client. The rest is handled by the redis server.
If you plan to scale up your application (give more RAM and CPU resources to your one running instance) the iMemory cache is probably fine.
If you plan to scale out (create multiple instances of your application), that is strongly suggested for all stateless applications, then RedisCache (or any other distributed cache) is an one way for you if you need a caching mechanism.
Value and key max size is 512MB so you are on the safe side regarding value data size.
Attention
Be sure to use the Connection multiplexer as it is suggested in the official documentation because it automatically re-establishes the connection in case it is lost. That was a bug earlier, when redis cache server was going into maintenance your calls where redirected to the fail over instance but the connection was failing, so you needed to restart your application.
So I'm trying to migrate a Legacy website from an AWS VM to an Azure VM and we're trying to get the same level of performance. The problem is I'm pretty new to setting up sites on IIS.
The authors of the application are long gone and we struggle with the application for many reasons. One of the problems with the site is when it's "warming up" it pulls back a ton of data to store in memory for the entire day. This involves executing long running stored procs and in memory processes which means first load of certain pages takes up to 7 minutes. It then uses a combination of in memory data and output caching to deliver the pages.
Sessions do seem to be in use although the site is capable of recovering session data from the database in some more relatively long running database operations so sessions are better to stick with where possible which is why I'm avoiding a web garden.
That's a little bit of background, however my question is really about upping the performance on IIS. When I went through their settings on the AWS box they had something call NUMA enabled with what appears to be the default settings and then the maximum worker processes set to 0 which seems to enable NUMA. I don't know why they enabled NUMA or if it was necessary, but I am trying to get as close to a like for like transition as possible and if it gives extra performance in this application we'll probably need it!
On the Azure box I can see options to set the maximum worker processes to 0 but no NUMA options. My question is whether NUMA is enabled with those default options or is there something further I need to do to enable NUMA.
Both are production sized VMs but the one on Azure I'm working with is a Standard D16s_v3 with 16 vCores and 64Gb RAM. We are load balancing across a few of them.
If you don't see the option in the Azure VM it's because the server is using symmetric processing and isn't NUMA aware.
Now to optimize your loading a bit:
HUGE CAVEAT - if you have memory leak type issues, don't do this! To ensure you don't, put on a private bytes limit roughly 70% the size of memory on the server. If you see that get hit/issue an IIS recycle (that event is logged by default) then you may want to ignore further steps. Either that or mess around with perfmon (or more easily iteratively check peak bytes in task manager where you'll have to add that column in the details pane)
Change your app pool startup mode to: AlwaysRunning
Change your web app to preloadenabled=true
Set an initialization page in your web.config (so that preloading knows what to load).
*Edit forgot some steps. Make sure your idle timeout is clear or set it to midnight.
Make sure you don't have the default recycle time enabled, clear that out.
If you want to get fancy you can add a loading page and set an http refresh or due further customizations seen below:
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization
I'm using Redis cache on the Azure.The Pricing tier of it as Standard 2.5 GB.So my question is, can you tell me how to see the current usage of memory on the cache ? In other words how much of more cache storage remaining for using in the future ? I have tried to find out it on the dash board. But unable to find out it.
You can configure redis cache diagnostics to get this information. Please refer to How to monitor Azure Redis Cache - Available metrics and reporting intervals for more details. From this link, one of the available metrics is Used Memory which I believe you're looking for.
Used Memory The amount of cache memory used for key/value pairs in the
cache in MB during the specified reporting interval. This value maps
to used_memory from the Redis INFO command. This does not include
metadata or fragmentation.
I have not used REDIS Cache personally but if my memory serves me right, I read somewhere that you can find this information by executing REDIS commands through REDIS Console available in the portal as well. For more information about this, please see this link: https://azure.microsoft.com/en-in/documentation/articles/cache-configure/#redis-console.
Run INFO memory command in Redis Console and look for used_memory_human parameter in output.
A large e-commerce site is looking to switch its session cache from Shared cache to dedicated cache.
It is usually running on medium-size servers (5-6)... During busy times, it's running on 20 medium servers. During the very busy times, it is not unreasonable to have 2000+ requests per second to the site
Is co-located cache good enough here or must cache be in the dedicate worker role?
Also, must high-availability be enabled for session data? The site relies upon session data to be present for good user experience. But the cache is persisted to Azure blob storage, so I'm not sure I totally get the high-availability option
The use of dedicated roles depends on how many roles you want to run, and whether or not the memory usage of your web roles determines if they scale. For example, if your web roles are always pushing memory usage, and it is memory and not CPU that is the trigger for scaling out - then consider using dedicated roles for the cache, as your web roles can then handle the load for longer. If your web roles are cpu intensive, then dedicating memory on each role to the cache may be preferred. You also need to consider that if running in dedicated roles, you need more than one role to handle the load and availability, so even during non-busy times, you will have at least 3 roles running the cache (but possibly fewer web roles). You may also want to use dedicated cache if you do lots of deployments or scaling down - where roles are shut down intentionally and frequently.
One consideration on co-located role caching is that if you had sticky sessions the latency would be lower, as the item is on the same machine. Unfortunately, the Azure load balancer is round robin, and not sticky at all, so the chance that a session gets back to the same machine is low (1/5 of the time for 5 roles). This means that most of the time the cache item will be fetched from another role in the cluster, so co-located latency benefits are lost.
The cache is distributed and in-memory - there is no blob storage that I am aware of (except for 'cluster's runtime state' - whatever that is. An item loaded into cache is made available to other machines on the cluster from the machine that it is stored (in memory) on (a read from machine B to machine A does not also store it on machine A - see comment below). Cached items are always in memory only, and the cache size is limited by available memory.
The high availability option copies the item to a separate machine (not storage), so if one machine fails, there is still a copy somewhere. High availability will also use more memory, as an item uses memory in two different places. The chances of failure maybe low enough for your e-commerce app - if an item is not cached (either through failure or expiry) it may be reconstructed from persisted data. If you are, for example, keeping the basket in cache and not persisted to storage, you don't want it lost if a role recycles - in which case high availability may be the best option.
Great answer #SimonMunro however in my experience the Azure Co-located Cache is not fit for production. Our load testing has shown us that when a server is recycled that it takes an exceptional long period of time for a the cache to recover. We have coded against this by fetching the data from our database however our site grinds to a halt due to the stress on the database. This not only happens when a node is recycled; but also if you scale your cloud services up and down; and even when you perform a VIP swap.
We have performed the same tests using the Azure dedicated cache and have found it to handle the situation of a cache worker role recycling with little to no effect to the performance of the site. It is my recommendation is to use the Azure Dedicated Cache in all cases if you want your site to perform.
I installed the Azure 1.8 tools/SDK and it upgraded my projects co-located caching from preview to final. However, it also decided to add 20GB to the Role's Local Storage (DiagnosticStore). I manually dialed it down to 500MB but then I get the following message in the Role's Property page (cloud proj => roles => right click role => properties i.e. GUI for ServiceDefinition.csdef):
Caching Diagnostics recommends 20GB of local storage. If you decrease
the size of local storage a full redeployment is required which will
result in a loss of virtual IP addresses for this Cloud Service.
I don't know who signed off on this operating model within MS but it begs a simple Why?. For better understanding, I'm breaking that "Why" into 3 "Why" subquestions for caching in Azure SDK 1.8:
Why is the diagnostics of caching coupled with the caching itself? We just need caching for performance...
Why is the recommendation for a whopping 20Gigs? What happens if I dial it down to 500MB?
Slightly off-topic but still related: why does the decreasing of local storage require a full redeployment? This is especially painful since Azure doesn't provide any strong controls to reserve IP addresses. So if you need to work with 3rd parties that use whitelisted IPs - too bad!?
PS: I did contemplate breaking it into 3 separate questions. But given that they are tightly coupled it seems this would be a more helpful approach for future readers.
Diagnostic store is used for storing cache diagnostic data which includes - server logs, crash dumps, counter data etc. which can be automatically uploaded to Azure Storage by configuring the cache diagnostics (CacheDiagnostics.ConfigureDiagnostics call in OnStart method - without this call, data is generated on local VM but not uplaoded into Azure Storage ). And the amount of data that is collected is controlled by diagnostic level (higher the level, more data is collected) which can be changed dynamically. More details on cache diagnostics is avialble at: http://msdn.microsoft.com/en-us/library/windowsazure/hh914135.aspx
Since you enabled cache, it will come with default diagnostic level that should help in diagnosing cache issues if they happen. This data is stored locally unless you call the ConfigureDiagnostics method in OnStart (which uploads the data to Azure storage).
If a lower storage value is provided (say 2GB), then higher diagnostic levels cannot be used since they need more space (crash dump itself can take upwards 12GB for XL VMs). And if you want higher levels, then you might want to upgrade the deployment with change in the diagnostic store size which defeats the purpose - change diagnostic level without redeployment/upgrade/update/restarts. That is the reason why a limit of 20GB is set to cater to all diagnostic levels (and they can be changed in a running deployment with cscfg change).
is answered above.
Hope this helps.
I'll answer question #3 - local storage decreases are one of the only deployment changes that can't be done in-place (increases are fine, as well as VM size changes and several other changes now possible without redeploy). See this post for details around in-place updates.