Tech Stack
Azure WebApp
.Net core 2.1 WebApi
We have around 4k reference data which is used during auto suggest lookup, so in this i was wondering whether i should cache this data on WebApp or should always get it from database / 3rd party API.
I know i can use RedisCache to solve this issue, but i would like to know how Azure WebApp works when it comes to caching, it will have memory pressure? When? Yes then scale-up is the only solution?
We are using IMemoryCache in .net Core to store reference data and it expires on daily basis or when Azure WebApp is restarted (So 1st user will get delay till it gets all data in cache).
Data size is in range of 500KB - 1MB & sometimes goes till 3MB+.
What is the best approach?
iMemoryCache is not suggested when using WebApps because it is tightly bound to your application instance, so if you try to scale out your app (in case of load surges during the day) your caching mechanism will be broken.
RedisCache is pretty much a dictionary, key-value pairs.
It is very fast on look-ups but it could be very slow in some other operations like a GetAllKeys when it has to run through the whole cache. That will bring your cache server to its knees, so it needs to be handled carefully.
It will not put any significant pressure in the memory consumption of your app, you only need to have a static client. The rest is handled by the redis server.
If you plan to scale up your application (give more RAM and CPU resources to your one running instance) the iMemory cache is probably fine.
If you plan to scale out (create multiple instances of your application), that is strongly suggested for all stateless applications, then RedisCache (or any other distributed cache) is an one way for you if you need a caching mechanism.
Value and key max size is 512MB so you are on the safe side regarding value data size.
Attention
Be sure to use the Connection multiplexer as it is suggested in the official documentation because it automatically re-establishes the connection in case it is lost. That was a bug earlier, when redis cache server was going into maintenance your calls where redirected to the fail over instance but the connection was failing, so you needed to restart your application.
Related
I've read all of the documentation about App Service Local Cache but I am struggling to see how it is useful. It claims to basically create a read-only copy of your Site directory, which for an MVC app is basically the whole app. But I can't find any information about use cases or why you'd want to do this.
I ask because it's been suggested that we move to implementing it, and I am trying to work out why we should do this.
I can see advantages if you do lots of reading/writing to disk but hardly any apps do that these days, we just use the database for everything, and logging goes directly to OMS.
Am I missing something major about this feature? To make my question non-vague, does this feature offer something useful for a simple MVC website that displays data from a database and writes back to the database?
Even if your app doesn't perform a large amount of I/O operations, you can still benefit from using App Service Local Cache due to:
Quicker app restarts (since files are local, latency to the shared network drive is removed). Helpful for app settings updates.
Less application downtime if your app loses connectivity with the shared network drive (which causes restarts), which can happen during Azure update/patch operations on the underlying VM
More are discussed in the Channel 9 video for Local Cache https://channel9.msdn.com/Shows/Cloud+Cover/Episode-201-Azure-Web-App-Local-Cache-with-Cory-Fowler
We are having issues with an Azure Application Service. One of our webservices (MVC) caches data from the database at startup (Application_Start) - this takes approximately 3 minutes. Until this is ready we can't handle requests.
This is known so we set it 'always on' and will aim to only restart it during off-peak times if necessary.
However, we expect heavy load to the server next month, and in our testing of the auto-scaling, we have found that when it adds additional instances, each of these instances goes through the same startup delay - but the traffic is split between the current running instance and the new one that's warming up so e.g. half of the requests start failing for that 3 minute period.
How can we configure Azure to delay using the new instance until it is ready? (or should we be using e.g. AWS instead?).
Some of the documentation points to using a custom Load Balancer Probe however it mainly talks about VM's whereas we are using PAAS.
Do try to reduce the data you need to load on app_start and try to lazy load data into Cache on first request. Some times even after doing all of this we do end up with large sets of data that is necessary on start.
There are two ways we can approach this.
One, assuming you are using in-memory caching and every instance of the app needs to hydrate its in-memory cache on App_Start. Try to use a external cache provider like Azure Cache for Redis, your new instance can just point to this external cache without having to reload the data.
Two, you can depend on Application Initialization Module which was introduced in IIS 7.5 (installed on Azure App Services' IIS). To use this feature, you need to add applicationInitialization section under web.server section of web.config. This will help you not make the instance available until the warm-up process is completed. More info on how to use ApplicationInitialization is available in this blog post
The best case would be to use the combination of both, applicationInitialization will point at a page in your application which checks if the external cache is available and hydrated, if yes, complete, else hydrate the external cache.
You can do this in Azure with other resource type than classic VM like an App Service. App Services scale up and down with instances that share the same memory pool and thread pool.
There is a lot of good information, in the link https://www.jan-v.nl/post/warming-up-your-app-service that was included in one of the comments.
Based on that information the functionality that you require is not available in the free tier.
I would approach the problem differently. Why does it take 3 mins to load the data from the database? Since it is only loaded on start it should be data that does not change often.
Could you:
Optimise the reading of data from the database?
Reduce the amount of data you read from the database?
Export the data to a file, and read it from a file?
My recommendation would be to use an Azure Load Balancer with a health probe
We want to save a few bucks and share our 1GB dedicated Azure Redis Cache between Development, Test, QA and maybe even production.
Is there a better way than prefixing all keys with an environment string like "Dev_[key]", "Test_[key]" etc.
We are using the StackExchange Redis client for .NET.
PS: We tried using the cheap 250GB (Shared infrastructure), but had very slow performance. Read operations were consistent between 600-800ms... without any load (for a ~300KB object). Upgrading to dedicated 1GB services changed that to 30-40ms. See more here: StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors
One approach is to use multiple Redis databases. I'm assuming this is available in your environment :)
Some advantages over prefixing your keys might be:
data is kept separate, you can flushdb in test and not touch the production data
keys are smaller and consume less memory
The main disadvantage would be not taking advantage of multiple cores, like you could do if you ran multiple instances of Redis on the same server. Obviously not an issue in this case. Also note that this feature is not deprecated, like one of the answers suggests.
Another thing I've seen people complain about is that databases are numbered, they don't have meaningful names. Some people create a hash in database 0 that maps each number to a name.
Here is another idea to save some bucks: use separate Redis cache machines for each environment - so no problems with the keys, but stop them when you don't use them, like in the weekend and during nights. Probably more than 50% of the time you are not using them. I think it would be easy to start and stop them with some PowerShell script, we are using AWS and here it is possible.
Now from what I see the Redis persistence in Azure is not enabled, but they started working on it http://feedback.azure.com/forums/169382-cache/status/191763 - it would be nice to do a RDB snapshot before stopping and then on start to load it. So if you need to save some values and reload them on start you should do it manually (with your own service).
I have a Node.js application that receives data via a Websocket connection and pushes each message to an Azure Redis cache. It stores a persistent array of messages in a variable for downstream use, and at regular intervals syncs that array from the cache. Bit convoluted, but at a later point I want to separate out the half of the application that writes to the cache from the half of it that reads from it..
At around 02:00 GMT, based on the Azure portal stats, I appear to have started getting "cache misses" on that sync, which last for a couple of hours before I started getting "cache hits" again sometime around 05:00.
The cache misses correspond to a sudden increase in CPU usage, which peaks at around 05:00. And when I say peaks, I mean it hits 81%, vs a previous max of about 6%.
So sometime around 05:00, the CPU peaks, then drops back to normal, the "cache misses" go away, but looking at the cache memory usage, I drop from about 37.4mb used to about 3.85mb used (which I suspect is the "empty" state), and the list that's being used by this application was emptied.
The only functions that the application is running against the cache are LPUSH and LRANGE, there's nothing that has any capability to remove data, and in case anybody was wondering, when the CPU ramped up the memory usage did not so there's nothing to suggest that rogue additions of data cropped up.
It's only on the Basic plan, so I'm not expecting it to be invulnerable or anything, but even without the replication features of the Standard plan I had expected that it wouldn't be in a position to completely wipe itself - I was under the impression that Redis periodically writes itself to disk and restores from that when it recovers from an error.
All of which is my way of asking:
Does anybody have any idea what might have happened here?
If this is something that others have been able to accidentally trigger themselves, are there any gotchas I should be looking out for that I might have in other applications using the same cache that could have caused it to fail so catastrophically?
I would welcome a chorus of people telling me that the Standard plan won't suffer from this sort of issue, because I've already forked out for it and it would be nice to feel like that was the right call.
Many thanks in advance..
Here my thoughts:
Azure Redis Cache stores information in memory. By default, it won't save a "backup" on disk, so, you had information in memory, for some reason the server got restarted and you lost your data.
PS: See this feedback, there is no option to persist information on disk using azure-redis cache yet http://feedback.azure.com/forums/169382-cache/suggestions/6022838-redis-cache-should-also-support-persistence
Make sure you don't use Basic plan. Basic plan doesn't suppose SLA and from my expirience it lost data quite often
Standard plan provides SLA and utilize 2 instances of Redis Cache. It's quite stable and it didn't lose our data, although such case still possible.
Now, if you're going to use Azure Redis as database, but not as a cache you need to utilize data persistance feature, which is already available in Azure Redis Cache Premium Tier: https://azure.microsoft.com/en-us/documentation/articles/cache-premium-tier-intro (see Redis data persistence)
James, using the Standards instance should give you much improved availability.
With the Basic tier any Azure Fabric update to the Master Node (or hardware failure), will cause you to loose all data.
Azure Redis Cache does not support persistence (writing to disk/blob) yet, even in Standard Tier. But the Standard tier does give you a replicated slave node, that can take over if you Master goes down.
A large e-commerce site is looking to switch its session cache from Shared cache to dedicated cache.
It is usually running on medium-size servers (5-6)... During busy times, it's running on 20 medium servers. During the very busy times, it is not unreasonable to have 2000+ requests per second to the site
Is co-located cache good enough here or must cache be in the dedicate worker role?
Also, must high-availability be enabled for session data? The site relies upon session data to be present for good user experience. But the cache is persisted to Azure blob storage, so I'm not sure I totally get the high-availability option
The use of dedicated roles depends on how many roles you want to run, and whether or not the memory usage of your web roles determines if they scale. For example, if your web roles are always pushing memory usage, and it is memory and not CPU that is the trigger for scaling out - then consider using dedicated roles for the cache, as your web roles can then handle the load for longer. If your web roles are cpu intensive, then dedicating memory on each role to the cache may be preferred. You also need to consider that if running in dedicated roles, you need more than one role to handle the load and availability, so even during non-busy times, you will have at least 3 roles running the cache (but possibly fewer web roles). You may also want to use dedicated cache if you do lots of deployments or scaling down - where roles are shut down intentionally and frequently.
One consideration on co-located role caching is that if you had sticky sessions the latency would be lower, as the item is on the same machine. Unfortunately, the Azure load balancer is round robin, and not sticky at all, so the chance that a session gets back to the same machine is low (1/5 of the time for 5 roles). This means that most of the time the cache item will be fetched from another role in the cluster, so co-located latency benefits are lost.
The cache is distributed and in-memory - there is no blob storage that I am aware of (except for 'cluster's runtime state' - whatever that is. An item loaded into cache is made available to other machines on the cluster from the machine that it is stored (in memory) on (a read from machine B to machine A does not also store it on machine A - see comment below). Cached items are always in memory only, and the cache size is limited by available memory.
The high availability option copies the item to a separate machine (not storage), so if one machine fails, there is still a copy somewhere. High availability will also use more memory, as an item uses memory in two different places. The chances of failure maybe low enough for your e-commerce app - if an item is not cached (either through failure or expiry) it may be reconstructed from persisted data. If you are, for example, keeping the basket in cache and not persisted to storage, you don't want it lost if a role recycles - in which case high availability may be the best option.
Great answer #SimonMunro however in my experience the Azure Co-located Cache is not fit for production. Our load testing has shown us that when a server is recycled that it takes an exceptional long period of time for a the cache to recover. We have coded against this by fetching the data from our database however our site grinds to a halt due to the stress on the database. This not only happens when a node is recycled; but also if you scale your cloud services up and down; and even when you perform a VIP swap.
We have performed the same tests using the Azure dedicated cache and have found it to handle the situation of a cache worker role recycling with little to no effect to the performance of the site. It is my recommendation is to use the Azure Dedicated Cache in all cases if you want your site to perform.