I'm running an Azure Cloud Service that uses the "new Azure Cache".
I've configured the Cloud Service to use 30% of memory (default), but still the CacheService keeps eating the memory to a point where the server starts to swap out memory to disk. The server has 3.5 GB RAM (Medium), and the CacheServices used 2GB after running for 3 days (and keeps growing). Se attached picture.
We don't even use the cache, so this makes me rather nervous.
Another weird thing is that the other server in the same deployment does not have this problem.
Can anyone tell me if this is normal, should I be worried, or is there a setting somewhere, that I'm missing?
Related
I've got an app service plan with 14gb of memory - it should be plenty for my application's needs. There are two application services running on it, each identical - the private memory consumption of these hovers around 1gb but can spike to 4gb during periods of high usage. One app has a heavier usage pattern than the other.
Lately, during periods of high usage, I've noticed that the heavily used service can become unresponsive, and memory usage stays at 100% in the App Service Plan.
The high traffic service is using 4gb of private memory and starting to massively slow down. When I head over to the /scm.../ProcessExplorer/ page, I can see that the low traffic service has 1gb private memory used and 10gb of 'Working Set'.
As I understand it, on a single machine at least, the working set should be freed up when that memory is needed on another process. Does this happen naturally when two App Services share a single Plan?
It looks to me like the working set on the low-traffic instance is not being freed up to supply the needs of the high-traffic App Service.
If this is indeed the case, the simple fix is to move them to separate App Service Plans, each with 7gb of memory. However this seems like it might potentially be just shifting the problem around - has anyone else noticed similar issues with multiple Apps on a single App Service Plan? As far as I understand it, these shouldn't interfere with one another to the extent that they all need to be separated. Or have I got the wrong diagnosis?
In some high memory-consumption scenarios, your app might truly require more computing resources. In that case, consider scaling to a higher service tier so the application gets all the resources it needs. Other times, a bug in the code might cause a memory leak. A coding practice also might increase memory consumption. Getting insight into what's triggering high memory consumption is a two-part process. First, create a process dump, and then analyze the process dump. Crash Diagnoser from the Azure Site Extension Gallery can efficiently perform both these steps. For more information.
refer Capture and analyze a dump file for intermittent high memory for Web Apps.
In the end we solved this one via mitigation, rather than getting to the root cause.
We found a mitigation strategy to our previous memory issues several months ago, which was just to restart the server each night using a powershell script. This seems to prevent the memory just building up over time, and only costs us a few seconds of downtime. Our system doesn't have much overnight traffic as our users are all based in the same geographic location.
However we recently found that the overnight restart was reporting 'success' but actually failing each night due to expired credentials. Which meant that the memory issues we were having in the question I posted were actually exacerbated by server uptimes of several weeks. Restoring the overnight restart resolved the memory issues we were seeing, and we certainly don't see our system ever using 10gb+ again.
We'll investigate the memory issues if they rear their heads again. KetanChawda-MSFT's suggestion of using memory dumps to analyse the memory usage will be employed for this investigation when it's needed.
Each week nearly two or three times, azure vm stops the responding. When it happens, Disk Read Bytes are dramatically increasing i dont know why. My application has no read disk function (only writes the logs).
As you see in the figure, Read Bytes and Disk Operations are increasing for a moment that causes server to freeze. I have to do restart the VM and it takes nearly 15 minutes to be available again.
Azure Ubuntu VM Stats
i am using Azure Ubuntu 16.04 VM and Size is Standard B1s (1 vcpus, 1 GiB memory).
i have alredy checked these conditions
possible memory leaks by my application
observed all source codes of application whether disk read function exists or not (i dont have any disk ready operation at my application)
i have changed default virtual ram size at ubuntu %60 to %1
i am using docker to run my application and i am using only 1 instance of docker image at vm
i removed all vm and i created it again (including all resources i.e. disk/network etc.)
i want to know why this is happening and how can i investigate what are cause this problem. I haven't seen any suspicious running process at ubuntu when it is happening.
This issue is solved by Azure Technical Support Engineers (Thanks to Micah_MSFT)
That seems Azure VM is going to out of memory. When it happens, VM is acting unsteady, so Disk Operations is rapidly increasing and VM is freezing (they are investigating the issue's details and reported to Linux Technicians).
As clarified by Azure Technical Support Engineer, There is a feature
named Swap Space which helps the saver to reserve storage space as
processing memory.
Here is the configuration
Open the file located under /etc/waagent.conf find these variables and modify them accordingly:
# Format if unformatted. If 'n', resource disk will not be mounted.
ResourceDisk.Format=n /// change this parameter to "y"
# Create and use swapfile on resource disk.
ResourceDisk.EnableSwap=n /// change this parameter to "y"
# Size of the swapfile.
ResourceDisk.SwapSizeMB=0 /// change this parameter to "1024"
Restart the service to apply the changes
sudo systemctl restart walinuxagent.service
That's it
In my opinion, unnecessary increase in disk usage as a result of Out of Memory limit will cause Azure VMs performance to decrease.
I ran into a situation where out of memory exceptions were generated in our Azure App Service for a .Net Core Web API even though memory & utilization topped 50% in the App Service Plan (P2V2: 7GB RAM).
I have looked at this SO article to check private bytes and other things but still don't see where the memory of exhaustion comes from. I see a max usage of 1.5GB on the memory working set which is well below the 7GB.
Nothing shows up under Support + Troubleshooting -> Resource Health or App Service Advisor.
I am not sure where to look next and any help would be appreciated.
Azure App Services caps memory usage at 1.5G by default. But you can change this behaviour with this application setting (to be added under Configuration):
WEBSITE_MEMORY_LIMIT_MB = 3072
See also my answer here:
Is there way to determine why Azure App Service restarted?
The Metrics view on the portal can only go up to a 1 minute granularity level.
(The default is 5 minutes)
This means that each metric point is an average value over a 60-second interval.
It may be spiking up and down over 60 seconds, so you need a more real-time view.
Try the SCM console (Advanced Tools > Go), and check the Process Explorer to see the actual memory consumption.
I have a simple Umbraco 7.2 website (no plugins/custom code/etc) on a Shared Azure Website. Azure has suspended the website twice in the last week because of over quota memory usage. I scaled it up to 6 instances for now. Looking at the dashboard right now, it shows me it's using 1.5gigs. I went on the Kudu interface (.scm.azurewebsites.net) and in the Process Explorer it shows that the the process is only taking ~150mb of both private memory and working set. It also says virtual memory is taking ~750mb.
Why is Azure saying it's taking up so much memory? Does increasing instance count actually mean more memory for my app or does it just mean more instances are running the same app... so it's basically 200mb*6 instances = 1200megs?
Thanks!
When you scale up the number of instances for an Azure Website it changes the number of VMs your website is running on - so it means more instances running the same app.
Similarly, when you log into the Kudu interface this is going to connect to the process running on one particular instance - so it won't show you total memory being used by all instances at once, just what's used by the one particular instance you're connected to.
I have recently installed NewRelic server monitoring to our Azure web role. The role is a small instance. We are on OSv4 (Win 2012 R2) using 2.2 Service Runtime.
Looking at memory usage I notice that WallSHost.exe (which I understand to be Azure related) it reported as consuming 219Mb (down from a peak of 250Mb) via NewRelic. Is that a lot of memory for it? Can I reduce it? Just seemed like a lot to be taking up.
CPU usage seems to aperiodically spike at about 4% for it. However CPU isn't really an issue as my instance rarely goes above 50%
First off, why do you care how much memory a process is taking up? All of that memory will be paged out to disk, and assuming it isn't being paged back in regularly then all it does is take up page file size which is usually irrelevant.
The WaIISHost process runs your role entry point code (OnStart, Run, StatusCheck, Changing, etc) and is typically implemented in WebRole.cs. If you want to reduce the memory size of this process then you can reduce the amount of memory being loaded by your role entry point code.
See http://blogs.msdn.com/b/kwill/archive/2011/05/05/windows-azure-role-architecture.aspx for more information about the WaIISHost.exe process and what it does.