I am on a medium instance which has 3GB of RAM. When I start my webapp the w3wp process starts with say 80MB. I notice that the more time passes this goes up and up.... Now I took a memory dump of the process when it was 570MB and the site was running for 5 days, to see whether there were any .NET objects which were consuming a lot but found out that the largest object was 18MB which were a set of string objects.
I am not using any cache objects since I'm using redis for my session storage, and in actual fact the dump showed that there was nothing in the cache.
Now my question is the following... I am thinking that since I have 3GB of memory IIS will retain some pages in memory (cached) so the website is faster whenever there are requests and that is the reason why the memory keeps increasing. What I'm concerned is that I am having some memory leak in some way, even if I am disposing all EntityFramework objects when being used, or any other appropriate streams which need to be disposed. When some specific threshold is reached I am assuming that old cached data which was in memory gets removed and new pages are included. Am I right in saying this?
I want to point out that in the past I had been on a small instance and the % never went more than 70% and now I am on medium instance and the memory is already 60%.... very very strange with the same code.
I can send memory dump if anyone would like to help me out.
There is an issue that is affecting a small number of Web Apps, and that we're working on patching.
There is a workaround if you are hitting this particular issue:
Go to Kudu Console for your app (e.g. https://{yourapp}.scm.azurewebsites.net/DebugConsole)
Go into the LogFiles folder. If you are running into this issue, you will have a very large eventlog.xml file
Make that file readonly, by running attrib +r eventlog.xml
Optionally, restart your Web App so you have a clean w3wp
Monitor whether the usage still goes up
The one downside is that you'll no longer get those events generated, but in most cases they are not needed (and this is temporary).
The problem has been identified, but we don't have an ETA for the deployment yet.
Related
We have a very heavily used asp.net web API app running as azure app service. The application shows memory usage trend as shown below i.e. the memory usage keeps on increasing and never comes down till it gets restarted (using Azure auto healing).
To further analyze we took 3 memory dumps i.e.
first dump after restarting the app
Ran a load test and took second dump
Again ran the same load test and took third dump
Below is how the comparison between dumps look like
Between 1st and 2nd dump.
Between 2nd and 3rd dump
In all these comparisons It seems maximum memory usage increase happens in some caching related objects e.g. ExpiresBucket, RuntimeCache, MemoryCache, CacheExpires etc.
I am not an expert and not sure what exactly these classes do and from where this increase is coming from.
Can somebody throw some light on what these objects mean and is this normal?
Just to be clear we have seen trend and see that memory only increases and never comes down.
Background
I have a relatively simple node js application (essentially just expressjs + mongoose). It is currently running in production on an Ubuntu Server and serves about 20,000 page views per day.
Initially the application was running on a machine with 512 MB memory. Upon noticing that the server would essentially crash every so often I suspected that the application might be running out of memory, which was the case.
I have since moved the application to a server with 1 GB of memory. I have been monitoring the application and within a few minutes the application tends to reach about 200-250 MB of memory usage. Over longer periods of time (say 10+ hours) it seems that the amount keeps growing very slowly (I'm still investigating that).
I have been since been trying to figure out what is consuming the memory. I have been going through my code and have not found any obvious memory leaks (for example unclosed db connections and such).
Tests
I have implemented a handy heapdump function using node-heapdump and I have now enabled --expore-gc to be able to manually trigger garbage collection. From time to time I try triggering a manual GC to see what happens with the memory usage, but it seems to have no effect whatsoever.
I have also tried analysing heapdumps from time to time - but I'm not sure if what I'm seeing is normal or not. I do find it slightly suspicious that there is one entry with 93% of the retained size - but it just points to "builtins" (not really sure what the signifies).
Upon inspecting the 2nd highest retained size (Buffer) I can see that it links back to the same "builtins" via a setTimeout function in some Native Code. I suspect it is cache or https related (_cache, slabBuffer, tls).
Questions
Does this look normal for a Node JS application?
Is anyone able to draw any sort of conclusion from this?
What exactly is "builtins" (does it refer to builtin js types)?
I have recently installed NewRelic server monitoring to our Azure web role. The role is a small instance. We are on OSv4 (Win 2012 R2) using 2.2 Service Runtime.
Looking at memory usage I notice that WallSHost.exe (which I understand to be Azure related) it reported as consuming 219Mb (down from a peak of 250Mb) via NewRelic. Is that a lot of memory for it? Can I reduce it? Just seemed like a lot to be taking up.
CPU usage seems to aperiodically spike at about 4% for it. However CPU isn't really an issue as my instance rarely goes above 50%
First off, why do you care how much memory a process is taking up? All of that memory will be paged out to disk, and assuming it isn't being paged back in regularly then all it does is take up page file size which is usually irrelevant.
The WaIISHost process runs your role entry point code (OnStart, Run, StatusCheck, Changing, etc) and is typically implemented in WebRole.cs. If you want to reduce the memory size of this process then you can reduce the amount of memory being loaded by your role entry point code.
See http://blogs.msdn.com/b/kwill/archive/2011/05/05/windows-azure-role-architecture.aspx for more information about the WaIISHost.exe process and what it does.
I am experiencing a really curious problem with a HttpHandler and I am hoping someobody here might be able to shed light on this. Many thanks in advance for reading this.
We have created a HttpHandler that sits in the pipeline of an IIS website that serves images, videos, and other assets. The HttpHandler is very lightweight. Its sole purpose is to check if the media asset requested exists on disk and, if it does not, to re-write the URL for the asset to a location where the asset does exist. The handler has been created in this way to allow us to migrate our media assets into a new folder structure. We also plan to use the handler (which I will refer to as the URLRewriter from here on) for SEO on image and video URLs.
As mentioned, the URLRewriter class is very lightweight. We have run memory profiling over it and determined that it only consumes about 12B of memory while running. However, when we put the handler into the IIS pipeline we see some strange behaviour that ultimately results in a large amount of memory consumption and, invariably, that the w3 worker process recycles. The behaviour we see is this:
When a request comes in for an image on http://www.ourimageserver.com/media/a/b/c/d/image1xxl.jpg (not an actual URL) we notice that W3WP.exe creates, and hangs on to, a handle for every single folder in the path to the image:
• /media
• /media/a
• /media/a/b
• /media/a/b/c
• /media/a/b/c/d
This is a big problem because we have hundreds of thousands of media assets that are stored in a very wide and very deep folder structure. The number of handles created by IIS/W3WP grows rapidly when the URLRewriter is deployed to our production environment, and the memory consumption of W3WP goes up correspondingly. Within less than an hour of running (at a relatively quiet period in terms of traffic) the number of handles held by W3WP was in excess of 22000 and the process died. We also noticed that the kernel memory usage had increased on the servers where the URLRewriter was deployed.
Careful inspection of W3WP’s behaviour using Process Explorer and Process Monitor (both with and without a VS debugger attached) have revealed that the handles are created before the URLRewriter is called. In fact, the handles are created before the BeginRequest event is fired. When the URLRewriter is removed from the pipeline none of these handles are created. Now, a really curious thing is that it appears that the handles are created as result of a NotifyChangeDirectory operation carried out by W3WP. Why would W3WP request to be notified for changes to these directories? And how can we prevent it from doing so? Surely this is not default/normal behaviour?
If you have any ideas as to what might be causing this problem I would be most grateful for your input. The behaviour is the same on IIS6 and IIS7.
We have a really strange problem. One of the servers in the server farm becomes really slow. We see a number of timeouts in the logs and overall response time is not where it should be (and is on other servers in the farm).
What is also strange is that it is not just the web app - Just logging into the server takes up to 1.5 min to show you the desktop. Once you are in, the system is as responsive as ever - unless you try to launch something, i.e. notepad - it takes another minute to launch and after launch it works fine.
I checked a number of things - memory utilization is reasonable, CPU is below 15%, windows handles, event logs do not show anything.
Recycling the aps.net process does not fix it - it still takes over a minute to log in. Rebooting the server helped, but now it started to slow down again.
After a closer look we found out that Windows Temp directory is full of temp files - over 65k files. This is certainly something to take care of. But my question is could it be the root cause of the sluggishness, or there is still something else lurking in the shadows?
Edit
After more digging I am zeroing in on the issue related to the size of temp directories. This article: describes something very similar. I am still not too sure because the fact that the server is slow to open even Notepad remains unexplained.
Is it possible that under such conditions creating a new temp file takes over a minute?
You might want to check how many threads your using in the ASP.NET thread pool when the timeouts occur. Another idea might be to look at the GC information in perfmon and see if the GC is running a gen2 collection?
Ok, It is official, all of this was grief caused by this issue. When one of our servers was again behaving badly we cleaned the temp directory and it fixed the problem, including the slow login.
This last part still baffles me - I do not understand how excessive number of files in a temp directory can cause login to take over 1 min, leave alone launching a program, but whatever it is clearing the directory fixed it and I can live with it.
Did you check virtual memory as well ? paging ? does you app logs a lot of data in different files ? also - check - maybe the utilization happens in kernel mode and not user mode.