I've got a Pyramid view that's misbehaving in an interesting way. What the view does is grab a pretty complex object hierarchy from a file (using pickle), does a little processing, then renders an html form. Nice and simple.
Setup:
I'm running Ubuntu 12.04 64bit, Python3.2, Pyramid 1.3.3, SQLAlchemy 0.7.8 and using the standard waitress server.
Symptoms
I was having some efficiency issues so used system monitor to try to see what was up and found that while pyramid is doing its processing and suchlike for the view I described my ram usage rose steadily. As the html form was displayed in my browser the ram usage leveled off but didn't fall. Reloading the view caused the ram usage to grow steadily from where it left off. If I keep doing this all my ram is used up and everything grinds to a halt.
If I kill the server then the ram usage drops back down immediately.
Question
What's going on? It's obvious that memory isn't being released between view renderings, but why is this happening? And how can I make it stop? I even tried calling del on stuff before returning from the view and nothing changed.
Pyramid's debug toolbar keeps objects alive. Deactivating it fixes most memory leak problems. The leak that was the cause of my searching for errors in Pyramid doesn't seem to be a problem with Pyramid at all
Related
Background
I have a relatively simple node js application (essentially just expressjs + mongoose). It is currently running in production on an Ubuntu Server and serves about 20,000 page views per day.
Initially the application was running on a machine with 512 MB memory. Upon noticing that the server would essentially crash every so often I suspected that the application might be running out of memory, which was the case.
I have since moved the application to a server with 1 GB of memory. I have been monitoring the application and within a few minutes the application tends to reach about 200-250 MB of memory usage. Over longer periods of time (say 10+ hours) it seems that the amount keeps growing very slowly (I'm still investigating that).
I have been since been trying to figure out what is consuming the memory. I have been going through my code and have not found any obvious memory leaks (for example unclosed db connections and such).
Tests
I have implemented a handy heapdump function using node-heapdump and I have now enabled --expore-gc to be able to manually trigger garbage collection. From time to time I try triggering a manual GC to see what happens with the memory usage, but it seems to have no effect whatsoever.
I have also tried analysing heapdumps from time to time - but I'm not sure if what I'm seeing is normal or not. I do find it slightly suspicious that there is one entry with 93% of the retained size - but it just points to "builtins" (not really sure what the signifies).
Upon inspecting the 2nd highest retained size (Buffer) I can see that it links back to the same "builtins" via a setTimeout function in some Native Code. I suspect it is cache or https related (_cache, slabBuffer, tls).
Questions
Does this look normal for a Node JS application?
Is anyone able to draw any sort of conclusion from this?
What exactly is "builtins" (does it refer to builtin js types)?
I have a Snap web app which serves some JS files and 1-pixel images (it's main task is to rather work fast then serve huge html/media content). There are several servers behind HAProxy.
I upgraded it from GHC 7.6 to 7.8, also upgrading some libs. After upgrade, app started leaking little by little (on all servers), ending in OOM every 15 minutes on 8GB-RAM machines (and much longer on 16Gb) and restarting afterwards.
The problem is, if I compile app for profiling and run app for some time, I can't see any memory leaks anymore. It just consumes 1 CPU and works in constant small memory.
So I wanted to ask for some general advices on how to find such a bottleneck, if running under profiling doesn't help much.
UPDATE: I noticed after playing with an app that if I remove -A100M runtime option it doen't OOM that fast, but with default value HAProxy's "sessions" gets to it's limit (so, basically it chokes). I'm playing with different RTS options now, hope some will help getting both, performance and long-lived memory consumption.
UPDATE 2: just for the record, I found that with -A30 rts option app, while being memory hungry, lives quite well. 8Gb machines OOM-kill app, but 16Gb one looks like this: http://i.imgur.com/3W9KpFS.png (green line is "RAM available", you can see deploy-procedure which restarted app on a graph). I'm happy with the result, but would be glad to know any techniques to profile memory of multi-threaded app anyway.
UPDATE 3: I'm voting to close this question as "too broad". In general, I see that if such generic set of tools that'll let you profile memory easier would exist, they'd definitely be documented elsewhere on wiki etc.
I have never used it myself but maybe ticky-ticky profiling could help? It's supposed to be immune to the optimization changes caused by ordinary profiling, but at the cost of being harder to interpret.
Basically compile and link the relevant modules with the -ticky and -rtsopts flags, and run with +RTS -rfoo.ticky flag to get heaps of data in foo.ticky.
I have an xpages 9.0.1 server running multiple online form type apps. The server runs fine and performance response is quite good. Pages load fast, users are happy.
Over time (yet to determine how long), the server performance degrades and ultimately grinds to an almost stop.
Each night I am scheduling -c "tel http restart" and it is getting me out of trouble.
I am not sure what page is causing the problem as the degrading happens over a couple of days.
Most of our xpages are SSJS, all of our java (of which there is not much) is appropriately recycled.
It does not seem to be effecting RAM memory - it bounces up a bit and down a bit but well with limits. There is no correlation with the increased response times to more memory used.
So where do I look and what tools can I use to isolate the problem. We are more Dev than Admin.
Cheers
Damien
There are profiling tools available that may help pinpoint which application is causing problems. From OpenNTF, XPages Toolbox is specifically for XPages and was contributed by Philippe Riand, who at the time was Chief Architect for XPages http://www.openntf.org/main.nsf/project.xsp?r=project/XPages%20Toolbox.
There are more heavy-duty, Java-specific tools like YourKit available.
Chapter 20 of Mastering XPages second edition specifically covers performance and there is also a lot of information in XPages Portable Command Guide about performance tuning.
If performance is degrading over time, it could be session timeout. By default, that's 30 minutes. You can extend it, but the danger then is that a browser cannot tell the server it's closing the session when the user closes the browser. So those sessions hang around. Equally if there are very long-running tasks, they would hang around until they complete and the session would then still be active until the timeout.
Are you recycling your SSJS?
If you go into the server tasks of Domino Admin what do you see the CPU usage of the HTTP task doing. Also what is the memory usage of your nHTTP task? You may want to watch that.
Have you gone into the console to see if you can see if there us anything that looks bad?
If you can't pinpoint a problem you may want to think of putting some of your pages on a different server to determine if which app if not all is causing this.
Are you using scoped variables that are session or application scope? Application scope variables stay alive so if you are creating those and have some sort of issue where you end up creating a bunch that can affect memory.
Also there is a server and application setting for when the XPages stay in memory. The suggested setting to Keep only the current page in memory and save the others to disk. This is in the XSP properties.
My Node.Js application occasionally fails with a "Segmentation fault" and I am at a loss as to how to diagnose the cause. As mentioned below I have dramatically reduced their frequency by raising maxlisteners but not made the problem go away.
The application runs on a BeagleBone Black under Node v0.8.22 and uses Socket.io to communicate realtime data to browser pages displayed on the BBB's LCD display. It collects data from a sensor connected via I2C using the korevec/node-i2c library. I have, however, isolated that library and still have failures.
The failures generally occur when I have streamed data for a while to the client though they will occasionally/rarely happen at other times as well. This is not surprising as my app uses socket.io to communicate on almost all pages but the streaming page is at much higher volume.
I am getting the below message:
(node) warning: possible EventEmitter memory leak detected. 11
listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
and have been doing so since day 1. I was also seeing the symptoms of a memory leak but since I raised the MaxListeners memory usage stays constant. Failure rate after making this change went down dramatically but has not gone away completely. I am using socket.io on top of http and raised MaxListeners for both socket.it and for http.
How does one go about diagnosing this problem? Is the memory leak error related? I can post code but there is quite a bit of it and I am not sure what parts are most relevant.
Thanks for any help,
Cheers, Will
I've written an app which is handling videos. As we know, video processing takes a huge amount of memory while dealing with HD resolution. My App always seemed to crash. But actually I am 100% sure, that there is no memory leak in my code. Instruments is showing no leak.
At the beginning I am startin up one OpenGLES view and the video engine. For a very short time the memory consumption is high, but falling down to normal level after the initializations are done. I am always getting memory warnings during this period. Normally this is no problem. But if I have a lot of apps in suspended mode running, the App seems to be crashing. Watching into the crash log and using the debugger shows up, that I am only running out of memory.
My customers are flooding my support mail with "app is crashing" mails. But I do know, that they have too much Apps running in the background, so there is no memory left to go. I think it's bad style programing saying the customer that he has to close Background tasks before running the app.
According to this post this is a common problem.
My question is: Is it possible to tell the OS that one needs a lot of memory so the OS should terminate some suspended Apps? This memory stuff makes me crazy, because it's no bug I could fix.
No. It is not possible to affect anything outside of your sandbox without API calls. None exist for affecting other processes in the public API.
Have you tried to minimize your memory usage? In my experience once a memory warning it thrown apps can be more likely to have problems once they are in the background, even when memory usages drops.
If you are using OpenGLES and textures, if you haven't already compress your textures. What is the specific cause of your memory allocation spike?