With RAM typically in the Gigabytes on all PC's now, should I be spending time hunting down all the small (non-growing) memory leaks that may be in my program? I'm talking about those holes that may be less than 64 bytes, or even a bunch that are just 4 bytes.
Some of these are very difficult to identify because they are not in my own code, but may be in third party code or in the development tool's code, and I may not even have direct access to the source. In those cases, it would involve lengthy communication with the vendors of these products.
I have seen the number one memory leak question here at SO: Are memory leaks ever ok? and the number one answer to that, as of now voted up 85 times, is: No.
But here I'm talking about small leaks that may take an inordinate amount of debugging, research and communication to track down.
And I'm only talking about a simple desktop app. I understand that apps running on servers must be as tight as possible.
So the question I am really asking is, if I know I have a program that leaks, say 40 bytes every time it is run, does that matter?
(source: beholdgenealogy.com)
Also see my followup question: What Operating Systems Will Free The Memory Leaks?
Postscript: I just purchased EurekaLog for my program development.
I found an excellent article by Alexander, the author of EurekaLog (who should know these things), about catching memory leaks. In that article, Alexander states the answer to my question very well and succinctly:
While any error in your application is always bad, there are types of errors, which can be not visible in certain environments. For example, memory or resources leaks errors are relatively harmless on client machines and can be deadly on servers.
This is completely a personal decision.
However, if:
So the question I am really asking is, if I know I have a program that leaks, say 40 bytes every time it is run, does that matter?
In this case, I'd say no. The memory will be reclaimed when the program terminates, so if it's only leaking 40 bytes one time during the operation of an executable, that's practically meaningless.
If, however, it's leaking 40 bytes repeatedly, each time you do some operation, that might be more meaningful. The longer running the application, the more significant that becomes.
I would say, though, that fixing memory leaks often is worthwhile, even if the leak is a "meaningless" leak. Memory leaks are typically indicators of some underlying problem, so understanding and correcting the leak will often make your program more reliable over time.
Leaks are bugs.
You probably have other bugs too.
When you ship a product, you ship it with known bugs. When you choose which (of the known) bugs to "fix" versus "ship with", you do so based on the cost and risk to fix versus the customer benefit.
Leaks are no different. If it's a small leak that happens during an infrequent operation in a non-server app (e.g. an app that runs for minutes or hours and then shuts down), it might be "ok" in the same way any other bug is ok.
Actually, leaks can be kinda different in one important way, which is that if you are shipping a library/API, you really should fix them, because the customer benefit is enormous (otherwise all your customer 'inherit' your leak, and will be phoning you just as you have to do to talk to 3rd party vendor now).
While I agree that every little leak adds up, I don't agree that it's always the best business decision to fix it.
What if you have a stateless legacy system and no coders who understand it? Now you are using it in a situation that has to scale... and it's 100X cheaper to spawn a new instance and swap them out before memory goes overboard.
Or let's say you have a batch processing system that runs 24x7 but for which there is no real user. If it's cheaper to monitor memory and tell the system to restart itself periodically, why hunt down the leak?
I think you should try real hard but be pragmatic about the business ramifications of the decision.
No, it does not matter, however, only if, as you pointed out, the memory leak must not be repetitive. Memory leaks that don't grow as a program progress is usually okay. Non-growing memory leaks will eventually be solved when a process terminate.
However, it is difficult to prove an observed memory leak is not growing; you have sufficient empirical data. In reality, many huge program (even written in Java/C#) have memory leaks, but most of them are non-growing leaks.
Seriously, we can't live without memory leaks, deadlocks, data races. Having these bugs itself are okay. Only when it kills your program, it matters.
But, I have to disagree with your opinion: "memory is cheap". That can't justify memory leaks. That's very dangerous.
Yes. Leaks matter. If your apps runs 24x7x365 and handles a few thousands transactions per second, a few bytes turns into gigabytes rapidly.
A memory leak really depends on several things:
How often the leak happens
How much memory is lost each time
How long is the program going to run
For example, if you lose 40 bytes every time a task happens, and that task happens when the program starts, then nobody cares. If you lose 40Mb every time the program starts, then it should be investigated. If you lose 40 bytes every frame in your video or game engine, then you should look into that, because you'll lose 1.2kB each second, and after an hour you would have lost almost 4Mb.
It also depends on how long the program is going to stick around for. For example, I have a small calculator app, that I open, run a calculation in, and then close again. If that app loses 4Mb in it's run, then it doesn't really matter, because the OS will reclaim that lost memory once I close it. If the hypothetical video/game engine mentioned earlier lost 4Mb an hour, and it ran a demo unit, for several hours a day at a stand at a convention, then I'd look into it.
An example of a famous memory leak is Firefox, which lost a lot of memory in it's earlier versions. If your browser leaked memory 10 years ago, then you probably wouldn't care. You shut down the computer every day, and you while running the browser you only had one page up at a time. Today I just let my laptop go to standby, and I never close Firefox. It is open for weeks at a time, and I have at least 10 tabs open at any given time. If memory leaks every time a tab is closed, then that is going to build up to a larger leak now than it did 10 years ago, and so it is more important.
Are memory leaks ever ok?
Sure, if it's a short-lived process.
Memory leaks over a long period of time are, as the 85-point answer implies, problematic. Take a simple desktop app, for example -- prior to versions 3.x, did you ever notice how you needed it reboot Firefox after a while to recover it from sluggishness?
As for the short term, no, it doesn't matter. Take CGI or PHP scripts for example, or the little Perl three-liner in your ~/bin directory. Nobody's going to call the memory police if you write a 30-line non-looping application in C with 5 lines of malloc() and not a single call to free().
I am in the same boat as you. I have small memory leaks that don't grow ever. Most of the leaks are caused by improperly tearing down COM objects. I have studied the leaks and come to realize the time and money to fix them is disproportional to the damage the leaks do. Windows cleans up most of the time so the true damage is only realized if the user runs his computer for years without rebooting.
I think it's acceptable to leave in the leaks. It sounds so taboo, but if the leaks never ever ever grow and they are small, it's pretty insignificant in the larger scheme of things.
I agree with the earlier responses that leaks do matter.
People may have tons of memory, but they are also running more and more programs, and unless your application is completely hogging up the processor, it needs to play nice with other programs, which also means not hogging up resources it doesn't need.
So, this small memory leak will add up and mean that the user will have other problems, and if they decide they are having memory issues, if they decide that running your app causes them problems then they will stop running it.
Besides, as has been pointed out, if you don't know what is causing the leak then you may have other problems you don't know about. It may be the tip of a bug iceberg.
It depends on the nature of your application. I work primarily with web sites and web applications. So by most definitions, my application "runs" once per request. Code that leaks a few bytes per request on a high volume site can be catastrophic. From experience, we had some code which leaked a few kb per request. Added up, that caused our web server worker processes to restart so often it caused minute-long outages throughout the day.
But web applications (and many other kinds) have an indefinite lifespan - they run continuously, forever. The shorter-lived your application, the better. If each session of your application has a finite and reasonably predictable end point, there's of course a reasonable amount of leakage you can tolerate. It's all about the ROI.
It all depends. Reasons not to worry: the process is short-lived, the leaks are small and/or infrequent, the cost of an out of memory exception is low (eg, a web server instance in a cluster needs restarting and a few fetches need retrying). So I agree that some leaks don't really matter in practical terms.
But on the other hand, if you do have cause to worry, or even feel a nagging sense of doubt that maybe you're not taking quality seriously enough, it's a small matter (in most cases) to run your software with a memory leak detector and fix the problems. There are many good leak detectors out there. And you might find that the leak is part of a more serious problem, such as not releasing other resources (like open files). You may even find that the harmless leak would turn quite dangerous in usage scenarios you haven't tested yet.
Yes, it matters. Every little leak adds up.
For one, if your leaky code is used in a context where it is repeatedly used, and it leaks a little bit each time, those little bits add up. Even if the leak is small, and infrequent, those things can add up to significant quantities over long periods of time.
Secondarily... if you're writing code that has memory leaks, then that code has problems. I'm not saying that good code doesn't from time to time have memory leaks, but the fact of their existence means that there are some serious problems going on. Many, many security holes are due to just this sort of oversight (unbounded string copy, anyone?).
Bottom line is, if you know about it, and don't do all you can to track it down and fix it, then you're causing problems.
Memory leaks are never OK in any program, however small it may be.
Slowly they will add up to fill up your entire memory. Suppose you have a calling system which leaks about 4 bytes of memory per call it handles. You can handle say, 100 calls a second (this is a very small number), so you end up leaking 400 bytes a second or 400x60x60(1440000B) an hour. So, even a small leak is not acceptable.
And if you dont know the source of the leak then it may be some real big issue and you end up having buggy software.
But, essentially it boils down to the questions like, for how much time the program runs and what number of times the leak happens. So, it may be ok it leaks a very small amount and is not repeated but still the leak may be a small part of a bigger problem.
So, I feel that memory leaks are never ok.
That's like asking if there was a crack in a dam is it ok? NO! NEVER! Avoid memory leaks as if your life depends on it because when your application grows and gets more use that leak is going to turn into a flood and sooner or later someone or something is going to drown in it. Just because you can have lots of memory doesn't mean that you can take shortcuts with your code. Once you find a leak do what you can to resolve it and if you can't solve it make sure you keep coming back to it until it's fixed.
If you can't resolve the leak then try to see if you can clean up after it. The bigger issues come when the leak is repetitive.
Last note: if you ever hand the software to someone else and that leak is still there it may be a long time before someone else finds and/or fixes it.
I wouldn't be so worried about the quantity but the frequency of memory which you leak, but if you leak even just a few bytes very very often, your malloc's data structures will grow and might make it dramatically slower to traverse them, to allocate new memory and free. Unless you hit the border where you have leaked more than a tiny fraction of your RAM, mainly your program will suffer under those performance problems and not the whole system. Does not apply to even remotely dlmalloc-based systems (FreeBSD, Linux, etc), there it's just don't care, all you loose there is memory (perhaps a few times more than the amount you think) and not performance.
A single allocation which is not reclaimed by your program is not a leak at all. If you write a small command line utility which takes a second to complete, you may not need to even reclaim any memory there. Upon termination, the OS reclaims RAM, file handles, should basically apply to any kind of system resource, but you cannot rely on some OSes as much as on others, but as long as it's just memory, even Windows 95 will manage it just right.
Oh and another thing follows from that, if you leak memory, don't bother cleaning up at the end of the program or after a long execution time, or you will just waste even more CPU time. Always fix the leaks as near to the timepoint where they are created as possible. Other reason: malloc implementations prefer to keep the RAM they got from the OS for future allocations instead of giving it back. Also you may suffer address space fragmentation.
If someone says memory leaks are ok in small amounts and as long as it doesn't crash the application, it is like saying, stealing is ok if in small amounts and as long as you are not caught :)
Memory leaks are very important in 32 bit applications because the application is limited to 2^32 bytes of memory which is approximately 4 GB. Once a 32 bit application attempts to allocate more than 2^32 bytes of memory the application may crash or hang.
Memory leaks are not as important in 64 bit applications because the application is limited to 2^64 bytes of memory which is approximately 16 EB; so with a 64 bit application you are more-so limited by hardware, but the OS will still likely impose an artificial limit at some time.
Bottom line is that having a memory leak in your code is bad programming; so fix it and be a better programmer.
Related
I am experiencing a 'slow' memory increase in my node process which runs for longer periods of time (~1GB in over 2 months), however the heap stays constant (which implies that my code/stack is growing). I also tried to manually call the garbage collector but memory usage remains the same.
How can I investigate this further ? I want to confirm my theory and figure out why is my code segment / stack part growing.
I am using node v8 LTS (I know this is EOL from this year, I just need to know if there's a way to figure this out)
(V8 developer here.)
Code generated by V8 is on the heap, so if the heap isn't growing, that means that code isn't growing either.
The stack size is limited by the operating system, usually to 1-8 MB. Since operating systems simply kill processes that run into the stack limit, V8 imposes an even lower limit (a little less than a megabyte, I think it's 984KB currently) onto itself, and will throw a RangeError if that's ever exceeded. So a growing stack can't be your problem either.
Since you say that the heap memory reported by Node/V8 remains constant, that also means that most "how to debug memory leaks in Node" tutorials don't apply to your situation; and that probably also means that the leak is not in your (JavaScript) code.
That leaves C++ "heap memory" (which is very different from V8's managed "heap"!) as the most likely culprit. Node itself as well as native extensions can freely allocate memory that they manage themselves. Maybe something doesn't get cleaned up properly there. That could simply be an upstream bug; or it could be that something in your code is accidentally holding on to some embedder memory.
What I would try first is to update Node and any native extensions you have installed. Maybe the leak has already been found and fixed.
If that doesn't help, then you could try to investigate where the memory is going. For instance, you could compile everything from source with LSan enabled, and see if that reports anything. It would probably be helpful to construct a stress-test, e.g. a fake client that floods (a test instance of) your server with real-looking requests, to try to trigger inspectable instances of the leak in seconds or minutes rather than months. Crafting such a fake client might also help narrow down where things go wrong (e.g.: maybe you'll notice that one type of request does not trigger the leak but another type of request does).
After full gc happens, we may want to know how it happens. Without heap dump, I think it is hard to do, but in production environment, we usually can't get it in time. So I want to use HeapDumpBeforeFullGC in my application when it runs online.
My question is should HeapDumpBeforeFullGC be used in product environment? Will it bring some bad effects(if we don't consider disk usage)?
Or do we have other effective way to find what cause full gc in production environment?
Thanks!
If you consider a full GC a problem in production then yes, adding a heap dump may help. But it will make the pause times on a full GC even worse.
As alternatives you can turn on detailed GC logging which are often a good start to identify the general cause (insufficient heap size, leaks, allocation spikes, misconfiguration, swapping, ...). You can also use less invasive profilers (e.g. async-profiler or jmc) to spot excessive allocations
We have a a very large project which is basically an application which uses Linux Application programming and runs on PowerPC processor. This project was initially developed by another company. We acquired the project from the company and now we are maintaining the project.
The application is reported to have a lot of memory leak issue. Since this is a large project, it is not possible to go to each source code file and find out the memory leak. We have used Valgrid, mpatrol and other memory leak detection tools. These tools did not help much and the memory leak has not decreased by a significant percentage.
In this situation, how to go about to reduce the memory leak by a significant amount.Is there a general method which people use in these case to reduce the memory leak other than the memory leak detection tools like mentioned above.
Usually Valgrind belongs to the best tools for this tasks. If it does not work correctly, there might only be a couple of things you can still do.
First question: What language is the application in? Valgrind is very good for C and C++, but will not help you with garbage collected or scripting language. So check the language first. There might be something similar for java, but I have not used that much java, so you would have to ask someone else.
Play around a lot with the settings of valgrind. There are several plugins, that can help with this. One example could be using --leak-check=full or similar options. There are also plugins for valgrind, that can enhance it detection capabilities.
You say, that the application was reported to have a memory leak. How was this detected? Did the application detect this by itself. If it was detected by the application on it's own without any external tools, this probably means someone has added their own memory tracker inside the application. Custom memory tracker, memory pools etc. mess up valgrind and any other leak detection system very bad. So in case any custom memory handling is present in the application, your only choice is to either deactivate it (if possible) or to hook into this custom mechanism. How this could be done depends on your application only.
Add your own memory tracker. For example in C++ it is possible to hook into new/delete calls and get them to track the memory. There are a couple of libraries you can use for this. You can also write your own new/delete replacement in about 500 LOC. If you decide to use this method, be sure to read a lot of tutorials on replacing new/delete, since there are several things that are unusual in the C++ world when attempting this task.
What makes you so sure, there is an memory leak in the application (i.e. how was this detected)? If a tool just reported huge numbers of allocated memory, this might not even mean, there is an actual memory leak. A memory leak means that the handles to the memory are lost and hence it becomes impossible to every reach and free that memory again. In case your application just get's a lot of memory and keeps it accessible, you probably have a completely different problem. For example you simply might use an algorithm with a bad space complexity at one point or the other, leading to many allocations. In this case you will not need a leak detector, but rather a memory profiler, which gives you more detailed overview of the memory footprint of the code parts. However I have never used a profiler for this kind of task before, so I cannot give you any more hints on this.
You could replace all memory allocation calls with calls to your own allocation methods, which should call original methods and at the same time count memory usage and where it was allocated. This will allow you to find the leaks and eliminate them by hand.
There might also be automated tools that allow you to do this - not sure, haven't used any. But this method works.
Perhaps you might also consider using Boehm's garbage collector (that is using GC_malloc instead of malloc etc... and not bother about free-ing data).
My .NET application has memory leak. Few people seem to recommend using CLR Profiler for this pupose I am a bit lost on the idea. To me in order to find a memory leak, tool should compare two memory states that can give you statistics like growth in objects between two states. So in my mind, if a tool cannot compare two (or more) memory states, it cannot be used for detecting memroy leak. Obviously things like performance counters is bit different concept where you can trend the memory usage.
So my question is really if someone can explain how exactly CLR Profiler can be used to detect memory leaks?
Well it depends on what kind of memory leak you have.
We had a reproducible one, where we new that a certain chain of events should always leave a clean table after work was done - but it wasn't.
So we simple setup a test where we did it a couple of thousand times - then we looked at those objects (bigger in number) in the heap graph and at the "root"-object the cause of why the objects where still alive. It helped to solve our problem...
When we say a program leaks memory, say a new without a delete in c++, does it really leak? I mean, when the program ends, is that memory still allocated to some non-running program and can't be used, or does the OS know what memory was requested by each program, and release it when the program ends? If I run that program a lot of times, will I run out of memory?
No, in all practical operating systems, when a program exits, all its resources are reclaimed by the OS. Memory leaks become a more serious issue in programs that might continue running for an extended time and/or functions that may be called often from the same program.
On operating systems with protected memory (Mac OS 10+, all Unix-clones such as Linux, and NT-based Windows systems meaning Windows 2000 and younger), the memory gets released when the program ends.
If you run any program often enough without closing it in between (running more and more instances at the same time), you will eventually run out of memory, regardless of whether there is a memory leak or not, so that's also true of programs with memory leaks. Obviously, programs leaking memory will fill the memory faster than an identical program without memory leaks, but how many times you can run it without filling the memory depends much rather on how much memory that program needs for normal operation than whether there's a memory leak or not. That comparison is really not worth anything unless you are comparing two completely identical programs, one with a memory leak and one without.
Memory leaks become the most serious when you have a program running for a very long time. Classic examples of this is server software, such as web servers. With games or spreadsheet programs or word processors, for instance, memory leaks aren't nearly as serious because you close those programs eventually, freeing up the memory. But of course memory leaks are nasty little beasts which should always be tackled as a matter of principle.
But as stated earlier, all modern operating systems release the memory when the program closes, so even with a memory leak, you won't fill up the memory if you're continuously opening and closing the program.
Leaked memory is returned by the OS after the execution has stopped.
That's why it isn't always a big problem with desktop applications, but its a big problem with servers and services (they tend to run long times.).
Lets look at the following scenario:
Program A ask memory from the OS
The OS marks the block X as been used by A and returns it to the program.
The program should have a pointer to X.
The program returns the memory.
The OS marks the block as free. Using the block now results in a access violation.
Program A ends and all memory used by A is marked unused.
Nothing wrong with that.
But if the memory is allocated in a loop and the delete is forgotten, you run into real problems:
Program A ask memory from the OS
The OS marks the block X as been used by A and returns it to the program.
The program should have a pointer to X.
Goto 1
If the OS runs out of memory, the program probably will crash.
No. Once the OS finishes closing the program, the memory comes back (given a reasonably modern OS). The problem is with long-running processes.
When the process ends, the memory gets cleared as well. The problem is that if a program leaks memory, it will requests more and more of the OS to run, and can possibly crash the OS.
It's more leaking in the sense that the code itself has no more grip on the piece of memory.
The OS can release the memory when the program ends. If a leak exists in a program then it is just an issue whilst the program is running. This is a problem for long running programs such as server processes. Or for example, if your web browser had a memory leak and you kept it running for days then it would gradually consume more memory.
As far as I know, on most OS when a program is started it receives a defined segment of memory which will be completely liberated once the program is ended.
Memory leaks are one of the main reason why garbage collector algorithms were invented since, once plugged into the runtime, they become responsible in reclaiming the memory that is no longer accessible by a program.
Memory leaks don't persist past end of execution so a "solution" to any memory leak is to simply end program execution. Obviously this is more of an issue on certain types of software. Having a database server which needs to go offline every 8 hours due to memory leaks is more of an issue than a video game which needs to be restarted after 8 hours of continual play.
The term "leak" refers to the fact that over time memory consumption will grow without any increased benefit. The "leaked" memory is memory neither used by the program nor usable by the OS (and other programs).
Sadly memory leaks are very common in unmanaged code. I have had firefox running for a couple days now and memory usage is 424MB despite only having 4 tabs open. If I closed firefox and re-opened the same tabs memory usage would likely be <100MB. Thus 300+ MB has "leaked".