I have been running a crypto-intensive application that was generating pseudo-random strings, with special structure and mathematical requirements. It has generated around 1.7 million voucher numbers per node in over the last 8 days. The generation process was CPU intensive, with very low memory requirements.
Mnesia running on OTP-14B02 was the storage database and the generation was done within each virtual machine. I had 3 nodes in the cluster with all mnesia tables disc_only_copies type. Suddenly, as activity on the Solaris boxes increased (other users logged on remotely and were starting webservers, ftp sessions, and other tasks), my bash shell started reporting afork: not enough space error.
My erlang Vms also, went down with this error below:
Crash dump was written to: erl_crash.dump
temp_alloc: Cannot reallocate 8388608 bytes of memory (of type "root_set").
Usually, we get memory allocation errors and not memory re-location errors and normally memory of type "heap" is the problem. This time, the memory type reported is type "root-set".
Qn 1. What is this "root-set" memory?
Qn 2. Has it got to do with CPU intensive activities ? (why am asking this is that when i start the task, the Machine reponse to say mouse or Keyboard interrupts is too slow meaning either CPU is too busy or its some other problem i cannot explain for now)
Qn 3. Can such an error be avoided? and how ?
The fork: not enough space message suggests this is a problem with the operating system setup, but:
Q1 - The Root Set
The Root Set is what the garbage collector uses as a starting point when it searches for data that is live in the heap. It usually starts off from the registers of the VM and off from the stack, if the stack has references to heap data that still needs to be live. There may be other roots in Erlang I am not aware of, but these are the basic stuff you start off from.
That it is a reallocation error of exactly 8 Megabyte space could mean one of two things. Either you don't have 8 Megabyte free in the heap, or that the heap is fragmented beyond recognition, so while there are 8 megabytes in it, there are no contiguous such space.
Q2 - CPU activity impact
The problem has nothing to do with the CPU per se. You are running out of memory. A large root set could indicate that you have some very deep recursions going on where you keep around a lot of pointers to data. You may be able to rewrite the code such that it is tail-calling and uses less memory while operating.
You should be more worried about the slow response times from the keyboard and mouse. That could indicate something is not right. Does a vmstat 1, a sysstat, a htop, a dstat or similar show anything odd while the process is running? You are also on the hunt to figure out if the kernel or the C libc is doing something odd here due to memory being constrained.
Q3 - How to fix
I don't know how to fix it without knowing more about what the application is doing. Since you have a crash dump, your first instinct should be to take the crash dump viewer and look at the dump. The goal is to find a process using a lot of memory, or one that has a deep stack. From there on, you can seek to limit the amount of memory that process is using. either by rewriting the code so it can give memory up earlier, by tuning the garbage collection setup for the process (see the spawn options in the erlang man pages for this), or by adding more memory to the system.
I've set up a script in 3ds max to render a bunch of animations into frames. To do this, I open up a file with all of the materials, load an animation (as a bip) onto the figure, then render.
We were seeing a problem where eventually the script would fail because it was unable to open the next file-- max had consumed all of the system memory. Closing max, of course, freed the memory, and we were able to continue with the script.
I checked out the heapfree variable, hoping to see a memory leak within my script, hoping to see a memory leak within my own (maxscript) code-- but the amount of free space was the same after every animation.
Then, it must be 3ds max which is consuming all of that memory. Nothing in max need be saved from animation to animation-- is there some way to get max to free that memory? (I've tried resetMaxFile() and manually deleting all of the objects in the scene). Is there any known sets of operations that cause max to grow out of control?
Have you tried to add this at the end of your loop:
gc()
it does a garbarge collect and frees up some space.
However I suspect the bip part to be leaky.
The first line of questioning needs to be, do you have any locally-created plugins loaded? Could they be leaking memory?
I haven't worked with 3dsmax since version 5 but I don't remember any particular memory leaks that were problematic. However, I seem to recall (from others' experiences) that batch operations needed to restart MAX from time to time just to keep things sane. E.g. break up your batch job into smaller sets of work and call them sequentially. However, the stuff we were doing in MAX5 didn't need such kludges. YMMV of course. ;)
Autodesk has the Autodesk Developer Network, also; that's a great resource and not too much cash if your company is serious about its use of 3DS.
With RAM typically in the Gigabytes on all PC's now, should I be spending time hunting down all the small (non-growing) memory leaks that may be in my program? I'm talking about those holes that may be less than 64 bytes, or even a bunch that are just 4 bytes.
Some of these are very difficult to identify because they are not in my own code, but may be in third party code or in the development tool's code, and I may not even have direct access to the source. In those cases, it would involve lengthy communication with the vendors of these products.
I have seen the number one memory leak question here at SO: Are memory leaks ever ok? and the number one answer to that, as of now voted up 85 times, is: No.
But here I'm talking about small leaks that may take an inordinate amount of debugging, research and communication to track down.
And I'm only talking about a simple desktop app. I understand that apps running on servers must be as tight as possible.
So the question I am really asking is, if I know I have a program that leaks, say 40 bytes every time it is run, does that matter?
(source: beholdgenealogy.com)
Also see my followup question: What Operating Systems Will Free The Memory Leaks?
Postscript: I just purchased EurekaLog for my program development.
I found an excellent article by Alexander, the author of EurekaLog (who should know these things), about catching memory leaks. In that article, Alexander states the answer to my question very well and succinctly:
While any error in your application is always bad, there are types of errors, which can be not visible in certain environments. For example, memory or resources leaks errors are relatively harmless on client machines and can be deadly on servers.
This is completely a personal decision.
However, if:
So the question I am really asking is, if I know I have a program that leaks, say 40 bytes every time it is run, does that matter?
In this case, I'd say no. The memory will be reclaimed when the program terminates, so if it's only leaking 40 bytes one time during the operation of an executable, that's practically meaningless.
If, however, it's leaking 40 bytes repeatedly, each time you do some operation, that might be more meaningful. The longer running the application, the more significant that becomes.
I would say, though, that fixing memory leaks often is worthwhile, even if the leak is a "meaningless" leak. Memory leaks are typically indicators of some underlying problem, so understanding and correcting the leak will often make your program more reliable over time.
Leaks are bugs.
You probably have other bugs too.
When you ship a product, you ship it with known bugs. When you choose which (of the known) bugs to "fix" versus "ship with", you do so based on the cost and risk to fix versus the customer benefit.
Leaks are no different. If it's a small leak that happens during an infrequent operation in a non-server app (e.g. an app that runs for minutes or hours and then shuts down), it might be "ok" in the same way any other bug is ok.
Actually, leaks can be kinda different in one important way, which is that if you are shipping a library/API, you really should fix them, because the customer benefit is enormous (otherwise all your customer 'inherit' your leak, and will be phoning you just as you have to do to talk to 3rd party vendor now).
While I agree that every little leak adds up, I don't agree that it's always the best business decision to fix it.
What if you have a stateless legacy system and no coders who understand it? Now you are using it in a situation that has to scale... and it's 100X cheaper to spawn a new instance and swap them out before memory goes overboard.
Or let's say you have a batch processing system that runs 24x7 but for which there is no real user. If it's cheaper to monitor memory and tell the system to restart itself periodically, why hunt down the leak?
I think you should try real hard but be pragmatic about the business ramifications of the decision.
No, it does not matter, however, only if, as you pointed out, the memory leak must not be repetitive. Memory leaks that don't grow as a program progress is usually okay. Non-growing memory leaks will eventually be solved when a process terminate.
However, it is difficult to prove an observed memory leak is not growing; you have sufficient empirical data. In reality, many huge program (even written in Java/C#) have memory leaks, but most of them are non-growing leaks.
Seriously, we can't live without memory leaks, deadlocks, data races. Having these bugs itself are okay. Only when it kills your program, it matters.
But, I have to disagree with your opinion: "memory is cheap". That can't justify memory leaks. That's very dangerous.
Yes. Leaks matter. If your apps runs 24x7x365 and handles a few thousands transactions per second, a few bytes turns into gigabytes rapidly.
A memory leak really depends on several things:
How often the leak happens
How much memory is lost each time
How long is the program going to run
For example, if you lose 40 bytes every time a task happens, and that task happens when the program starts, then nobody cares. If you lose 40Mb every time the program starts, then it should be investigated. If you lose 40 bytes every frame in your video or game engine, then you should look into that, because you'll lose 1.2kB each second, and after an hour you would have lost almost 4Mb.
It also depends on how long the program is going to stick around for. For example, I have a small calculator app, that I open, run a calculation in, and then close again. If that app loses 4Mb in it's run, then it doesn't really matter, because the OS will reclaim that lost memory once I close it. If the hypothetical video/game engine mentioned earlier lost 4Mb an hour, and it ran a demo unit, for several hours a day at a stand at a convention, then I'd look into it.
An example of a famous memory leak is Firefox, which lost a lot of memory in it's earlier versions. If your browser leaked memory 10 years ago, then you probably wouldn't care. You shut down the computer every day, and you while running the browser you only had one page up at a time. Today I just let my laptop go to standby, and I never close Firefox. It is open for weeks at a time, and I have at least 10 tabs open at any given time. If memory leaks every time a tab is closed, then that is going to build up to a larger leak now than it did 10 years ago, and so it is more important.
Are memory leaks ever ok?
Sure, if it's a short-lived process.
Memory leaks over a long period of time are, as the 85-point answer implies, problematic. Take a simple desktop app, for example -- prior to versions 3.x, did you ever notice how you needed it reboot Firefox after a while to recover it from sluggishness?
As for the short term, no, it doesn't matter. Take CGI or PHP scripts for example, or the little Perl three-liner in your ~/bin directory. Nobody's going to call the memory police if you write a 30-line non-looping application in C with 5 lines of malloc() and not a single call to free().
I am in the same boat as you. I have small memory leaks that don't grow ever. Most of the leaks are caused by improperly tearing down COM objects. I have studied the leaks and come to realize the time and money to fix them is disproportional to the damage the leaks do. Windows cleans up most of the time so the true damage is only realized if the user runs his computer for years without rebooting.
I think it's acceptable to leave in the leaks. It sounds so taboo, but if the leaks never ever ever grow and they are small, it's pretty insignificant in the larger scheme of things.
I agree with the earlier responses that leaks do matter.
People may have tons of memory, but they are also running more and more programs, and unless your application is completely hogging up the processor, it needs to play nice with other programs, which also means not hogging up resources it doesn't need.
So, this small memory leak will add up and mean that the user will have other problems, and if they decide they are having memory issues, if they decide that running your app causes them problems then they will stop running it.
Besides, as has been pointed out, if you don't know what is causing the leak then you may have other problems you don't know about. It may be the tip of a bug iceberg.
It depends on the nature of your application. I work primarily with web sites and web applications. So by most definitions, my application "runs" once per request. Code that leaks a few bytes per request on a high volume site can be catastrophic. From experience, we had some code which leaked a few kb per request. Added up, that caused our web server worker processes to restart so often it caused minute-long outages throughout the day.
But web applications (and many other kinds) have an indefinite lifespan - they run continuously, forever. The shorter-lived your application, the better. If each session of your application has a finite and reasonably predictable end point, there's of course a reasonable amount of leakage you can tolerate. It's all about the ROI.
It all depends. Reasons not to worry: the process is short-lived, the leaks are small and/or infrequent, the cost of an out of memory exception is low (eg, a web server instance in a cluster needs restarting and a few fetches need retrying). So I agree that some leaks don't really matter in practical terms.
But on the other hand, if you do have cause to worry, or even feel a nagging sense of doubt that maybe you're not taking quality seriously enough, it's a small matter (in most cases) to run your software with a memory leak detector and fix the problems. There are many good leak detectors out there. And you might find that the leak is part of a more serious problem, such as not releasing other resources (like open files). You may even find that the harmless leak would turn quite dangerous in usage scenarios you haven't tested yet.
Yes, it matters. Every little leak adds up.
For one, if your leaky code is used in a context where it is repeatedly used, and it leaks a little bit each time, those little bits add up. Even if the leak is small, and infrequent, those things can add up to significant quantities over long periods of time.
Secondarily... if you're writing code that has memory leaks, then that code has problems. I'm not saying that good code doesn't from time to time have memory leaks, but the fact of their existence means that there are some serious problems going on. Many, many security holes are due to just this sort of oversight (unbounded string copy, anyone?).
Bottom line is, if you know about it, and don't do all you can to track it down and fix it, then you're causing problems.
Memory leaks are never OK in any program, however small it may be.
Slowly they will add up to fill up your entire memory. Suppose you have a calling system which leaks about 4 bytes of memory per call it handles. You can handle say, 100 calls a second (this is a very small number), so you end up leaking 400 bytes a second or 400x60x60(1440000B) an hour. So, even a small leak is not acceptable.
And if you dont know the source of the leak then it may be some real big issue and you end up having buggy software.
But, essentially it boils down to the questions like, for how much time the program runs and what number of times the leak happens. So, it may be ok it leaks a very small amount and is not repeated but still the leak may be a small part of a bigger problem.
So, I feel that memory leaks are never ok.
That's like asking if there was a crack in a dam is it ok? NO! NEVER! Avoid memory leaks as if your life depends on it because when your application grows and gets more use that leak is going to turn into a flood and sooner or later someone or something is going to drown in it. Just because you can have lots of memory doesn't mean that you can take shortcuts with your code. Once you find a leak do what you can to resolve it and if you can't solve it make sure you keep coming back to it until it's fixed.
If you can't resolve the leak then try to see if you can clean up after it. The bigger issues come when the leak is repetitive.
Last note: if you ever hand the software to someone else and that leak is still there it may be a long time before someone else finds and/or fixes it.
I wouldn't be so worried about the quantity but the frequency of memory which you leak, but if you leak even just a few bytes very very often, your malloc's data structures will grow and might make it dramatically slower to traverse them, to allocate new memory and free. Unless you hit the border where you have leaked more than a tiny fraction of your RAM, mainly your program will suffer under those performance problems and not the whole system. Does not apply to even remotely dlmalloc-based systems (FreeBSD, Linux, etc), there it's just don't care, all you loose there is memory (perhaps a few times more than the amount you think) and not performance.
A single allocation which is not reclaimed by your program is not a leak at all. If you write a small command line utility which takes a second to complete, you may not need to even reclaim any memory there. Upon termination, the OS reclaims RAM, file handles, should basically apply to any kind of system resource, but you cannot rely on some OSes as much as on others, but as long as it's just memory, even Windows 95 will manage it just right.
Oh and another thing follows from that, if you leak memory, don't bother cleaning up at the end of the program or after a long execution time, or you will just waste even more CPU time. Always fix the leaks as near to the timepoint where they are created as possible. Other reason: malloc implementations prefer to keep the RAM they got from the OS for future allocations instead of giving it back. Also you may suffer address space fragmentation.
If someone says memory leaks are ok in small amounts and as long as it doesn't crash the application, it is like saying, stealing is ok if in small amounts and as long as you are not caught :)
Memory leaks are very important in 32 bit applications because the application is limited to 2^32 bytes of memory which is approximately 4 GB. Once a 32 bit application attempts to allocate more than 2^32 bytes of memory the application may crash or hang.
Memory leaks are not as important in 64 bit applications because the application is limited to 2^64 bytes of memory which is approximately 16 EB; so with a 64 bit application you are more-so limited by hardware, but the OS will still likely impose an artificial limit at some time.
Bottom line is that having a memory leak in your code is bad programming; so fix it and be a better programmer.