The unmanaged memory usage is high when I'm read files and process the data. After manually triggering the memory trimming function, the unmanaged memory usage decrease significantly.
import ctypes
def trim_memory() -> int:
libc = ctypes.CDLL("libc.so.6")
return libc.malloc_trim(0)
client.run(trim_memory)
However, I tried automatically trim memory by:
Set the MALLOC_TRIM_THRESHOLD_ = 0
Modify distributed/distributed.yaml nanny: MALLOC_TRIM_THRESHOLD_=0
Send env to worker when starting client client=Client(env={"MALLOC_TRIM_THRESHOLD_":0})
All these efforts seems fail, the unmanaged memory usage stay on the peak level.
What should be the right way to trim memory?
Reproduced.
Issue tracker and simple workaround until it is fixed:
https://github.com/dask/distributed/issues/5971
Related
I am currently running node.js using pm2.
And recently, I was able to check "custom metrics" using the pm2 monit command.
Here, information such as Heap size, used heap size, and active requests are shown.
I don't know how the heap size is determined. Actually, I checked pm2 running on different servers.
Each was set to 95mib / 55mib, and accordingly, the used heap size was different.
Also, is the heap usage closer to 100% the better?
While searching on "StackOverflow" to find related information, I saw the following article.
What does Heap Usage mean in PM2
Also what means active requests ? It is continuously zero.
Thank you!
[Edit]
env : ubuntu18.04 [ ec2 - t3.micro ]
node version : v10.15
[Additional]
server memory : 1GB [ 40~50% used ]
cpu : vCPU (2) [ 1~2% used ]
The heap is the RAM used by the program you're asking PM2 to manage and monitor. Heap space, in Javascript and similar language runtimes, is allocated when your program creates objects and released upon garbage collection. Your runtime asks your OS for more heap space whenever it needs it: when active allocations exceed the free space. So your heap size will probably grow as your program starts up. That's normal.
Most programs allocate and release lots of objects as they do their work, so you should not try to optimize the % usage of your heap. When your program is running at a steady state – that is, after it has started up — you'll find the % utilization creeping up until garbage collection happens, and then dropping back. For example, a nodejs/express web server allocates req and res objects for each incoming request, then uses them, then drops them so the garbage collector can reclaim their RAM.
If your allocated heap size keeps growing, over minutes or hours, you probably have a memory leak. That is a programming bug: a problem you should do your best to solve. You should look up how that works for your application language. Other than that, don't worry too much about heap usage.
Active requests count work being done via various asynchronous objects like file writers and TCP connections. Unless your program is very busy it stays near zero.
Keep an eye on loop delay if your program does computations. If it creeps up, some computation function is hogging Javascript.
I have been into a little trouble lately: The memory used by GenServer processes is super high, probably because of large binary leaks.
The problem comes from here: we receive large binaries through the GenServer and we pass them to the consumer, which then interacts with that data. Now, these large binaries are never assigned to a variable and the GC doesn't go over them.
I have tried hibernating the processes after managing the data, which partially worked because the memory used by processes lowered a lot, but since binaries were not getting GC'd, the amount of memory used by them increased slowly but steadily, from 30 MBs without hibernating to 200MBs with process hibernation in about 25 minutes.
I have also tried to set :erlang.system_flag(:fullsweep_after, 0), which has also worked and lowered the memory used by processes by around 20%.
Before and after.
I must say it goes down to 60-70MB used by processes from time to time.
Edit: Using :recon.bin_leak(15) frees a lot of memory -- result of :recon.bin_leak(15)
Anyhow the memory used is still high and I'm completely sure it can be fixed.
Here you have a screenshot taken from the observer in the Processes tab. As you can see, GenServer is the one eating the memory like the cookie monster.
I have researched a lot about this topic, tried all the suggestions and possible solutions that were given out there, and nevertheless, I am still in this position.
Any help is welcome.
The code is in this Github Repository
Code of interest that is probably causing this + Applications tree. 3 out of 4 processes there (<0.294.0>, <0.295.0>, <0.297.0> are using 27MB of memory.
Thank you beforehand for reading.
You can try to add the :hibernate atom to your handle_events return values in your GenStage related modules. For example:
def handle_events(events, _from, %{handler: handler, public: public} = state) do
public = handle(handler, events, public)
{:noreply, [], %{state | public: public}, :hibernate}
end
Another option is to record the PIDs after :recon.bin_leak() and then pass them to Process.info(PID) to get some more information about the offending GenServers.
Some additional resources:
https://elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/23
https://www.erlang-in-anger.com/ (Specifically Chapter 7 on Memory Leaks)
I have a base class and a derived class.
Things work well with this setup. I have added another class in base class, so its a nested class.
On allocation of memory for the new nested class, I see some memory corruption.
I want to know when we have nested class, does the size of base class increase ?
Odds are the changes are hiding an application code problem, just as certain memory bugs don't corrupt a running process the same way when run under a debugger, which also changes the environment.
mallopt is used to frob the value however, with this doc:
MALLOC_ARENA_SIZE
The size of the arena, a chunk of memory that the memory allocator allocates
and deallocates from the system. This value must be a multiple of 4 KB, and
currently is limited to being less than 256 KB. Environment variable:
MALLOC_ARENA_SIZE.
But like I said, if changes this looks like it fixes the corruption, don't believe it. Better to set the arena size to whatever value exposes the application bug, then fix the bug itself.
I need to check for a memory leak in an embedded system.
The IDE is HEW and we are using uCOSIII RTOS.
Valgrind does not support the above configurations. Can you please suggest a tool or a method to check for memory leaks?
First rule of dynamically allocating memory in embedded systems is "don't". Allocate it all once at the start of execution and then leave well alone. Otherwise you have to assess and decide what to do when a malloc (or similar operation) fails.
If you must dynamically allocate memory at runtime, then at its simplest you may be able to use a logging infrastructure to track calls to malloc/free by writing wrappers around them. Then you can track where and when the allocations and deallocations are happening and hopefully see what is missing.
Take a look at libtalloc, the core memory allocator used in Samba. It may not work out-of-the-box for you if you don't have atexit() or stdio.h, but it shouldn't take too much work to port it to your environment.
Have a look at talloc_enable_leak_report_full() and talloc_report_full() (among others) to get you started.
I have been giving some thoughts about it, and here is a random try on how to do this with embedded systems:
First you need to check in which thread leakage occur. When doing alloc, you should also count for each thread how many active allocation. Where number of allocation keeps growing without deallocation, this is suspicious task
Secondly, you need to count number of allocations for allocs comming from that thread. To do this, replace alloc with a macro. Using macro you can save name of the file and line number where the call originated.
for example
#define alloc(x) my_alloc(x, __LINE__, __FILE__)
void * my_alloc(size_t size, int line, char * file)
{
// increase number of allocations and dealocations for each combination line/file
}
Similarly you need to define my_free.
After this, run the program and printf from time to time allocations that keep growing. This should help find memory leaks.
P.S. I didn't test this, but I saw somebody do something similar in our code :)
Your requirement is not completely clear. If you are looking for the tool as "valgrind" that can be able find the memory leak in your environment; that is difficult to find out.
If you are having some code than you can check all the memory allocations & freeing of the memory in the particular application. As link1 Link2
Also there are some files available by executing them you can find the memory leak.
http://code.axter.com/debugalloc.cpp
http://code.axter.com/debugalloc.h
http://code.axter.com/debuglogger.cpp
http://code.axter.com/debuglogger.h
http://code.axter.com/debuglog.c
http://code.axter.com/debuglog.h
debugalloc.* code has the ability to track memory leaks, and it has
description and usage information in comments.
debuglogger.* code has some code for profileing your code.
debuglog.* is some limited C version of the code.
What exactly is a memory leak?
And how will it affect the system the program is running on?
When your process allocates memory from the OS on an ongoing basis, and never frees up any of it, you will eventually be using more memory than there is physically in the machine. At this point, the OS will first swap out to virtual memory (degrades performance) if it has any, and at some point your process will reach a point where the OS can no longer grant it more memory, because you've exceeded the maximum amount of addressable space (4GB on a 32bit OS).
There are basically two reasons this can happen: You've allocated memory and you've lost the pointer to it (it has become unreachable to your program), so you cannot free it any longer. That's what most people call a memory leak. Alternatively, you may just be allocating memory and never freeing it, because your program is lazy. that's not so much a leak, but in the end, the problems you get into are the same ones.
A memory leak is when your code allocates memory and then loses track of it, including the ability to free it later.
In C, for example, this can be done with the simple sequence:
void *pointer = malloc (2718); // Alloc, store address in pointer.
pointer = malloc (31415); // And again.
free (pointer); // Only frees the second block.
The original block of memory is still allocated but, because pointer no longer points to it, you have no way to free it.
That sequence, on its own, isn't that bad (well, it is bad, but the effects may not be). It's usually when you do it repeatedly that problems occur. Such as in a loop, or in a function that's repeatedly called:
static char firstDigit (int val) {
char *buff = malloc (100); // Allocates.
if (val < 0)
val = -val;
sprintf (buff, "%d", val);
return buff[0]; // But never frees.
}
Every time you call that function, you will leak the hundred bytes (plus any housekeeping information).
And, yes, memory leaks will affect other things. But the effects should be limited.
It will eventually affect the process that is leaking as it runs out of address space for allocating more objects. While that may not necessarily matter for short-lived processes, long-lived processes will eventually fail.
However, a decent operating system (and that includes Windows) will limit the resources that a single process can use, which will minimise the effects on other processes. Since modern environments disconnect virtual from physical memory, the only real effect that can be carried from process to process is if one tries to keep all its virtual memory resident in physical memory all the time, reducing the allocation of that physical memory to other processes.
But, even if a single process leaks gigabytes of memory, the memory itself won't be being used by the process (the crux of the leak is that the process has lost access to the memory). And, since it's not being used, the OS will almost certainly swap it out to disk and never have to bring it back into RAM again.
Of course, it uses up swap space and that may affect other processes but the amount of disk far outweighs the amount of physical RAM.
Your program will eventually crash. If it does not crash itself, it will help other programs crash because of lack of memory.
When you leak memory, it means that you are dynamically creating objects but are not destroying them. If the leak is severe enough, your program will eventually run out of address space and future allocation attempts will fail (likely causing your application to terminate or crash, since if you are leaking memory, you probably aren't handling out of memory conditions very well either), or the OS will halt your process if it attempts to allocate too much memory.
Additionally, you have to remember that in C++, many objects have destructors: when you fail to destroy a dynamically allocated object, its destructor will not be called.
A memory leak is a situation when a program allocates dynamic memory and then loses all pointers to that memory, therefor it can neither address nor free it. memory remains marked as allocated, so it will never be returned when more memory is requested by the program.
The program will exhaust limited resources at some speed. Depending on the amount of memory and swap file this can cause either the program eventually getting "can't allocate memory" indication or the operating system running out of both physical memory and swap file and just any program getting "can't allocate memory" indication. The latter can have serious consequences on some operating systems - we sometimes see Windows XP completely falling apart with critical services malfunctioning severely once extreme memory consumption in one program exhausts all memory. If that happens the only way to fix the problem is to reboot the system.