Is there any way to increase the heap size for Java ME on a device? I'm developing an application for a Nokia N95, but am bumping into memory issues when I try to do image processing of larger images. The phone has plenty of heap space available, but seems to restrict the amount allowed to be used by Java ME to 1MB.
Short answer: No.
Longer answer: Heap-size is hardcoded in device VM. If you are running out of memory you need to split you task into more smaller sub tasks to allow GC to kick in. With limited memory only limited operations are possible. You figure...
On some J2ME handsets setting Midlet-Heap-Size jad attributes works. Say a handset has only 2MB heap, setting this jad attribute to say 3MB increases the Heap to 3MB. You can try this out, it worked on very few handsets for me though.
Related
Recently we had memory spike issue in one of our .Net core 2.0 application. When we analyze the memory dump, we found around 2GB of free space which is not released.
Following is the statistics from windbg tool.
Statistics:
MT Count TotalSize Class Name
000000e32eaf4ca0 15404 2584481422 Free
Can you please let me know what could be the reason for memory not getting released?
At least for the following reasons:
.NET thinks that your program needs that memory again soon
The free parts of the memory do not make up a complete block to give back to the OS
Regarding 1.), .NET might keep the memory which it got from the operating system, because giving it back to the OS and then reclaiming it from the OS has an impact on performance. If .NET thinks your program will request memory in the near future, it could simply "cache" it for you.
Regarding 2.), that's best explained in an example. Assume that your application requested 10 MB. .NET might take a 50 MB block from the OS. The application then requests 1 MB. .NET puts it into the existing 50 MB block. The 10 MB get freed. However, .NET cannot give 49 MB back to the OS, because giving memory back to the OS can only be done at the same size in which it was given.
.NET operates directly on VirtualAlloc() and VirtualFree(), therefore the possibilities are somewhat limited.
If you look at the Count column, you'll see that your 2.5 GB are not contiguous. It is split into 15000 fragments. Your program might suffer from memory fragmentation. Memory fragmentation can occur on the small object heap due to pinned handles and on the large object heap because that one does not get compacted.
I've boosted up the RAM size allocation for Android Studio to about 4000MB.
But What it actually helps in? I mean, i know it helps in handling executions which needs more RAM to open/compile but if my android project (let's say) needs 1000MB max RAM (i.e a light project) and my current allocated RAM is 4000MB, does it makes the AndroidStudio faster or the speed will be the same?
At a point it caps off. I use the default memory amount (700 and something MB). At a point it isn't about how much ram you use, it is about how good the processor is. If the logic is too heavy for the CPU, it will take a long time no matter how much ram you give it.
IMO, 4 gigabyte is too much. You just allocate a ton of RAM you may need somewhere else, which slows down the other programs. Giving it 2 may be fine, but you don't need to give it 4 gigs unless you are running extremely heavy Gradle tasks that makes 700 MB unreasonably low. RAM is mostly memory allocation for fields, the rest is on the CPU (or GPU for applicable programs). If you don't have a CPU that is good enough, adding more RAM isn't going to help.
"One topic you might hear people discussing when they're talking shop about computers is how much random access memory (RAM) they need to add to their computer. Up to a point, adding RAM will normally cause your computer to seem faster on certain types of operations. RAM is important because it eliminates the need to "swap" programs in and out." (source)
So it only works up to a certain point, which varies. You need a certain amount of RAM depending on what you do in Android Studio, but you don't need 4 gigs. The speedup as a result of giving a program more RAM gets lower the more you give it, and eventually there is no boost.
My project has started using java 8 from java 7.
After switching to java 8, we are seeing issues like the memory consumed is getting higher with time.
Here are the investigations that we have done :
Issues comes only after migrating from java7 and from java8
As metaspace is the only thing related to memory which is changes from hava 7 to java 8. We monitored metaspace and this does not grow more then 20 MB.
Heap also remains consistent.
Now the only path left is to analyze how the memory gets distributes to process in java 7 and java 8, specifically private byte memory. Any thoughts or links here would be appreciated.
NOTE: this javaw application is a swing based application.
UPDATE 1 : After analyzing the native memory with NMT tool and generated a diff of memory occupied as compare to baseline. We found that the heap remained same but threads are leaking all this memory. So as no change in Heap, I am assuming that this leak is because of native code.
So challenge remains still open. Any thoughts on how to analyze the memory occupied by all the threads will be helpful here.
Below are the snapshots taken from native memory tracking.
In this pic, you can see that 88 MB got increased in threads. Where arena and resource handle count had increased a lot.
in this picture you can see that 73 MB had increased in this Malloc. But no method name is shown here.
So please throw some info in understanding these 2 screenshot.
You may try another GC implementation like G1 introduced in Java 7 and probably the default GC in Java 9. To do so just launch your Java apps with:
-XX:+UseG1GC
There's also an interesting functionality with G1 GC in Java 8u20 that can look for duplicated Strings in the heap and "deduplicate" them (this only works if you activate G1, not with the default Java 8's GC).
-XX:+UseStringDeduplication
Be aware to test thoroughly your system before going to production with such a change!!!
Here you can find a nice description of the diferent GCs you can use
I encountered the exact same issue.
Heap usage constant, only metaspace increase, NMT diffs showed a slow but steady leak in the memory used by threads specifically in the arena allocation. I had tried to fix it by setting the MALLOC_ARENAS_MAX=1 env var but that was not fruitful. Profiling native memory allocation with jemalloc/jeprof showed no leakage that could be attributed to client code, pointing instead to a JDK issue as the only smoking gun there was the memory leak due to malloc calls which, in theory, should be from JVM code.
Like you, I found that upgrading the JDK fixed the problem. The reason I am posting an answer here is because I know the reason it fixes the issue - it's a JDK bug that was fixed in JDK8 u152: https://bugs.openjdk.java.net/browse/JDK-8164293
The bug report mentions Class/malloc increase, not Thread/arena, but a bit further down one of the comments clarifies that the bug reproduction clearly shows increase in Thread/arena.
consider optimising the JVM options
Parallel Collector(throughput collector)
-XX:+UseParallelGC
concurrent collectors (low-latency collectors)
-XX:+UseConcMarkSweepGC
use String Duplicates remover
-XX:+UseStringDeduplication
optimise compact ratio
-XXcompactRatio:
and refer
link1
link2
In this my answer you can see information and references how to profile native memory of JVM to find memory leaks. Shortly, see this.
UPDATE
Did you use -XX:NativeMemoryTracking=detail option? The results are straightforward, they show that the most memory allocated by malloc. :) It's a little bit obviously. Your next step is to profile your application. To analyze native methods and Java I use (and we use on production) flame graphs with perf_events. Look at this blog post for a good start.
Note, that your memory increased for threads, likely your threads grow in application. Before perf I recommend analyze thread dumps before/after to check does Java threads number grow and why. Thread dumps you can get with jstack/jvisualvm/jmc, etc.
This issue does not come with Java 8 update 152. The exact root cause of why it was coming with earlier versions is still not clearly identified.
Hello I developed a multi-threaded TCP server application that allows 10 concurrent connections receives continuous requests from them, after some processing requests, responds them to clients. I'm running it on a TI OMAP l137 processor based board it runs Monta Vista Linux. Threads are created per client ie 10 threads and it's pre-threaded. it's physical memory usage is about %1.5 and CPU usage is about %2 according to ps, top and meminfo. It's vm usage rises up to 80M where i have 48M (i reduced it from u-boot to reserve some mem for DSP). Any help is appreciated, how can i reduce it??.(/proc/sys/vm/.. tricks doesn't help :)
Thanks.
You can try using a drop in garbage collecting replacement for malloc(), and see if that solves your problem. If it does, find the leaks and fix them, then get rid of the garbage collector.
Its 'interesting' to chase these kinds of problems on platforms that most heap analyzers and profilers (e.g. valgrind) don't fully (if at all) support.
On another note, given the constraints .. I'm assuming you have decreased the default thread stack size? I think the default is 8M, you probably don't need that much. See pthread_attr_setstacksize() if you haven't adjusted it.
Edit:
You can check the default stack size with pthread_attr_getstacksize(). If it is at 8M, you've already blown your ceiling during thread creation (10 threads, as you mentioned).
Most VM is probably just for stacks. Of course, it's virtual, so it doesn't get commited if you don't use it.
(I'm wondering if thread's default stack size has anything to do with ulimit -s)
Apparently yes, according to
this other SO question
Does it rise to that level and stay there? Or does it eventually run out of memory? If the former, you simply need to figure out a way to have a smaller working set. If the latter, you have a memory leak and need to fix it.
At work we have an application to play 2K (2048*1556px) OpenEXR film sequences. It works well.. apart from when sequences that are over 3GB (quite common), then it has to unload old frames from memory, despite the fact all machines have 8-16GB of memory (which is addressable via the linux BIGMEM stuff).
The frames have to he cached into memory to play back in realtime. The OS is a several-year old 32-bit Fedora Distro (not possible to upgradable to 64bit, for the foreseeable future). The per-process limitation is 3GB per process.
Basically, is it possible to cache more than 3GB of data in memory, somehow? My initial idea was to spread the data between multiple processes, but I've no idea if this is possible..
One possibility may be to use mmap. You would map/unmap different parts of your data into the same virtual memory region. You could only have one set mapped at a time, but as long as there was enough physical memory, the data should stay resident.
How about creating a RAM drive and loading the file into that ... assuming the RAM drive supports the BIGMEM stuff for you.
You could use multiple processes: each process loads a view of the file as a shared memory segment, and the player process then maps the segments in turn as needed.
My, what an interesting problem :)
(EDIT: Oh, I just read Rob's ram drive post...I got all excited by the problem...but have a bit more to suggest, so I won't delete)
Would it be possible to...
setup a multi-gigabyte ram disk, and then
modify the program to do all it's reading from the "disk"?
I'd guess the ram disk part is where all the problem would be, since the size of the ram disk would be OS and file system dependent. You might have to create multiple ram disks and have your code jump between them. Or maybe you could setup a RAID-0 stripe set over multiple ram disks. Or, if there are still OS limitations and you can afford to drop a couple grand (4k?), setup a hardware RAID-0 strip set with some of those new blazing fast solid state drives. Or...
Fun, fun, fun.
Be sure to follow up!
I assume you can modify the application. If so, the easiest thing would be to start the application several times (once for each 3GB chunk of video), have each one hold a chunk of video, and use another program to synchronize them so they each take control of the framebuffer (or other video output) in turn.
The synchronization is going to be a little messy, perhaps, but it can be simplified if each app has its own framebuffer and the sync program points the video controller to the correct framebuffer inbetween frames when switching to the next app.
#dbr said:
There is a review machine with an absurd fiber-channel-RAID-array that can play 2K files direct from the array easily. The issue is with the artist-workstations, so it wouldn't be one $4000 RAID array, it'd be hundreds..
Well, if you can accept a limit of ~30GB, then maybe a single 36GB SSD drive would be enough? Those go for ~US$1k each I think, and the data rates might be enough. That very well maybe cheaper than a pure RAM approach. There are smaller sizes available, too. If ~60GB is enough you could probably get away with a JBOD array of 2 for double the cost, and skip the RAID controller. Be sure only to look at the higher end SSD options--the low end is filled with glorified memory sticks. :P