I have been working to eliminate memory leaks in our mono touch and learned a lot in last couple of days e.g. that it is almost always some event that needs to be unhook before garbage collecting is succeeded :)
But now I have been playing around with the profiller tool and I can see that most of the memory is used by strings (or it seems), please see the following screendump:
But as you can see some memory is also used by mono. I have been working our viewmodels and views and most of these are garbage collected correct. If I look into the strings they sometimes are referenced by and I have no clue what to do with this info.
Do you guys have any suggestion if I can reduce the amount of memory used by strings :) I have tried to find any tutorial or similar that might enlight what these numbers means, but with no luck. Any help is appreciated.
Some answers from my personal experience:
For tutorial I only really know about http://docs.xamarin.com/ios/Guides/Deployment%252c_Testing%252c_and_Metrics/Monotouch_Profiler
I find the 'Inverse References' option one of the most useful features - what matter is not that you have a lot of strings, but rather what owns those strings.
I find the best way of hunting these bugs is to reproduce them in a simple test harness and/or test sequence - as apps get bigger and I use more and more components - MvvmCross, JSON.Net, SQLite-net, etc - in more and more asynchronous ways, then I find I need to cut down on the number of these components to identify the leaks.
Once you have a simple test harness, the filter option in the HeapShot helps - as it lets you focus on classes which are in known namespaces.
Once you have a simple test harness, then comparing two HeapShot's can also help - which actions in your test UI cause what increases between HeapShots?
Differences are what matter - some libraries deliberately cache things in memory - e.g. some of the PropertyInfo's in your HeapShot images might be deliberately cached by one of the libraries in order to improve deserialisation speed.
For easier cross-referencing, adding links to linked questions:
Garbage collecting issue with Custom viewbinding in mono touch and mvvmcross
when to release objects in mono touch / mvvmcross
MVVMCross - SqlBits Memory Leak
Helping the GC in mono droid using mvvmCross
In addition to Stuart's great answer, I'd like to stress that you should profile on device. Execution on device is tweaked for runtime performance, while the simulator is tweaked for build performance (so that the edit-debug cycle is as fast as possible). Among other things it means that in the simulator more memory is used at runtime than in on device.
Related
I got very interested in browser exploitation, particularly in memory corruptions like UAF or type confusion vulnerabilities. Started learning some things, but can't understand some concepts.
First, I know tat fuzzing is one of the methods to find the bugs. Not sure how fuzzing can find those complex vulnerabilties.
Second, want to find out whether it is possible to find uaf bugs manually.
Third, can you please explain how uaf bugs occur in browsers in detailed manner? I know that uaf bug happens when freed memory is reused by code, and when attacker controlled data is placed in the freed memory, you get the code execution. But I can't understand how people generate all those HTML or POC codes to crash the software using the UAF bugs.
Fourth, what are type confusion vulnerabilties?
For web browsers :
1> fuzzing is the efficient way to find bugs, but using an existing fuzzer, in most of cases lead to finding existing vulnerabilities (already reported to editors).
In all times, after finding a bug, a manual work is needed to clean up the poc code
2> Spend time to make better your fuzzing strategy and generate a use cases (focus in allocation memory, feeing memory, copying references...) is the best manually work you can do to find a UAF Vuln.
3> you can find severals tutorial about the UAF on internet.
Good luck
I understand that this question verges into implementation specific domains, but at this point, Rakudo/MoarVM specific answers would help me too.
I am working on some NativeCall modules, and wondering how to debug memory leaks. Some memory is handled in the C library, and I have a good handle over there. I know that domain is my responsibility and there is nothing that MoarVM can do over there. What can I do in the MoarVM domain? what is the best way to check for dangling objects, circular references, etc.?
Is there a way at the end of a series of operations, where I think all of my Perl objects are out of scope to say "Run Garbage Collection and tell me about anything left"?
Is there some Rakudo/NQP/MoarVM specific code I can run to help me? This isn't to release in production, just for testing/diagnostics while I am developing.
Garbage Collection in MoarVM gives a tantalizing overview, but not enough information for me to do anything with it.
Firstly, while leaked memory on the C-side isn't your problem in this case, it's worth knowing that Rakudo installs a perl6-valgrind-m that runs the program under valgrind. I've used this a number of times to figure out segfaults and leaks when writing native library bindings.
For looking into objects managed by MoarVM, it's possible to get the VM to dump heap snapshots. They are taken after each GC run, and an extra GC run is forced and a final snapshot taken at the end of the program. To record snapshots, run with --profile=heap. The output file can then be fed to moar-ha, which can be installed using zef install App::MoarVM::HeapAnalyzer (it's implemented in Perl 6, which may be worth knowing should you wish to extend it in some way to help you solve you problems).
If you have any idea of what kind of objects might be leaking, then it can be useful to search for objects of that type with the find command. There is then a path command that shows how that object is being kept alive. It can also be useful to look at counts of objects between different heap snapshots, to see what is growing in use. Unfortunately there's not yet a snapshot diff feature.
One thing to note is that the snapshots include everything that runs atop of the VM. That means the Perl 6 compiler will be in memory, as well as a bunch of objects for things from the language built-ins. (The tool was developed to help track down managed leaks in the compiler and built-ins, so this is considered a feature. :-) Some kind of filtering may be feasible in the future, however.)
Finally, you mentioned circular references. These are not a problem in Perl 6, since GC is done through tracing, not reference counting.
When developing XPages applications it seems to have become very popular to mainly use Java methods and beans instead of server-side JavaScript (SSJS). SSJS of course takes longer to execute because the code has to be evaluated at runtime. However, can anyone provide information about the QUANTITATIVE gain in performance when using Java? Are there any benchmarks for how much the execution times differ, for example depending on the length of the SSJS code or the functions used?
You have to use your own benchmarks. The increase in time might not be measurable. It is more around capabilities and your development process. Switching from SSJS to Java an expecting an instant increase in performance most likely won't happen.
Unless of course Java allows you to code things differently. So most of the decisions are based on capabilities, not speed. You are most welcome to run some tests and share the insights. What you can expect e.g. opening a document in SSJS vs. Java: the difference should be in the space of a rounding error, since most of the time is needed for the C call below.
SSJS and Java run at almost the same speed after the SSJS has been evaluated, so you have some onramp time and similar speed thereafter.
I agree about the performance gain being negligible. I will chime in to say this. Right now I am trying to learn to support an existing XPages application written without using any java, and entirely in SSJS. There is code here, there, and everywhere. It is very hard to follow.
Depending on your environment, you should consider programmer productivity when considering how to build your applications, especially when you know both. Productivity for you, and those coming after you.
Stephan's answer is right on point: though Java as a language IS faster (you'd probably see performance gains proportional to the complexity of the block of code more than the number of operations running), the primary benefit is program structure. My experience has been that using Java extensively makes my code much cleaner, easier to debug, and MUCH easier to understand after coming back to it months later.
One of the nice side effects of this structural change does happen to be performance, but not because of anything inherent to Java: by focusing on classes and getters/setters, it makes it easier to really pay attention to expensive operations and caching. While you CAN cache your data excellently in SSJS using the various scopes, it's easier for your brain - both now and after you've forgotten what you did next year - to think about that sort of thing in Java.
Personally, even if Java executed more slowly than SSJS but the programming models in XPages were the same as they are now, I would still use Java primarily.
You are asking about the pure processing performance - the speed of the computer running the code. And as Stephen stated Java is going to be a "little" faster because it doesn't need to do the extra step of the string parsing the code first. Ok in the big picture that's really not a big deal.
I think the real "performance" gain that you get by moving to Java in XPages is cleaner code with more capabilities. Yes you're putting a lot of code in SSJS Libraries. And that can work really well. But I assume those are more individual functions that you use over and over rather then true objects that you can put in memory and they're they're when you need them. When you get your core business logic inside Java Objects in my experience the speed of development goes significantly faster. It's not even close.
Take the Domino document object. That's a rather handy object. Imagine if it wasn't an "object" but simply a library of 50 or so functions that you need to first paste into each database. Doesn't seem right. And of course in the Domino API it's not just the domino object. There's like 60 or so different objects!
Typical XPages with Java development moves much - not all - but much of the code away from the .xsp page and into Java Classes which are very similar to custom classes on LotusScript. The not only creates separation between frontend code - making the .xsp pages easier to work with - but puts the business logic inside Java which is similar to working to the the Domino backend objects. So then the backend gets easier to work with, maintain and add onto.
And that's where a big part of the development speed improvements come from.
Getting back to your original question, which is about computer speed. I would suggest that it's much easier to cache frequently used data via Java Objects and managed beans then it is with SSJS. Not having to hit the disc as much would be a real speed advantage.
I would recommend you to consider performance gain in a wider context.
performance gain in quicker running?
performance gain in typing?
performance gain in not making mistakes because of the editor?
performance gain of using templating in the Java editor?
performance gain in better reusability, eventually to server-wide plugins?
performance gain in being comfortable building your own classes to hold complex objects?
performance gain in easier debugging?
performance gain in being comfortable with Validators, Converters, Phase Listeners, VariableResolvers etc?
performance gain in being comfortable looking at Extension Libraries to investigate or extend?
performance gain of being able to find answers more easily on StackOverflow or Google because you're using a standard language vs a proprietary language?
performance gain in using third party Java code like Apache Commons, Apache POI etc?
To be honest, when you have got that far and understand how much code is run during a page load or partial request, performance gain in runtime of Java vs SSJS is minimal compared to something like using loaded where possible instead of rendered. The gains of Java over SSJS are much wider, and I have not even mentioned the gains in professional development.
My answer is way too long for a stackOverflow answer, so as promised, here is a link to my blog post about this issue. Basically it has nothing to do with performance, but with Maintainability, Readability, Usability
PetraVM recently came out with a Beta release of their Jinx product. Has anyone checked it out yet? Any feedback?
By good, I mean:
1) easy to use
2) intuitive
3) useful
4) doesn't take a lot of code to integrate
... those kinds of things.
Thanks guys!
After literally stumbling across Jinx while poking around on Google, I have been on the beta and pre-beta tests with a fair amount of usage already under my belt. As with any beta related comments please understand that things may change or already have changed, so do keep this in mind and take the following with a grain of salt.
So, going through the list of questions one by one:
1) Install and go. Jinx adds a control panel to Visual Studio which you can mostly ignore as the defaults are typically good for most cases. Otherwise you just work normally and forget about it. Jinx does not instrument your code, require any additional libraries linked in or the numerous other things some tools require.
2) The question of "intuitive" is really up to the user. If you understand threaded code and the sorts of bugs possible, Jinx just makes those bugs happen much more frequently, which by itself is a huge benefit to people doing threaded code. While Jinx attempts to stop the code in a state that makes the problem as obvious as possible, "obvious" and "intuitive" are really up to the skill of the programmer.
3) Useful? Anyone who has done threaded code before knows that a race condition can happen regularly or once every month based on cosmic ray counts, that randomness makes debugging threaded code very difficult. With Jinx, even the most minor race condition can be reproduced usually on the first run consistently. This works even for lockless code that other static analysis or instrumenting tools would generally miss.
This sort of quick reproduction of problems is amazingly useful. Jinx has helped me track down a "one instruction in the wrong place" sort of bug that would actually hit once a week at most. Jinx forced the crash to happen almost immediately and allowed me to focus on the actual cause of the bug instead being completely in the dark as to the real source.
4) Integration with Jinx is a breeze. If you don't mind your machine becoming a bit slow, you can tell Jinx to watch the entire machine. It slows the machine down as it is actually watching everything on the machine, including the OS. While interresting and useful if your software consists of multiple processes on the same machine, this is not suggested as it can become painful to work with the machine.
Instead of using the global system, adding an include and two lines of code does the primary work needed of registering the process with Jinx such that Jinx can watch just the registered items. You can help Jinx by using the Jinx specific asserts and registering regions of code that are more important. In the case of the crash mentioned above though, I didn't have to do any of that and Jinx found the problem without the additional integration work. In any case, the integration is extremely simple.
After using Jinx for the last couple months, I have to say that overall it has been a great pleasure. I won't write new threaded code without Jinx running in the background anymore simply because it does its intended job of forcing obscure threading issues to be immediate assert/crashes. As mentioned, things that you could go weeks without seeing become problems almost immediately, this is a wonderful thing to have during initial test and implementation.
KRB
BTW, PetraVM has changed its name to Corensic and you can find Jinx Beta 2 over at www.corensic.com.
--Prashant, the marketing guy at Corensic
So my company uses a delightfully buggy program called Rational Purify (as a plugin to Microsoft Visual Developer Studio) to manage memory leaks. The program is deigned to let you click on a memory leak after you have encountered it, and then jump to the line that the leak occurs on.
Unfortunately Purify is malfunctioning and Purify will not jump to the place that the leak occurred it only mentions the class and method that the leak occurs in. Unfortunately, sometimes this is about as useful as hiring a guide to help you hunt bears and having him point to the forest and tell you there are bears there.
Does anyone with Purify experience have any idea how I might fix this problem or have a good manual to look though?
Generally you have two options, one exclude modules DLL's from instrumentation in Purify, it helps some times. Second is get BoundsChecker, this does compile time instrumentation much slower but the level of detail is an order of magnitude better.
We generally use Purify on check-in, sanity checking, and BoundsChecker when we know a bug/crash exists.
BoundsChecker has some nice features like only instrument files A.cpp & B.cpp, excluding all the rest.
Be aware neither of these two applications function on 64 bit operating systems, and BoundsChecker will not install on 64 bit OS. Most frustrating if you make the switch to native 64 bit development with 32 bit back port!
Purify is like a swiss knife. If you know how to use it, you will get some results, not the best but still results. If you don't, it will crash, because it is just another program running on Windows.
In the end you will need a lot of patience, rebuilds and a bit of luck.
Purify comes with a script called ScanVSSolutionForPurifyPlus.pl which will ensure that your project files have all the right settings for Purify to work properly. If you haven't run it, give it a go.
(I've personally used ScanVSSolutionForPurifyPlus.pl on a large solution, and it worked like a charm. One caveat: when you give it the name of your .sln file, you might need to give it the full pathname.)
Are you sure you have debug build? Or rather you have all PDB's enabled? Try WindDbg on your executable and check with !lmi command what is visible.
Is whole code properly instrumented?
Also consider using something else like free Visual Leak Detector or Microsoft's tool LeakDiag.
I used Purify about 5 years ago. It was really flaky then. They kept promising to fix all the bugs in the 'next release'. We gave up on it in the end. One can only wonder if they used their own QA tools on their products. Oh the irony...