I have found some piece of code (function) in library which could be improved by the optimization of compiler (as the main idea - to find good stuff to go deep into compilers). And I want to automate measurement of time execution of this function by script. As it's low-level function in library and get arguments it's difficult to extract this one. Thus I want to find out the way of measurement exactly this function (precise CPU time) without library/application/environment modifications. Have you any ideas how to achieve that?
I could write wrapper but I'll need in near future much more applications for performance testing and I think to write wrapper for every one is very ugly.
P.S.: My code will run on ARM (armv7el) architecture, which has some kind of "Performance Monitor Control" registers. I have learned about "perf" in linux kernel. But don't know is it what I need?
It is not clear if you have access to the source code of the function you want to profile or improve, i.e. if you are able to recompile the considered library.
If you are using a recent GCC (that is 4.6 at least) on a recent Linux system, you could use profilers like gprof (assuming you are able to recompile the library) or better oprofile (which you could use without recompiling), and you could customize GCC for your needs.
Be aware that like any measurements, profiling may alter the observed phenomenon.
If you are considering customizing the GCC compiler for optimization purposes, consider making a GCC plugin, or better yet, a MELT extension, for that purpose (MELT is a high-level domain specific language to extend GCC). You could also customize GCC (with MELT) for your own specific profiling purposes.
Related
I am trying to figure out a bug (a serious performance downgrade). Unfortunately, I wasn't able to figure out why by going back many different versions of my code.
I am suspecting it could be some modifications to libraries that I've updated, not to mention in the meanwhile I've updated to GHC 7.6 from 7.4 (and if anybody knows if some laziness behavior has changed I would greatly appreciate it!).
I have an older executable of this code that does not have this bug and thus I wonder if there are any tools to tell me the library versions I was linking to from before? Like if it can figure out the symbols, etc.
GHC creates executables, which are notoriously hard to understand... On my Linux box I can view the assembly code by typing in
objdump -d <executable filename>
but I get back over 100K lines of code from just a simple "Hello, World!" program written in Haskell.
If you happen to have the GHC .hi files, you can get some information about the executable by typing in
ghc --show-iface <hi filename>
This won't give you the assembly code, but you can get some extra information that may prove useful.
As I mentioned in the comment above, on Linux you can use "ldd" to see what C-system libraries you used in the compile, but that is also probably less than useful.
You can try to use a disassembler, but those are generally written to disassemble to C, not anything higher level and certainly not Haskell. That being said, GHC compiles to C as an intermediary (at least it used to; has that changed?), so you might be able to learn something.
Personally I often find view system calls in action much more interesting than viewing pure assembly. On my Linux box, I can view all system calls by running using strace (use Wireshark for the network traffic equivalent):
strace <program executable>
This also will generate a lot of data, so it might only be useful if you know of some specific place where direct real world communication (i.e., changes to a file on the hard disk drive) goes wrong.
In all honesty, you are probably better off just debugging the problem from source, although, depending on the actual problem, some of these techniques may help you pinpoint something.
Most of these tools have Mac and Windows equivalents.
Since much has changed in the last 9 years, and apparently this is still the first result a search engine gives on this question (like for me, again), an updated answer is in order:
First of all, yes, while Haskell does not specify a bytecode format, bytecode is also just a kind of machine code, for a virtual machine. So for the rest of the answer I will treat them as the same thing. The “Core“ as well as the LLVM intermediate language, or even WASM could be considered equivalent too.
Secondly, if your old binary is statically linked, then of course, no matter the format your program is in, no symbols will be available to check out. Because that is what linking does. Even with bytecode, and even with just classic static #include in simple languages. So your old binary will be no good, no matter what. And given the optimisations compilers do, a classic decompiler will very likely never be able to figure out what optimised bits used to be partially what libraries. Especially with stream fusion and such “magic”.
Third, you can do the things you asked with a modern Haskell program. But you need to have your binaries compiled with -dynamic and -rdynamic, So not only the C-calling-convention libraries (e.g. .so), and the Haskell libraries, but also the runtime itself is dynamically loaded. That way you end up with a very small binary, consisting of only your actual code, dynamic linking instructions, and the exact data about what libraries and runtime were used to build it. And since the runtime is compiler-dependent, you will know the compiler too. So it would give you everything you need, but only if you compiled it right. (I recommend using such dynamic linking by default in any case as it saves memory.)
The last factor that one might forget, is that even the exact same compiler version might behave vastly differently, depending on what IT was compiled with. (E.g. if somebody put a backdoor in the very first version of GHC, and all GHCs after that were compiled with that first GHC, and nobody ever checked, then that backdoor could still be in the code today, with no traces in any source or libraries whatsoever. … Or for a less extreme case, that version of GHC your old binary was built with might have been compiled with different architecture options, leading to it putting more optimised instructions into the binaries it compiles for unless told to cross-compile.)
Finally, of course, you can profile even compiled binaries, by profiling their system calls. This will give you clues about which part of the code acted differently and how. (E.g. if you notice that your new binary floods the system with some slow system calls where the old one just used a single fast one. A classic OpenGL example would be using fast display lists versus slow direct calls to draw triangles. Or using a different sorting algorithm, or having switched to a different kind of data structure that fits your work load badly and thrashes a lot of memory.)
I am developing an online code judging software for c/c++/java codes.
I want to include various parameters for judging a code ,namely compilation time,execution time,memory usage ,just like the IDEONE API provides with.
How can i extract these parameters while compiling/executing a code in a LINUX environment?Are there any specific commands?
Also are there any other parameters which can be used to judge a code?
The judge verb is a bit strange in your question (which is perhaps too imprecise). Maybe you mean evaluate ?
Assuming the evaluated source code is compiled by a recent GCC compiler (i.e. version 4.7 or 4.8 of GCC) and that you can parameterize (or just repeat) its compilation, you could consider extending & customizing the GCC compiler for evaluation or metric purposes. This is possible either directly thru GCC plugins (painfully coded in C or C++), or thru MELT extensions (MELT is a domain specific language to extend and customize GCC).
You'll need some work to go this route, because you need to dive into GCC internals. The MELT probe might help you in understanding more the Gimple representation (inside GCC). You could also try compiling some sample code with gcc -fdump-tree-all which produce many dump files.
So the idea is that you would take time (days, perhaps weeks) to develop a MELT extension (e.g. in some file shiven.melt) for analysis, metrics and evaluation purposes, and that you would [re-] compile the example.c source code, e.g. with
gcc -fplugin=melt \
-fplugin-arg-melt-extra=shiven \
-fplugin-arg-melt-mode=shivenanalysis \
-c example.c
(of course you'll add other compiler flags, e.g. -O -I/some/include/dir/ ...)
Then, you could make a MELT extension to measure some characteristics of the compiled code, like number of functions, number of basic blocks, number of Gimple instructions, etc. This will happen at compilation time. Your MELT extension (in your file shiven.melt) could e.g. write some statistics in some database.
Extending GCC is meaningful for C, C++, Fortran, Ada .... source code, but much less for Java (because nobody uses GCC to compile Java, even if gcj exists, and because gcj probably supports a subset of some old Java standard).
Please subscribe to gcc-melt#googlegroups list and ask there for MELT related questions. Mention explicitly your MELT interest (perhaps your question) in your subscription.
There is the time command which gives you the execution time of a binary. With that you can get the compilation time, time gcc code.c, or execution time, time ./a.out. For memory usage you can use valgrind, or ps. With ps, if you are using stdin for input it should be simple. Just start the application, run ps at certain intervals in the backgound, and supply the input to the application.
I am trying to use a Wifi-Dongle with a Raspberry Pi. The vendor of the dongle provides a Linux driver that I can compile successfully on the ARM-architecture, however, one object file, that comes with the driver, was precompiled for a x86-architecture, which causes the linker to fail.
I know it would be much easier to compile that (quite big) file again, but I don't have access to the source code.
Is it possible to convert that object file from a x86-architecture to an ARM-architecture?
Thank you!
Um, no, it looks to me like a waste of time. Wi-Fi driver is complex, and you say this one troublesome object file is 'large'. Lots of pain to translate, and chance of successful debug slim to none. Also, any parameter passing between this one object file and the rest of the system would not translate directly between x86 and ARM.
In theory, yes. Doing it on a real kernel driver without access to source code will be difficult.
If you had high quality dis-assembly of the object file, and the code in the object file is "well behaved" (using standard calling conventions, no self modifying code) then you could automatically translate the X86 instructions into arm instructions. However, you probably don't have high quality dis-assembly. In particular, there can be portions of the object file that you will not be able to properly classify as code or data doing normal recursive descent dis-assembly. If you misinterpret data as code, it will be translated to ARM code, rather than copied as is, and so will have the wrong values. That will likely cause the code to not work correctly.
Even if you get lucky, and can properly classify all of the addresses in the object file, there are several issues that will trip you up:
The calling conventions on X86 are different than the calling conventions on ARM. This means you will have to identify patterns related to X86 calling conventions and change them to use ARM calling conventions. This is a non trivial rewrite.
The hardware interface on ARM is different than on X86. You will have to understand how the driver works in order to translate the code. That would require either a substantial X86 hardware comparability layer, or reverse engineering of how the driver works. If you can reverse engineer the driver, then you don't need to translate it. You could just write an arm version.
The internal kernel APIS are different between ARM and X86. You will have to understand those difference and how to translate between them. That's likely non trivial.
The Linux Kernel uses an "alternatives" mechanism, which will rewrite machine code dynamically when code is first loaded into the kernel. For example, on uni-processor machines, locks are often replaced with no-ops to improve perf. Instructions like "popcnt" are replaced with function calls on machines that don't support it, etc. It's use in the Kernel is extremely common. This means there's a good chance the code in the object is file is not "well behaved", according to the definition given above. You would have to either verify that the object file doesn't use that mechanism, or find a way to translate uses of it.
X86 uses a different memory model than ARM does. To "safely" translate X86 code to ARM (without introducing race conditions) you would have to introduce memory fences after every memory access. That would result in REALLY BAD performance on an ARM chip. Figuring out when you need to introduce memory fences (without doing it everywhere) is an EXTREMELY hard problem. The most successful attempts at that sort of analysis require custom type systems, which you won't have in the object file.
Your best bet (quickest route to success) would be to try and reverse engineer what the object file in question does, and then just replace it.
There is no reasonable way of doing this. Contact the manufacturer and ask if they can provide the relevant code in ARM code, as x86 is useless to you. If they are not able to do that, you'll have to find a different supplier of either the hardware [that has an ARM version, or fully open source, of all the components], or supplier of the software [assuming there is another source of that].
You could translate the x86 assembly manually by installing x86 GNU binutils and disassemble
the object file with objdump. Probably some addresses will differ but should be straight forward.
Yes, you could most definitely do a static binary translation. x86 disassembly is painful though, if this was compiled from high level then it isnt as bad as it could be.
Is it really worth the effort? Might try an instruction set simulator instead. Have you done an analysis of the number of instructions used? System calls required, etc?
How far have you gotten so far on the disassembly?
Maybe the file only contains a binary dump of the wifi firmware? If so you need no instruction translation and a conversion can be done using objcopy.
You can you use objdump -x file.o and look if any real executable code is inside the obj-file or if it's only data.
If you have access to IDA with Hex-Rays decompiler, you can (with some work) decompile the object file into C code and then try to recompile it for ARM.
I'd like to perform source code analysis of Linux kernel, but to do that, I first need to parse it. What are my options? I'd prefer an AST usable from python, but any other language is ok too.
Apparently CIL is able to parse whole kernel, but it's not clear from the website, how to do that.
I'd recommend starting with the sparse static analysis tool. Because sparse was designed specifically to assist the kernel developers in performing static analysis on the kernel, you can have some level of assurance that it really ought to parse the combination of C99 and GNU extensions that are used in the kernel sources. The code I've examined looked clean and straight forward but I never tried to extend it in any fashion. The Documentation/sparse.txt file has a very short synopsis of using sparse on the kernel sources, if you want a very high-level overview.
Another option is GCC MELT, a tool designed to make it easier to build plugins for the gcc compiler. Using it would require knowing enough gcc internals to find your way around, but MELT does look far easier than coding a similar plugin directly in C.
You can check the page Parsing Kernel
about tools comparision. The winner seems to be KDevelop.
Regards,
Do you really need an AST? Or a lower level intermediate representation would be just enough? For both options, you can use Clang, and either analyse its AST (sadly, with C++ only) or an LLVM IR.
CIL is also an option, but you'd need to write your analysis tool in OCaml. cilly is its drop-in replacement for gcc, but it might need some hacking for using it with such a non-trivial build sequence as the Linux kernel. Just using --merge won't be sufficient.
What are the input limitations of a bare metal cross compiler...as in does it not compile programs with pointers or mallocs......or anything that would require more than the underlying hardware....also how can 1 find these limitations..
I also wanted to ask...I built a cross compiler for target mips..i need to create a mips executable using this cross compiler...but i am not able to find where the executable is...as in there is 1 executable which i found mipsel-linux-cpp which is supposed to compile,assemble and link and then produce a.out but it is not doing so...
However the ./cc1 gives a mips assembly.......
There is an install folder which has a gcc executable which uses i386 assembly and then gives an exe...i dont understand how can the gcc exe give i386 and not mips assembly when i have specified target as mips....
please help im really not able to understand what is happ...
I followed the foll steps..
1. Installed binutils 2.19
2. configured gcc for mips..(g++,core)
I would suggest that you should have started two separate questions.
The GNU toolchain does not have any OS dependencies, but the GNU library does. Most bare-metal cross builds of GCC use the Newlib C library which provides a set of syscall stubs that you must map to your target yourself. These stubs include low-level calls necessary to implement stream I/O and heap management. They can be very simple or very complex depending on your needs. If the only I/O support is to a UART to stdin/stdout/stderr, then it is simple. You don't have to implement everything, but if you do not implement teh I/O stubs, you won't be able to use printf() for example. You must implement the sbrk()/sbrk_r() syscall is you want malloc() to work.
The GNU C++ library will work correctly with Newlib as its underlying library. If you use C++, the C runtime start-up (usually crt0.s) must include the static initialiser loop to invoke the constructors of any static objects that your code may include. The run-time start-up must also of course initialise the processor, clocks, SDRAM controller, timers, MMU etc; that is your responsibility, not the compiler's.
I have no experience of MIPS targets, but the principles are the same for all processors, there is a very useful article called "Building Bare Metal ARM with GNU" which you may find helpful, much of it will be relevant - especially porting the parts regarding implementing Newlib stubs.
Regarding your other question, if your compiler is called mipsel-linux-cpp, then it is not a 'bare-metal' build but rather a Linux build. Also this executable does not really "compile, assemble and link", it is rather a driver that separately calls the pre-processor, compiler, assembler and linker. It has to be configured correctly to invoke the cross-tools rather than the host tools. I generally invoke the linker separately in order to enforce decisions about which standard library to link (-nostdlib), and also because it makes more sense when a application is comprised of multiple execution units. I cannot offer much help other than that here since I have always used GNU-ARM tools built by people with obviously more patience than me, and moreover hosted on Windows, where there is less possibility of the host tool-chain being invoked instead (one reason why I have also avoided those tool-chains that rely on Cygwin)
EDIT
With more time available, I have rewritten my original answer in an attempt to provide something more useful.
I cannot provide a specific answer for your question. I have never tried to get code running on a MIPS machine. What I do have is plenty of experience getting a variety of "bare metal" boards up and running. All kinds of CPUs and all kinds of compilers and cross compilers. So I have an understanding of the principles that apply in all such situations. I will point out the kind of knowledge you will need to absorb before you can hope to succeed with a job like this, and hopefully I can list some links to resources to get you started on learning that knowledge.
I am worried you don't know that pointers are exactly the kind of thing a bare metal compiler can handle, they are a basic machine primitive. This tells me you are probably not an expert embedded developer who is just stuck in this particular scenario. Never mind. There isn't anything magic about programming an embedded system, and you can learn what you need to know.
The first step is getting to understand the relationship between C and the machine you wish to run code on. Basically C is a portable assembly language. This means that C is good for manipulating the basic operations of the machine. In this sense the basic operations of the machine are reading and writing memory locations, performing arithmetic and boolean operations on the data read from memory, and making branching and looping decisions based on that data. In particular the C concept of pointers allows you to manipulate data at locations in memory that you specify.
So far so good, but just doing raw computations in memory is not usually enough - you need a way to input and output data from memory. To do that you need to manipulate the hardware peripherals on your board. If the hardware peripherals are memory mapped then the machine registers used to control the peripherals look exactly like memory locations and C can manipulate them directly. Even in that case though, it is much more likely that doing useful I/O is best handled by extending the C core language with a library of routines provided just for that purpose. These library routines handle all the nasty details (timers, interrupts, non-memory mapped I/O) involved in manipulating the peripheral hardware on the board, and wrap them up with a convenient C function call interface. The idea is that you can go simply printf("hello world"); and the library call take care of the details of displaying the string.
An appropriately skilled developer knows how to adapt an existing I/O library to a new board, or how to develop new library routines to provide access to non-standard custom hardware. The classic way to develop these skills is to start with something simple, usually a LED for an output device, and a switch for an input device. Write a program that pulses a LED in a predictable way, or reads a switch and reflects in on a LED. The first time you get this working will be hugely satisfying.
Okay I have rambled enough. It is time to provide some more resources for you to study. The good news is that there's never been a better time to learn how things work at the interface between hardware and software. There is a wealth of freely available code and docs. Stackoverflow is a great resource as you know. Good luck! Links follow;
Embedded systems overview
Knowing the C language well is fundamental
Why not get your code working on a simulator before you try real hardware
Another emulated environment
Linux device drivers - an overlapping subject
Another book about bare metal programming