I've been coding for some time, and I always thought the linking phase of compilation was pretty straight forward, but recently I had to add a plugin system to an app, and I ran into quite a few compilation and runtime problems, due to my complete lack of knowledge on the matter. I've now got it working fine and learned a lot in the process, but now that I' aware I still have a lot to learn on the subject, I wanted to explore the subject more. So I was wondering if anyone had good pointers on articles, blog posts, or books on the whole "code sections, symbols, linking, dynamic/static libraries..." business.
I'm developing on Linux using gcc and ld, but I'm also interested in the M$ way of doing things, I thirst for knowledge :)
Thanks!
Here are some good online resources:
Static, Shared Dynamic and Loadable Linux Libraries
Program Library HOWTO
Ulrich Drepper (maintainer of glibc) provides a very thorough description of the linux implementation of shared libraries in his paper How To Write Shared Libraries, definitely worth a read if you want to know the nitty-gritty details. I actually can't think of a book that does a great job covering the details.
Related
I've recently been looking at the Rust programming language. How does it work? Rust code seems to be compiled into ELF or PE (etc) binaries, but I've not been able to find any information on how that's done? Is it compiled to an intermediate format then compiled the rest of the way with gxx for example? Any help (or links) would be really appreciated.
The code-generation phase of the Rust compiler is mainly done by LLVM. LLVM is a set of tools for building a compiler, most notably used by the C[++] Compiler clang[++].
First, the Rust compiler (just like clang, for example) does all the Rust specific stuff like type and borrow checking; in the end, it generates LLVM-IR. IR stands for intermediate representation and it's... comparable to assembly, but a tiny bit more high level and most importantly: platform independent. Then the Rust compiler just calls ☏ LLVM and says:
Hey buddy, could you please take this IR and generate machine code for the current platform? That would be fantastic ◕ ◡ ◕
To which LLVM responds:
🌈 Sure, no problem, new friend. Here is your highly optimized machine code for [e.g.] x86_64! ♫♪♫
Afterwards they invite a few more friends to wrap it all up in a nice little [e.g.] ELF package and beautifully place it in the users file system. (and the user is like...)
Information like this can be found in the official FAQ which contains a lot of interesting information anyway. For more in-depth details on the Rust compiler, you can read the "Rustc Guide". For this question, the chapter "High Level Overview" is pretty interesting.
I've got some unused functionality in my codebase, but it's hard to identify. The code has evolved over the last year as I explore its problem space and possible solutions. What I'm needing to do is find that unused code so I can get rid of it. I'm happy if it deals with the problem on an exportable name basis.GHC has warnings that deal with non-exported unused code. Any tools specific to this task would be of interest.
However, I'm curious about a comprehensive cross referencing tool. I can find the unused code with such a tool. Years ago when I was working in C and assembler, I found that a good xref was a pretty handy tool, useful for many different purposes.
I'm getting nowhere with googling. Apparently in Haskell the dominant meaning of cross-reference is within literate programming. Though maybe something there would be useful.
I don’t know of such a tool, so in the past I have done a bit of a hack instead.
If you have a comprehensive test suite, you can run it with GHC’s code coverage tracing enabled. Compile with -fhpc and use hpc markup to generate annotated source. This gives you the union of unused code and untested code, both of which you would probably like to address anyway.
SourceGraph can give you a bunch of information which you may also find useful.
There is now a tool for this very purpose: https://hackage.haskell.org/package/weeder
It's been around since 2017, and while it has limitations, it definitely helps with large codebases.
Is there a Linux distro (other than Minix) with good documentation for the source code? Or, is there some good documentation to describe the general Linux source code?
I have downloaded the Kernel source code, but, it is (unsurprisingly) a little overwhelming to find my way around and I wondered if there were some higher-level documentation to go with how the Linux kernel works?
Have you tried having a look on The linux documentation project I've find it quietly exhaustive regarding linux
They have a section The Linux Kernel wich is an online book that explains
how the linux kernel works and why it does behaves in certain ways, you should deffinitely
look into it because it's very well made.
Some of the Linux kernel code has decent commenting as documentation, but if you're going to be getting into kernel development, I'd recommend picking up a good book. A good, relatively easy-to-read one is Linux Kernel Development, by Robert Love. I got started on the Second Edition when I was in college, and keep a copy of the third on my bookshelf now.
I also find the Linux Cross Reference site helpful in jumping around the kernel source code. It's nice for tracking down functions that are in different files, and getting at what you need.
If you want to learn about operating systems and their basics, I strongly suggest you to start with a small kernel and then ramp up to learn about Linux. Starting with an operating system like Linux would be overwhelming in terms of code and documentation.
There is XV6 operating system which follows the basic Unix notion of files and processes. You can get the code listing and the documentation explaining the code properly. Here is a link to it. link.
Since academia is using this course as a baseline, I think you should get good support for understanding the same.
Linux Core Kernel Commentary is a little dated, but is still an excellent source of info.
For something which is not obsolete (like kernel.org/doc is), you may see:
Free Electrons Linux/Documentation/ (3.8)
Linux Cross Reference kernel/Documentation/
kernel-doc (3.6.10)
The first is the one I prefer personally (clean, readable, pleasant, up‑to‑date).
The second is the most well known.
The third, is for download, if you wish to browse and search it off‑line (may be handy in some case).
My two cents as a side note before I leave: I feel it's weird how for such a famous stuff as the Linux kernel is, when you search the web for documentation, you get masses of obsolete documentations, and how the rather up‑to‑date ones seems to be rather hidden and far from the top position of search engines.
I am looking for good books/resources for introducing how to use the thread library with C++0x. I have searched amazon.com and SO without informative post.
I asked a similar question myself recently: Where can I find good, solid documentation for the C++0x synchronization primitives?
And I got back a fantastic answer: C++ Concurrency in Action by Anthony Williams
The JustThread library at the end of that link also has good Doxygen documentation as well as implementations of a lot of the C++ threading stuff, though it's a commercial library :-/.
Lastly, you can get a pre-release PDF of this book. I've gotten it myself, and I can tell you that it's a pretty good book.
gcc/g++ implements more of this than they let on. While it's not yet complete, they have a decent implementation of the classes for threads and futures and they also implement the atomic family of classes which allow for some fairly fine-grained synchronization that you would normally only be able to achieve by somehow getting memory barrier instructions into your code by hand.
Right now there are very few (at least not gcc) that support the thread section of C++0x.
Therefore you have to use boost which closely follows the C++0x specification.
I find that the best resource for using boost libraries is their own online documentation, which can be found at http://www.boost.org/doc/libs/1_47_0/doc/html/thread.html.
I need to do a rather unusual thing: manually execute an elf executable. I.e. load all sections into right places, query main() and call it (and cleanup then). Executable will be statically linked, so there will be no need to link libraries. I also control base address, so no worries about possible conflicts.
So, is there are any libraries for that?
I found OSKit and its liboskit_exec, but project seems to be dead since 2002.
I'm OK with taking parts of projects (respecting licenses, of course) and tailoring them to my need, but as I'm quite a noob in the linux world, I dont even know where to find those parts! :)
PS. I need that for ARM platform.
UPD Well, the matter of loading elfs seems to require some good knowledge about it (sigh), so I'm out to read some specs and manuals. And I think I will stick to bionic/linker and libelfsh. Thanks guys!
Summarized findings:
libelf: http://directory.fsf.org/project/libelf/
elfsh and libelfsh (are now part of eresi): http://www.eresi-project.org/
elfio (another elf library): http://sourceforge.net/projects/elfio/
OSKit and liboskit_exec (outdated): http://www.cs.utah.edu/flux/oskit/
bionic/linker: https://android.googlesource.com/platform/bionic
A quick apt-cache search suggests libelf1, libelfg0 and/or libelfsh0. I think the elfsh program (in the namesake package) might be an interesting practical example of how to use libelfsh0.
I haven't tried any myself, but I hope they might be helpful. Good luck :-)
Google's Android, in it's "bionic" libc implementation, has a completely reimplemented ELF loader. It's reasonably clean, and probably a better source than gilbc if you're looking for something simple.
Take a look at libelf for reading the executable format. You are going to have trouble with this I think.
Sounds like, as you don't need libraries for anything, why not just mmap your executable, set data about various memory areas and jmp/b in?
I don't know if ARM has an NX-bit equivalent, but worth checking.
This tool contains an ELF loader: http://bitwagon.com/rtldi/rtldi.html
I reused the code from rtldi for an ELF chainloader in another project. The code is here: http://svn.gna.org/viewcvs/plash/trunk/chroot-jail/elf-chainloader/?rev=877 and there is some background here: http://plash.beasts.org/wiki/Story16. (Apparently I have to break these links because stackoverflow won't let me post >1 link!)