On Linux you can inspect /proc/$PID/pmaps to see the libraries loaded by a particular program, and a program can open /proc/self/pmaps to examine the libraries it itself has loaded.
I know pmaps will only contain dynamic libraries, and obviously the kernel can't predict which libraries we might dlopen at a later point, so I expect those aren't included in /proc/self/maps. But I'm unsure of a few other other scenarios:
Are libraries that have been linked at build time but we haven't called any function in yet included? My understanding is the Linux delays linking symbols until the first time they are used, so I'm not sure if they'll show up.
Does pmaps contain all the libraries used recursively? E.g. if I look at each library in pmaps and run ldd on it, and then run ldd on those, ad nauseum, I shouldn't find any new libraries that weren't in the original pmaps? I tried this on a couple binaries and it appears to be so but maybe I'm getting lucky.
Are libraries that have been linked at build time but we haven't called any function in yet included?
Yes: the runtime loader will mmap every library that your executable directly depends on, before your program starts running.
You can find the list of such libraries by running
readelf -d a.out | grep NEEDED
Does pmaps contain all the libraries used recursively?
Yes: if a library that you directly depend on itself depends on some other library, the runtime loader will mmap the recursive dependencies as well.
My understanding is the Linux delays linking symbols until the first time they are used
That is mosty correct for function symbols, but false for data symbols, which can't be resolved lazily.
Also, whether the symbols are resolved lazily or not depends on LD_BIND_NOW environment variable, and on an equivalent setting in the executable dynamic section, controlled by -znow linker flag.
None of that changes the mmap pciture though; if you have a DT_NEEDED entry for foo.so in your dynamic section, then foo.so will be mmaped (and will show in /proc/self/*map*) independent of lazy or non-lazy resolution.
/proc/$pid/maps is not only going to list libraries that are loaded, but also ALL other mapped memory segments.
Read this thread and the article in there:
Understanding Linux /proc/id/maps
Related
I was going through disassembly of elf executables and understanding the elf format. In there, I saw lib64/ld-linux-x86-64.so.2 used as program interpreter in the generated executable.
My guess is: I had used printf in the source code, which had to be dynamically linked. When I checked through dynamic section, I was able to find a reference to libc.so.6 shared library (tag:DT_NEEDED). In my system, I found multiple files with that name in different directories:
sourav#ubuntu-VirtualBox:/$ sudo find / -name libc.so.6
/usr/lib/x86_64-linux-gnu/libc.so.6
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied
/snap/snapd/13170/lib/x86_64-linux-gnu/libc.so.6
/snap/snapd/11107/lib/x86_64-linux-gnu/libc.so.6
/snap/core18/1988/lib/i386-linux-gnu/libc.so.6
/snap/core18/1988/lib/x86_64-linux-gnu/libc.so.6
/snap/core18/2128/lib/i386-linux-gnu/libc.so.6
/snap/core18/2128/lib/x86_64-linux-gnu/libc.so.6
So, I guess purpose of program interpreter is to resolve these names to the proper libraries and load them during execution. Is this correct?
It seems, we can also have executables with no program interpreter (which is the case for program interpreter itself). In that case, does system/os itself loads the shared library? If so, how does it resolves the path of library?
Is it possible to generate executable with no program interpreter using gcc? My gcc version is 'gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)'.
So, I guess purpose of program interpreter is to resolve these names to the proper libraries and load them during execution. Is this correct?
Yes, but that that's a bit minimalistic. Loading dynamic libraries involves locating them, loading or mapping them into memory if necessary, and resolving dynamic symbols within, possibly lazily, for multiple kinds of relocations. It involves recursively loading the libraries' own needed libraries. Also, in a dynamically linked executable, the program interpreter provides the program entry point (from the kernel's perspective), so it is also responsible for setting up and entering the program-specific entry point (for example, main() in a C or C++ program).
It seems, we can also have executables with no program interpreter (which is the case for program interpreter itself). In that case, does system/os itself loads the shared library? If so, how does it resolves the path of library?
You can have ELF executables without a program interpreter, but they are not dynamically linked, at least not in the ELF sense. There are no shared libraries to load, and certainly the system does not load any.
Is it possible to generate executable with no program interpreter using gcc? My gcc version is 'gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)'.
If you have static versions of all needed libraries available then you should be able to achieve that by including the -static option on the command line when you link the program. It is entirely possible, however, that you do not have the needed static libraries, even if libc is the only library you need.
Some other people in the office are discussing about reducing the memory footprint and load time by cutting out non-essential parts of an internal library into separate libraries, and load them on-demand.
I did some Googling tonight and learned that ld.so loads libraries with lazy binding, and both ld.so and dlopen loads the library with mmap. This seems to mean that those none-essential parts of the library are already loaded on-demand, i.e. as long as the functions are not used / data pages in the library are not read,
It doesn't take space and
It doesn't take time to load and allocate those pages.
So I thought they just need to make sure they don't actively touch all the non-essential components.
Is this true? Is this true (practically) for all posix systems?
Any library that directly links into your program or is a direct dependency still gets mapped and loaded at run time. To see what libraries will be open, you can use ldd some_program. Some of these will be indirect dependencies that will also be loaded. To see only what the direct depencies are you can use objdump -x my_program | grep NEEDED. Not only does every library have to be read from disk (can be relatively slow), but all the necessary symbols (sometimes many) from them must be mapped.
It may be that one dependency that isn't necessarily needed at startup, is pulling in the majority of your total library size. If you can split apart different capabilities into functional units that can be loaded as modules, it can be straight forward to reduce startup time and to some extent the memory footprint.
The easiest example I can come up with is an image viewer application where the GUI libraries are linked into the binary, but the actual loading of images is done by modules. (feh would be be an example of this using Imlib2 or optionally gdk-pixbuf for the image loading). This allows the main GUI to start up without ever loading libpng, libjpeg, libXPM, libcairo, librsvg, etc... This reduces the startup time because none of those need to be loaded, and may never be loaded, thus reducing the memory footprint.
In C this is commonly done with dlopen(), dlsym() and dlclose() (embedded systems using a flat binary format can use a specially linked module) but most toolkits will have their own module implementation (for instance Gtk+ uses glib's cross-platform module loading).
It doesn't have to be that cut and dry. You could just as easily split your initial GUI into parts and sequentially dlopen() them as an alternative to having a separate "splash screen". I don't know of any example of a toolkit that allows this but one could even open an X11 window with title and dimensions, then load the GUI and associated libraries into that window. You could load different binary format parsers depending on the file's extension/magic value or load different binary format generators depending on what format the user decides to Save_As.
Another alternative is to statically link library dependencies so that only the required symbols get loaded (as long as there are no licensing issues with doing so). Why load up 100+ widgets, when you only use 10? This works best with libraries that are built with static linking in mind such as musl-libc (glibc is notoriously useless for static linking). Compiling with the -flto compiler flag (or -combine -fwhole-program on older versions of gcc) along with -ffunction-sections and -fdata-sections and linking with --gc-sections typically helps (I normally see 15-50% size reduction) This is also a good option if you can't rely on the shared system libraries to be sane.
I apologize if my question is not precise because I don't have a lot
of Linux related experience. I'm currently building a Linux from
scratch (mostly following the guide at linuxfromscratch.org version
7.3). I ran into the following problem: when I build an executable it
gets a hardcoded path to something called ELF interpreter.
readelf -l program
shows something like
[Requesting program interpreter: /lib/ld-linux.so.2]
I traced this library ld-linux-so.2 to be part of glibc. I am not very
happy with this behaviour because it makes the binary very unportable
- if I change the location of /lib/ld-linux.so.2 the executable no
longer works and the only "fix" I found is to use the patchelf utility
from NixOS to change the hardcoded path to another hardcoded path. For
this reason I would like to link against a static version of the ld
library but such is not produced. And so this is my question, could
you please explain how could I build glibc so that it will produce a
static version of ld-linux.so.2 which I could later link to my
executables. I don't fully understand what this ld library does, but I
assume this is the part that loads other dynamic libraries (or at
least glibc.so). I would like to link my executables dynamically, but
I would like the dynamic linker itself to be statically built into
them, so they would not depend on hardcoded paths. Or alternatively I
would like to be able to set the path to the interpreter with
environment variable similar to LD_LIBRARY_PATH, maybe
LD_INTERPRETER_PATH. The goal is to be able to produce portable
binaries, that would run on any platform with the same ABI no matter
what the directory structure is.
Some background that may be relevant: I'm using Slackware 14 x86 to
build i686 compiler toolchain, so overall it is all x86 host and
target. I am using glibc 2.17 and gcc 4.7.x.
I would like to be able to set the path to the interpreter with environment variable similar to LD_LIBRARY_PATH, maybe LD_INTERPRETER_PATH.
This is simply not possible. Read carefully (and several times) the execve(2), elf(5) & ld.so(8) man pages and the Linux ABI & ELF specifications. And also the kernel code doing execve.
The ELF interpreter is responsible for dynamic linking. It has to be a file (technically a statically linked ELF shared library) at some fixed location in the file hierarchy (often /lib/ld.so.2 or /lib/ld-linux.so.2 or /lib64/ld-linux-x86-64.so.2)
The old a.out format from the 1990s had a builtin dynamic linker, partly implemented in old Linux 1.x kernel. It was much less flexible, and much less powerful.
The kernel enables, by such (in principle) arbitrary dynamic linker path, to have various dynamic linkers. But most systems have only one. This is a good way to parameterize the dynamic linker. If you want to try another one, install it in the file system and generate ELF executables mentioning that path.
With great pain and effort, you might make your own ld.so-like dynamic linker implementing your LD_INTERPRETER_PATH wish, but that linker still has to be an ELF shared library sitting at some fixed location in the file tree.
If you want a system not needing any files (at some predefined, and wired locations, like /lib/ld.so, /dev/null, /sbin/init ...), you'll need to build all its executable binaries statically. You may want (but current Linux distributions usually don't do that) to have a few statically linked executables (like /sbin/init, /bin/sash...) that will enable you to repair a system broken to the point of not having any dynamic linker.
BTW, the /sbin/init -or /bin/sh - path is wired inside the kernel itself. You may pass some argument to the kernel at boot load time -e.g. with GRUB- to overwrite the default. So even the kernel wants some files to be here!
As I commented, you might look into MUSL-Libc for an alternative Libc implementation (providing its own dynamic linker). Read also about VDSO and ASLR and initrd.
In practice, accept the fact that modern Linuxes and Unixes are expecting some non-empty file system ... Notice that dynamic linking and shared libraries are a huge progress (it was much more painful in the 1990s Linux kernels and distributions).
Alternatively, define your own binary format, then make a kernel module or a binfmt_misc entry to handle it.
BTW, most (or all) of Linux is free software, so you can improve it (but this will take months -or many years- of work to you). Please share your improvements by publishing them.
Read also Drepper's Hwo to Write Shared Libraries paper; and this question.
I ran into the same issue. In my case I want to bundle my application with a different GLIBC than comes system installed. Since ld-linux.so must match the GLIBC version I can't simply deploy my application with the according GLIBC. The problem is that I can't run my application on older installations that don't have the required GLIBC version.
The path to the loader interpreter can be modified with --dynamic-linker=/path/to/interp. However, this needs to be set at compile time and therefore would require my application to be installed in that location (or at least I would need to deploy the ld-linux.so that goes with my GLIBC in that location which goes against a simple xcopy deployment.
So what's needed is an $ORIGIN option equivalent to what the -rpath option can handle. That would allow for a fully dynamic deployment.
Given the lack of a dynamic interpreter path (at runtime) leaves two options:
a) Use patchelf to modify the path before the executable gets launched.
b) Invoke the ld-linux.so directly with the executable as an argument.
Both options are not as 'integrated' as a compiled $ORIGIN path in the executable itself.
I have been reading John R. Levine's Linkers and Loaders and I read that the properties of an object file will include one or more of the following.
file should be linkable
file should be loadable
file should be executable
Now, considering this example:
#include<stdio.h>
int main() {
printf("testing\n");
return 0;
}
Which I would compile and link with:
$ gcc -c t.c
$ gcc -o t t.o
I tried inspecting t.o using objdump and its type shows up as REL. What all properties does t.o satisfy? I believe that its linkable, non-executable. I would have believed that its not loadable(unless you create an .so file from the .o file); however the type REL means that its supposed to be relocated, and relocation would occur only in the context of loading, so I'm having a confusion here.
My doubts summarized :-
Are ".o" files loadable?
Reading resources regarding the sections present in a ".o", ".so" file - differences etc?
An object file (i.e., a file with the .o extension) is not loadable. This is because it lacks critical information about how to resolve all the symbols within it: in this case, the println symbol in particular would need additional information. (C compilers do not bind library identities into the object files they create, which is occasionally even useful.)
When you link the object file into a shared library (.so), you are adding that binding. Typically, you're also grouping a number of object files together and resolving references between them (plus a few more esoteric things). That then makes the result possible to load, since the loader can then just do resolution of references and loading of dependencies that it doesn't already know about.
Going from there to executable is typically just a matter of adding on the OS-defined program bootstrap. This is a small piece of code that the OS will start the program running by calling, and it typically works by loading the rest of the program and dependencies and then calling main() with information about the arguments. (It's also responsible for exiting cleanly if main returns.)
Just to set the context this link states somethings similar (emphasis for readability only);
A file may be linkable, used as input by a link editor or linking
loader. It my be executable, capable of being loaded into
memory and run as a program, loadable, capable of being loaded
into memory as a library along with a program, or any combination of
the three.
A .o file is a linker object file, which is according to this definition not executable and definitely linkable. Loadable is a tougher call, but since .o files are not loadable without some definitely not cross platform trickery, I'd say the spirit is that it's not loadable.
I have a stripped ld.so that I want to replace with the unstripped version (so that valgrind works). I have ensured that I have the same version of glib and the cross compiler.
I have compiled the shared object, calling 'file' on it shows that it is compiled correctly (the only difference with the original being the 'unstripped' and being about 15% bigger). Unfortunately, it then causes a kernel panic (unable to init) on start up. Stripping the newly compiled .so, readelf-ing it and diff-ing it with the original, shows that there were extra symbols in the new version of the .so . All of the old symbols were still present, so what I don't understand is why the kernel panics with those extra symbols there.
I would expect the extra symbols to have no affect on the kernel start up, as they should never be called, so why do I get a kernel panic?
NB: To be clear - I will still need to investigate why there are extra symbols, but my question is about why these unused symbols cause problems.
The kernel (assuming Linux) doesn't depend on or use ld.so in any way, shape or form. The reason it panics is most likely that it can't exec any of the user-level programs (such as /bin/init and /bin/sh), which do use ld.so.
As to why your init doesn't like your new ld.so, it's hard to tell. One common mistake is to try to replace ld.so with the contents of /usr/lib/debug/ld-X.Y.so. While that file looks like it's not much different from the original /lib/ld-X.Y.so, it is in fact very different, and can't be used to replace the original, only to debug the original (/usr/lib/debug/ld-X.Y.so usually contains only debug sections, but none of code and data sections of /lib/ld-X.Y.so, and so attempts to run it usually cause immediate SIGSEGV).
Perhaps you can set up a chroot, mimicking your embedded environment, and run /bin/ls in it? The error (or a core dump) this will produce will likely tell you what's wrong with your ld.so.