Is there an equivalent of dyld for Linux? - linux

Mac OS X provides a useful library for dynamic loading, called dyld. Among the many interesting functions for dynamic loading handling are functions to allow one to install callbacks which will be called by dyld whenever an image is loaded or unloaded, by dlopen and dlclose, respectively. Those functions are void _dyld_register_func_for_add_image(void (*func)(const struct mach_header* mh, intptr_t vmaddr_slide)) and void _dyld_register_func_for_remove_image(void (*func)(const struct mach_header* mh, intptr_t vmaddr_slide)), respectively.
I know it's not possible to have an exact port for Linux, because dyld functions deal with mach-o files and Linux uses ELF files.
So, is there an equivalent of the dyld library for Linux. Or, at least, is there an equivalent of those two functions, _dyld_register_func_for_add_image and _dyld_register_func_for_remove_image, in any Linux library? Or will I have to implement my own versions of these two by myself, which is not so hard, but I would have to find a way to make dlopen and dlclose call callback functions whenever they get called.
EDIT
To must things clearer, I need to make a library that has a callback function that must be called whenever an external library is dynamically loaded by dlopen. My callback function must perform some operations on any dynamic loaded library.

Yes, it is called dlopen(3) using the -ldl standard library
More precisely:
compile your plugin's source code using the -fPIC flag to get position independent code object files *.pic.o
make a shared library plugin by linking with gcc -shared your *.pic.o files (and you can also link another shared library).
use GCC function attributes, notably constructor and destructor functions (or static C++ data with explicit constructors & destructors, hence the name). The functions with __attribute__((constructor)) are called during dlopen time of your plugin, those with __attribute__((destructor)) in your plugin are called during dlclose time
linking the main program with the -rdynamic attribute is useful & needed, as soon as the plugin call some functions in the main program.
don't forget to declare extern "C" your C++ plugin functions (needed for the program)
use dlsym inside your main program to fetch function or data addresses inside your plugin.
There is indeed no hooks for dlopen like _dyld_register_func_for_add_image does. You may want to use constructor functions and/or dl_iterate_phdr(3) to mimic that.
If you can change the plugin (the shared object which you dlopen) you could play constructor tricks inside to mimic such hooks. Otherwise, use some own convention (e.g. that a plugin having a module_start function gets that module_start function called just after dlopen etc...).
Some libraries are wrapping dlopen into something of higher level. For example Qt has QPluginLoader & QLibrary etc...
There is also the LD_PRELOAD trick (perhaps you might redefine your own dlopen & dlclose thru such a trick, and have your modified functions do the hooks). The ifunc function attribute might also be relevant.
And since Gnu Libc is free software providing the dlopen - there is also MUSL Libc, you could patch it to suit your needs. dladdr(3) could be useful too!
addenda
If you are making your own runtime for some Objective-C, you should know well the conventions of the Objective-C compiler using that runtime, and you probably could have your own module loader, instead of overloading dlopen...

Related

Error while executing the fortran code through gfortran [duplicate]

I am trying to build a Fortran program, but I get errors about an undefined reference or an unresolved external symbol. I've seen another question about these errors, but the answers there are mostly specific to C++.
What are common causes of these errors when writing in Fortran, and how do I fix/prevent them?
This is a canonical question for a whole class of errors when building Fortran programs. If you've been referred here or had your question closed as a duplicate of this one, you may need to read one or more of several answers. Start with this answer which acts as a table of contents for solutions provided.
A link-time error like these messages can be for many of the same reasons as for more general uses of the linker, rather than just having compiled a Fortran program. Some of these are covered in the linked question about C++ linking and in another answer here: failing to specify the library, or providing them in the wrong order.
However, there are common mistakes in writing a Fortran program that can lead to link errors.
Unsupported intrinsics
If a subroutine reference is intended to refer to an intrinsic subroutine then this can lead to a link-time error if that subroutine intrinsic isn't offered by the compiler: it is taken to be an external subroutine.
implicit none
call unsupported_intrinsic
end
With unsupported_intrinsic not provided by the compiler we may see a linking error message like
undefined reference to `unsupported_intrinsic_'
If we are using a non-standard, or not commonly implemented, intrinsic we can help our compiler report this in a couple of ways:
implicit none
intrinsic :: my_intrinsic
call my_intrinsic
end program
If my_intrinsic isn't a supported intrinsic, then the compiler will complain with a helpful message:
Error: ‘my_intrinsic’ declared INTRINSIC at (1) does not exist
We don't have this problem with intrinsic functions because we are using implicit none:
implicit none
print *, my_intrinsic()
end
Error: Function ‘my_intrinsic’ at (1) has no IMPLICIT type
With some compilers we can use the Fortran 2018 implicit statement to do the same for subroutines
implicit none (external)
call my_intrinsic
end
Error: Procedure ‘my_intrinsic’ called at (1) is not explicitly declared
Note that it may be necessary to specify a compiler option when compiling to request the compiler support non-standard intrinsics (such as gfortran's -fdec-math). Equally, if you are requesting conformance to a particular language revision but using an intrinsic introduced in a later revision it may be necessary to change the conformance request. For example, compiling
intrinsic move_alloc
end
with gfortran and -std=f95:
intrinsic move_alloc
1
Error: The intrinsic ‘move_alloc’ declared INTRINSIC at (1) is not available in the current standard settings but new in Fortran 2003. Use an appropriate ‘-std=*’ option or enable ‘-fall-intrinsics’ in order to use it.
External procedure instead of module procedure
Just as we can try to use a module procedure in a program, but forget to give the object defining it to the linker, we can accidentally tell the compiler to use an external procedure (with a different link symbol name) instead of the module procedure:
module mod
implicit none
contains
integer function sub()
sub = 1
end function
end module
use mod, only :
implicit none
integer :: sub
print *, sub()
end
Or we could forget to use the module at all. Equally, we often see this when mistakenly referring to external procedures instead of sibling module procedures.
Using implicit none (external) can help us when we forget to use a module but this won't capture the case here where we explicitly declare the function to be an external one. We have to be careful, but if we see a link error like
undefined reference to `sub_'
then we should think we've referred to an external procedure sub instead of a module procedure: there's the absence of any name mangling for "module namespaces". That's a strong hint where we should be looking.
Mis-specified binding label
If we are interoperating with C then we can specify the link names of symbols incorrectly quite easily. It's so easy when not using the standard interoperability facility that I won't bother pointing this out. If you see link errors relating to what should be C functions, check carefully.
If using the standard facility there are still ways to trip up. Case sensitivity is one way: link symbol names are case sensitive, but your Fortran compiler has to be told the case if it's not all lower:
interface
function F() bind(c)
use, intrinsic :: iso_c_binding, only : c_int
integer(c_int) :: f
end function f
end interface
print *, F()
end
tells the Fortran compiler to ask the linker about a symbol f, even though we've called it F here. If the symbol really is called F, we need to say that explicitly:
interface
function F() bind(c, name='F')
use, intrinsic :: iso_c_binding, only : c_int
integer(c_int) :: f
end function f
end interface
print *, F()
end
If you see link errors which differ by case, check your binding labels.
The same holds for data objects with binding labels, and also make sure that any data object with linkage association has matching name in any C definition and link object.
Equally, forgetting to specify C interoperability with bind(c) means the linker may look for a mangled name with a trailing underscore or two (depending on compiler and its options). If you're trying to link against a C function cfunc but the linker complains about cfunc_, check you've said bind(c).
Not providing a main program
A compiler will often assume, unless told otherwise, that it's compiling a main program in order to generate (with the linker) an executable. If we aren't compiling a main program that's not what we want. That is, if we're compiling a module or external subprogram, for later use:
module mod
implicit none
contains
integer function f()
f = 1
end function f
end module
subroutine s()
end subroutine s
we may get a message like
undefined reference to `main'
This means that we need to tell the compiler that we aren't providing a Fortran main program. This will often be with the -c flag, but there will be a different option if trying to build a library object. The compiler documentation will give the appropriate options in this case.
There are many possible ways you can see an error like this. You may see it when trying to build your program (link error) or when running it (load error). Unfortunately, there's rarely a simple way to see which cause of your error you have.
This answer provides a summary of and links to the other answers to help you navigate. You may need to read all answers to solve your problem.
The most common cause of getting a link error like this is that you haven't correctly specified external dependencies or do not put all parts of your code together correctly.
When trying to run your program you may have a missing or incompatible runtime library.
If building fails and you have specified external dependencies, you may have a programming error which means that the compiler is looking for the wrong thing.
Not linking the library (properly)
The most common reason for the undefined reference/unresolved external symbol error is the failure to link the library that provides the symbol (most often a function or subroutine).
For example, when a subroutine from the BLAS library, like DGEMM is used, the library that provides this subroutine must be used in the linking step.
In the most simple use cases, the linking is combined with compilation:
gfortran my_source.f90 -lblas
The -lblas tells the linker (here invoked by the compiler) to link the libblas library. It can be a dynamic library (.so, .dll) or a static library (.a, .lib).
In many cases, it will be necessary to provide the library object defining the subroutine after the object requesting it. So, the linking above may succeed where switching the command line options (gfortran -lblas my_source.f90) may fail.
Note that the name of the library can be different as there are multiple implementations of BLAS (MKL, OpenBLAS, GotoBLAS,...).
But it will always be shortened from lib... to l... as in liopenblas.so and -lopenblas.
If the library is in a location where the linker does not see it, you can use the -L flag to explicitly add the directory for the linker to consider, e.g.:
gfortran -L/usr/local/lib -lopenblas
You can also try to add the path into some environment variable the linker searches, such as LIBRARY_PATH, e.g.:
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib
When linking and compilation are separated, the library is linked in the linking step:
gfortran -c my_source.f90 -o my_source.o
gfortran my_source.o -lblas
Not providing the module object file when linking
We have a module in a separate file module.f90 and the main program program.f90.
If we do
gfortran -c module.f90
gfortran program.f90 -o program
we receive an undefined reference error for the procedures contained in the module.
If we want to keep separate compilation steps, we need to link the compiled module object file
gfortran -c module.f90
gfortran module.o program.f90 -o program
or, when separating the linking step completely
gfortran -c module.f90
gfortran -c program.f90
gfortran module.o program.o -o program
Problems with the compiler's own libraries
Most Fortran compilers need to link your code against their own libraries. This should happen automatically without you needing to intervene, but this can fail for a number of reasons.
If you are compiling with gfortran, this problem will manifest as undefined references to symbols in libgfortran, which are all named _gfortran_.... These error messages will look like
undefined reference to '_gfortran_...'
The solution to this problem depends on its cause:
The compiler library is not installed
The compiler library should have been installed automatically when you installed the compiler. If the compiler did not install correctly, this may not have happened.
This can be solved by correctly installing the library, by correctly installing the compiler. It may be worth uninstalling the incorrectly installed compiler to avoid conflicts.
N.B. proceed with caution when uninstalling a compiler: if you uninstall the system compiler it may uninstall other necessary programs, and may render other programs unusable.
The compiler cannot find the compiler library
If the compiler library is installed in a non-standard location, the compiler may be unable to find it. You can tell the compiler where the library is using LD_LIBRARY_PATH, e.g. as
export LD_LIBRARY_PATH="/path/to/library:$LD_LIBRARY_PATH"
If you can't find the compiler library yourself, you may need to install a new copy.
The compiler and the compiler library are incompatible
If you have multiple versions of the compiler installed, you probably also have multiple versions of the compiler library installed. These may not be compatible, and the compiler might find the wrong library version.
This can be solved by pointing the compiler to the correct library version, e.g. by using LD_LIBRARY_PATH as above.
The Fortran compiler is not used for linking
If you are linking invoking the linker directly, or indirectly through a C (or other) compiler, then you may need to tell this compiler/linker to include the Fortran compiler's runtime library. For example, if using GCC's C frontend:
gcc -o program fortran_object.o c_object.o -lgfortran

How does LD_PRELOAD update the library function

I know we can use the LD_PRELOAD trick to replace a libc function, e.g., malloc(). So how exactly LD_PRELOAD works? Does it replace the GOT/PLT entry?
It puts the preloaded library at the start of the symbol search path. This way, all references to malloc are bound to its implementation, and not the implementation in libc.
In glibc, you can see the code in elf/rtld.c (the call to _dl_map_object_deps) and the implementation of the _dl_map_object_deps function in elf/dl-deps.c.

Why are some foreign functions statically linked while others are dynamically linked?

I'm working on a program that needs to manipulate git repositories. I've decided to use libgit2. Unfortunately, the haskell bindings for it are several years out of date and lack several functions that I require. Because of this I've decided to write the portions that use libgit2 in C and call them through the FFI. For demonstration purposes one of them is called git_update_repo.
git_update_repo works perfectly when used in a pure C program, however when it's called from haskell an assertion fails indicating that the libgit2 global init function, git_libgit2_init, hasn't been called. But, git_libgit2_init is called by git_update_repo. And if I use gdb I can see that git_libgit2_init is indeed called and reports that the initialization has been successful.
I've used nm to examine the executables and found something interesting. In a pure C executable, all the libgit2 functions are dynamically linked (as expected). However, in my haskell executable, git_libgit2_init is dynamically linked, while the rest of the libgit2 functions are statically linked. I'm certain that this mismatch is the cause of my issue.
So why do certain functions get linked dynamically and others statically? How can I change this?
The relevant settings in my .cabal file are
cc-options: -g
c-sources:
src/git-bindings.c
extra-libraries:
git2

Force mapping between symbols and shared libraries

I have an executable with four shared libraries and the dependency tree look like this: Executable app does a dlopen of foo.so and bar.so. foo.so in turn links to fooHelper.so and bar.so links to barHelper.so.
Now, the issue is that fooHelper.so and barHelper.so have some of the same symbols. For instance, let us say we have a func with different implementations in fooHelper.so and barHelper.so. Is there a way to force foo.so to use fooHelper.so's implementation and bar.so to use barHelper.so's? What happens at present is that depending on the order of linking of the helpers, only one of the implementations of func is used by both foo.so and bar.so. This is because of the default Unix linkage model, if the definition of a symbol is already loaded, then any other definitions from shared libraries loaded subsequently are just discarded. Basically, func will be picked up from the helper library linked first. I need a way to explicitly specify the appropriate mapping without changing the source code of the shared libraries.
I'm working on Linux with g++ 4.4.
Is there a way to force foo.so to use fooHelper.so's implementation and bar.so to use barHelper.so's?
Yes: that's what RTLD_LOCAL is for (when dlopening foo.so and bar.so).
RTLD_LOCAL
This is the converse of RTLD_GLOBAL, and the default if neither flag
is specified. Symbols defined in this library are not made available
to resolve references in subsequently loaded libraries.
If both funcs happen to be in the same name-space, you're in a bit of trouble - if you are programming in C. The term to look for is "function overloading". There have been previous discussions on this topic, e.g. this one:
function overloading in C
EDIT: http://litdream.blogspot.de/2007/03/dynamic-loading-using-dlopen-api-in-c.html

Extraneous Library Linkage

I have a question which may be somewhat silly because I'm pretty sure I may know the answer already.
Suppose you have static library A, and dynamic shared object library B and your program C under linux. Suppose that library A calls functions from library B and your program calls functions from library A. Now suppose that all functions that C calls in A make no use of functions in B.
To compile C will it be enough to link just A and omit B and furthermore can your program C be run on a system without library B installed?
If your program calls functions in A that don't reference B then B is not required either at link or load time, assuming that the functions in A are in separate compilation units, which is usually the case for a library.
The linker will pull the functions from the library that C uses and since none of them call functions in B, B will not be needed.
Holy placeholder name overload, batman. Let's first replace A, B, and C, with libstatic, libshared, and myapp to make things a little more legible:
Suppose you have static library libstatic, and
dynamic shared object library libshared and
your program myapp under linux. Suppose
that library libstatic calls functions from
library libshared and your program (myapp) calls
functions from library libstatic. Now suppose
that all functions that myapp calls in libstatic
make no use of functions in libshared.
To compile myapp will it be enough to link
just libstatic and omit libshared and furthermore can
your program myapp be run on a system
without library libshared installed?
So the way I understand your question, there is a library libstatic, some functions in which make use of libshared. You want to know: if I don't use any of the libstatic functions that are dependent on libshared, will myapp link and run without libshared?
The answer is yes, so long as two things are true:
The calls you make into libstatic do not depend on libshared directly or indirectly. Meaning that if myapp calls a function in libstatic which calls another function in libstatic which calls a function in libshared, then myapp is now dependent on libshared.
The calls you make into libstatic do not depend on any function in libstatic whose implementation appears in the same compilation unit (object file) with a call to libshared. The linker brings in code from the static library at the level of object files, not at the level of individual functions. And remember, this dependency is similarly chained, so if you call a function in foo.o, and something else in foo.o calls a function in bar.o, and something in bar.o depends on libshared, you're toast.
When you link in a static library into an application, only the object files that contain the symbols used (directly or indirectly) are linked. So if it turns out that none of the object files that myapp ends up needing from libstatic depend on libshared, then myapp doesn't depend on libshared.

Resources