whats are *.so.*.* libs - linux

When I do ls -l in /usr/lib I see lots of libs with "sameName.so.*.*" extension.
What is the significance of these extensions?
Why softlinks are created? what are their use?
One example will help a lot in understanding.

This is a trick used to version shared object files. It's a way of avoiding the dreaded DLL hell which came about because of lazy linking.
The advantage of lazy linking (or late binding) is that components of your executable can be changed without actually re linking those executables. This allows for bug fixes in third party components without having to ship a new executable, among other things.
The disadvantage is exactly the same as the advantage. Your executable can find that assumptions it made about the underlying libraries have been changed and this is likely to cause all sorts of issues.
Versioning of shared objects is one way to avoid this. Another would be to not share objects at all but that also has pros and cons which I won't get into here.
By way of example, let's say you have version 1 of xyz.so. You have a file and a symbolic link to that file:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.1
Now, when you create an executable file exe1, linking it with xyz.so, it will follow the symbolic link so that it stores xyz.so.1 in the executable as the thing it needs to load at runtime.
That way, when you upgrade the shared library thus:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1
-rw-r--r-- 1 pax paxgroup 67890 Nov 18 2009 xyz.so.2
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.2
your original executable exe1 will still load version 1 of the shared object.
However, any executables you create now (such as exe2) will be linked with version 2 of the shared object.
The actual implementation details may vary somewhat (I'm basing my answer on earlier UNIXes and Linux appears to do versioning a little more intelligently than just following symbolic links). IBM developerWorks has a nice article on how it's done here.
When you create a shared object, you give it both a real name and an soname. These are used to install the shared object (which creates both the object and a link to it).
So you can end up with the situation:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so.1 -> xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.1
with xyz.so.1.5 possessing the SONAME of xyz.so.1.
When the linker links in xyz.so, it follows the links all the way to xyz.so.1.5 and uses its SONAME of xyz.so.1 to store in the executable. Then, when you run the executable, it tries to load xyz.so.1 which will point to a specific xyz.so.1.N (not necessarily version 1.5).
So you could install xyz.so.1.6 and update the xyz.so.1 link to point to it instead and already-linked executables would use that instead.
One advantage of this multi-layer method is that you can have multiple potentially incompatible libraries of the same name (xyz.so.1.*, xyz.so.2.*) but, within each major version, you can freely upgrade them since they're supposed to be compatible.
When you link new executables:
Those linking with xyz.so will get the latest minor version of the latest major version.
Others linking with xyz.so.1 will get the latest minor version of a specific major version.
Still others linking with xyz.so.1.2 will get a specific minor version of a specific major version.

It's a versioning scheme for shared libraries. Every library should have 3 names:
Real name: the actual library name, libfoo.so.1.2.3
"SONAME": the name recorded in the executable, and the name dynamic linker looks for, libfoo.so.1.2. This name is actually written inside the library binary itself, and will be recorded in the executable at link time. It is usually a symlink to library's real name (usually latest version).
Linker name: the name you give to the linker when building your program. Usually links to the latest SONAME.
Example
Say you have libfoo version 1 installed: libfoo.so -> libfoo.so.1.0 -> libfoo.so.1.0.0. You build your program bar with -lfoo. it now links to libfoo and will load libfoo.so.1.0 at runtime due to SONAME. Then you upgrade to a patched but binary-compatible libfoo.so.1.0.1 by replacing real binary. bar still links to libfoo.so.1.0 and doesn't need to be rebuilt.
Now imagine you want to build a new program baz that takes advantage of incompatible changes in libfoo v1.1. You install new version and your system now have two versions installed in parallel:
libfoo.so.1.0 -> libfoo.so.1.0.1
libfoo.so -> libfoo.so.1.1 -> libfoo.so.1.1.0
Note linker name was updated to the latest version (this is the version corresponding to the headers you installed in /usr/include).
You build baz, and it links to libfoo.so and loads libfoo.so.1.1 at runtime. Not that bar still runs against libfoo.so.1.0 and doesn't need to be updated.

Related

Linux: dynamic loader selecting the wrong library?

I run two versions of subversion on my system, using different libraries. This worked until yesterday, with the old version picking up the system libraries in /usr/lib64, and the new version picking up the new libraries in ~/local/lib, but this has now broken.
The old version was installed with the distro, and I built the new one from source a few months ago. I built the new one in ~/local, with the new binary in ~/local/bin/svn, and the new libraries in ~/local/lib. I also created a new file in /etc/ld.so.conf.d, which included the single line /home/me/local/lib.
This worked fine until I tried to install Qt5 yesterday: I could run both the old Subversion (1.7.14), as /usr/bin/svn, and the new one (1.12.2), as ~/local/bin/svn.
If I now try to run the old svn it tells me that there's a missing symbol in one of the Subversion libraries. If I run
$ ldd /usr/bin/svn
it turns out that the old svn now tries to load the new libraries in ~/local/lib, so it doesn't work. Running ldconfig makes no difference. The old and new libraries are completely different, but somehow have the same version number:
$ ll /usr/lib64/libsvn_client-1.so.*
lrwxrwxrwx. 1 root root 24 Jan 3 11:37 /usr/lib64/libsvn_client-1.so.0 -> libsvn_client-1.so.0.0.0
-rwxr-xr-x. 1 root root 379656 Nov 20 2015 /usr/lib64/libsvn_client-1.so.0.0.0
$ ll /home/me/local/lib/libsvn_client-1.so*
lrwxrwxrwx. 1 me me 24 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so -> libsvn_client-1.so.0.0.0
lrwxrwxrwx. 1 me me 24 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so.0 -> libsvn_client-1.so.0.0.0
-rwxr-xr-x. 1 me me 2986816 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so.0.0.0
There's obviously something I don't understand about shared libs here. Presumably the dynamic loader selects the required library from the minor version numbers, but they're all 0 here. So how did this ever work? And surely svn doesn't ship with all-zero version numbers? Is it possible that the Qt5 install has caused a problem? Is there some way to fix this short of deleting one of the versions?

Multiple versions of same the same library in LD_LIBRARY_PATH [duplicate]

Windows provides the resource file for version information for an application and DLL. The resource file includes information like version, copyright and manufacturer.
We have a shared library and would like to add version information.
How can we do it on Linux with a shared library?
The short version is that you do this via the soname of the library. Read chapter 3 at http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html as well as chapter 3.3 ABI Versioning at http://www.akkadia.org/drepper/dsohowto.pdf
The best way to handle this is using libtool, which does the versioning for you.
Essentially, version information is not (or not primarily, don't know from my head) encoded in the library itself, but rather in its filename. Version numbers are normally given in three-dot format, with the major number increasing for each break in downward ABI compatibility, the middle for breaks in upward ABI compatibility, and the minor for patches that did not change the ABI.
Like qdot noted, symlinks in the lib directory provide the essential versioning. There is a symlink without a version number (libfoo.so) for the currently installed development headers, a symlink with a major number for each installed major version (libfoo.so.1) and a real file with the full version number. Normally, programs are linked to use libfoo.so.1 at runtime so that multiple major versions may coexist.
Linux uses the following strategy - you (the system maintainer) provide symlinks from a 'specific' shared library file, like this:
lrwxrwxrwx 1 root root 16 2011-09-22 14:36 libieee1284.so -> libieee1284.so.3
lrwxrwxrwx 1 root root 20 2011-09-22 14:36 libieee1284.so.3 -> libieee1284.so.3.2.2
-rw-r--r-- 1 root root 46576 2011-07-27 13:08 libieee1284.so.3.2.2
This way, developers can link either against -lieee1284 (any version ABI), or libieee1284.so.3 or even to the specific release and patch version (3.2.2)

But I don't *want* gcc to follow symlinks

Background:
I wrote a program that uses the OpenBLAS libraries for a very heterogeneous compute cluster. OpenBLAS uses different libraries for different architectures.
So on one machine 'ls -l /usr/lib64/libopenblas.so' results in
lrwxrwxrwx 1 root root 31 Mar 6 15:13 /usr/lib64/libopenblas.so -> libopenblas_barcelona-r0.2.6.so*
On another, the result is
lrwxrwxrwx 1 root root 33 Mar 4 09:43 /usr/lib64/libopenblas.so -> libopenblas_sandybridge-r0.2.6.so*
There may be others, but these are the two I have checked. Both implement the same API, just use different optimizations.
The problem:
When I compile my own shared object file using these I use
gcc ... -lopenblas and it compiles just fine, and it runs on machines with similar architecture (i.e., those with 'libopenblas_barcelona-r0.2.6.so'), but on other architectures it fails to run.
ldd shows that it links against libopenblas_barcelona-r0.2.6.so rather than libopenblas.so.
Is there any way I can tell gcc to link against the symlink, rather than the result of that symlink, so that it will be "right" regardless of the architecture of the machine it's running on?

How to do versioning of shared library?

Windows provides the resource file for version information for an application and DLL. The resource file includes information like version, copyright and manufacturer.
We have a shared library and would like to add version information.
How can we do it on Linux with a shared library?
The short version is that you do this via the soname of the library. Read chapter 3 at http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html as well as chapter 3.3 ABI Versioning at http://www.akkadia.org/drepper/dsohowto.pdf
The best way to handle this is using libtool, which does the versioning for you.
Essentially, version information is not (or not primarily, don't know from my head) encoded in the library itself, but rather in its filename. Version numbers are normally given in three-dot format, with the major number increasing for each break in downward ABI compatibility, the middle for breaks in upward ABI compatibility, and the minor for patches that did not change the ABI.
Like qdot noted, symlinks in the lib directory provide the essential versioning. There is a symlink without a version number (libfoo.so) for the currently installed development headers, a symlink with a major number for each installed major version (libfoo.so.1) and a real file with the full version number. Normally, programs are linked to use libfoo.so.1 at runtime so that multiple major versions may coexist.
Linux uses the following strategy - you (the system maintainer) provide symlinks from a 'specific' shared library file, like this:
lrwxrwxrwx 1 root root 16 2011-09-22 14:36 libieee1284.so -> libieee1284.so.3
lrwxrwxrwx 1 root root 20 2011-09-22 14:36 libieee1284.so.3 -> libieee1284.so.3.2.2
-rw-r--r-- 1 root root 46576 2011-07-27 13:08 libieee1284.so.3.2.2
This way, developers can link either against -lieee1284 (any version ABI), or libieee1284.so.3 or even to the specific release and patch version (3.2.2)

How do applications resolve to different versions of shared libraries at run time?

I'm a noob to how shared libraries work on linux. I am trying to understand how do applications resolve different revisions of the same shared library at run-time on linux.
As far as I understand, a shared library has three "names", for example,
libmy.so.1.2 (real-name i.e. the actual obj file)
libmy.so.1 (SONAME, which is embedded in the actual obj file)
libmy.so (linker name, provided to the linker at link time and embedded in executable)
When you install the library via LDCONFIG, it will create the following symbolic links
(2) => (1)
(3) => (2)
Now lets say I compile another version of the same library with the following real-name,
libmy.so.2.0. The SONAME by guidelines would be libmy.so.2.0
At application link time what is the linker name that I would provide with the "-l" flag. Following the guidelines I read (http://www.dwheeler.com/program-library/Program-Library-HOWTO/x36.html), wouldn't it have to be libmy.so and if so, how will both versions of the obj file be distinguished ?
Versioning of shared objects works as follows:
When you create a shared object, you give it both a real name and an soname. These are used to install the shared object (which creates both the object and a link to it).
So you can end up with the situation:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so.1 -> xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.1
with xyz.so.1.5 possessing the SONAME of xyz.so.1.
When the linker links in xyz.so, it follows the links all the way to xyz.so.1.5 and uses its SONAME of xyz.so.1 to store in the executable. Then, when you run the executable, it tries to load xyz.so.1 which will point to a specific xyz.so.1.N (not necessarily version 1.5).
So you could install xyz.so.1.6 and update the xyz.so.1 link to point to it instead and already-linked executables would use that instead.
One advantage of this multi-layer method is that you can have multiple potentially incompatible libraries of the same name (xyz.so.1.*, xyz.so.2.*) but, within each major version, you can freely upgrade them since they're supposed to be compatible.
When you link new executables:
Those linking with xyz.so will get the latest minor version of the latest major version.
Others linking with xyz.so.1 will get the latest minor version of a specific major version.
Still others linking with xyz.so.1.2 will get a specific minor version of a specific major version.
Now keep that last paragraph in mind as we examine your comment:
Now lets say I compile another version of the same library with the following real-name, libmy.so.2.0. The SONAME by guidelines would be libmy.so.2.0.
No, I don't believe so. The soname would be more likely to be libmy.so.2 so that you can make minor updates to the 2.x stream and get the latest behaviour.
At application link time, if you specify -lmy, the linker will search for a file named libmy.so. It will find this file, and link you executable with it. If this file is a symbolic link, then your application will be linked with the target of the symlink.
Application link time is the place to specify which version of the dynamic library you want to use with your application.
A complement to #paxdiablo's answer bullet one:
ldconfig doesn't set up the linker names; typically this is done during library installation, and the linker name is simply created as a symbolic link to the ``latest'' soname or the latest real name.
it means xxx.so will link to a latest minor version of the latest major version dynamic library file.
I think this explains why:
Those linking with xyz.so will get the latest minor version of the latest major version.
But I don't know how installation setup linker name and which tool make it(apt-get?)
Libs have different versions in the name.
Packages with name "lib" have only libs and have different versions in the name.
System will compile only with the latest library,
unless you define a different one.
The application uses only those libraries that it needs.
Check ldd , readelf.
Apps contain a link .so and .pc file, check rpm system for what.
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging#Devel_Packages

Resources