I run two versions of subversion on my system, using different libraries. This worked until yesterday, with the old version picking up the system libraries in /usr/lib64, and the new version picking up the new libraries in ~/local/lib, but this has now broken.
The old version was installed with the distro, and I built the new one from source a few months ago. I built the new one in ~/local, with the new binary in ~/local/bin/svn, and the new libraries in ~/local/lib. I also created a new file in /etc/ld.so.conf.d, which included the single line /home/me/local/lib.
This worked fine until I tried to install Qt5 yesterday: I could run both the old Subversion (1.7.14), as /usr/bin/svn, and the new one (1.12.2), as ~/local/bin/svn.
If I now try to run the old svn it tells me that there's a missing symbol in one of the Subversion libraries. If I run
$ ldd /usr/bin/svn
it turns out that the old svn now tries to load the new libraries in ~/local/lib, so it doesn't work. Running ldconfig makes no difference. The old and new libraries are completely different, but somehow have the same version number:
$ ll /usr/lib64/libsvn_client-1.so.*
lrwxrwxrwx. 1 root root 24 Jan 3 11:37 /usr/lib64/libsvn_client-1.so.0 -> libsvn_client-1.so.0.0.0
-rwxr-xr-x. 1 root root 379656 Nov 20 2015 /usr/lib64/libsvn_client-1.so.0.0.0
$ ll /home/me/local/lib/libsvn_client-1.so*
lrwxrwxrwx. 1 me me 24 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so -> libsvn_client-1.so.0.0.0
lrwxrwxrwx. 1 me me 24 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so.0 -> libsvn_client-1.so.0.0.0
-rwxr-xr-x. 1 me me 2986816 Oct 8 16:32 /home/me/local/lib/libsvn_client-1.so.0.0.0
There's obviously something I don't understand about shared libs here. Presumably the dynamic loader selects the required library from the minor version numbers, but they're all 0 here. So how did this ever work? And surely svn doesn't ship with all-zero version numbers? Is it possible that the Qt5 install has caused a problem? Is there some way to fix this short of deleting one of the versions?
Related
There are always jpeg decoder libraries pre-installed on Linux like:
/usr/lib/x86_64-linux-gnu/libjpeg.so
/usr/lib/x86_64-linux-gnu/libjpeg.so.62
/usr/lib/x86_64-linux-gnu/libjpeg.so.62.0.0
/usr/lib/x86_64-linux-gnu/libjpeg.so.8
/usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2
What is the difference between the so library? Does libjpeg.so.62 build from libjpeg-turbo?
Firstly, if you run:
ls -l /usr/lib/x86_64-linux-gnu/*jpeg*
you will see that most of the files are just symlinks to the one with the full version, so programs can link against the latest one by specifying an unversioned library in the knowledge that it will point to the latest version:
lrwxrwxrwx 1 root root 17 Oct 20 2016 libjpeg.so -> libjpeg.so.62.2.0
lrwxrwxrwx 1 root root 17 Oct 20 2016 libjpeg.so.62 -> libjpeg.so.62.2.0
-rw-r--r-- 1 root root 436224 Oct 20 2016 libjpeg.so.62.2.0
Secondly, unfortunately I don't have the same files as you else I would help further, but in general, you can find which package a given file comes from like this:
dpkg -S someFile
So, on my system, I can see that libjpeg.a for example, comes from package libjpeg62-turbo-dev
dpkg -S libjpeg.a
libjpeg62-turbo-dev:amd64: /usr/lib/x86_64-linux-gnu/libjpeg.a
Recently I ran into a problem with the Python function ctypes.util.find_library. The function is used to locate shared libraries by name; for example, CuPy uses it to locate cuDNN. In my case, I had several versions of cuDNN installed, and it picked up the latest (as per the documentation). However, the contents of the directory look like this:
$ l /usr/local/cuda-8.0/lib64 | grep -i cudnn
lrwxrwxrwx 1 root root 13 Oct 3 08:21 libcudnn.so -> libcudnn.so.6*
lrwxrwxrwx 1 1000 users 17 Jul 27 2016 libcudnn.so.5 -> libcudnn.so.5.1.5*
-rwxrwxr-x 1 1000 users 79337624 Jul 27 2016 libcudnn.so.5.1.5*
lrwxrwxrwx 1 root root 18 Oct 3 08:21 libcudnn.so.6 -> libcudnn.so.6.0.21*
-rwxr-xr-x 1 1000 users 154322864 Apr 12 2017 libcudnn.so.6.0.21*
lrwxrwxrwx 1 root root 17 Oct 2 10:32 libcudnn.so.7 -> libcudnn.so.7.0.3*
-rwxrwxr-x 1 1000 1000 217188104 Sep 16 05:09 libcudnn.so.7.0.3*
-rw-r--r-- 1 1000 users 143843808 Apr 12 2017 libcudnn_static.a
Even though the latest version is 7.0.3, judging from the symbolic link hierarchy, I would have expected version 6.0.21 to be picked up. My questions are:
Which version would the gcc (or clang) toolchain have picked up during compile-time?
Which version would a C/C++ executable have picked up during run-time?
Is there any kind of information source (an article, a man page, a book, ...) out there that contains explicitly the answers to the first two questions? I tried googling it, but nothing definitive came up.
Traditionally you'd build with a command like gcc -lcudnn. This would find libcudnn.so which points to libcudnn.so.6 which points to libcudnn.so.6.0.21. So libcudnn.so.6.0.21 would be linked at build time.
Traditionally a shared library will contain a "SONAME" which indicates the ABI compatible version to be loaded at runtime. I'm pretty sure that would be libcudnn.so.6 in this case. So building against libcudnn.so.6.0.21 would give you a runtime dependency on libcudnn.so.6 (you can verify this using ldd myprog | grep libcudnn.so).
Probably, but that's not an on-topic question for Stack Overflow ("recommending an off-site resource").
Background:
I wrote a program that uses the OpenBLAS libraries for a very heterogeneous compute cluster. OpenBLAS uses different libraries for different architectures.
So on one machine 'ls -l /usr/lib64/libopenblas.so' results in
lrwxrwxrwx 1 root root 31 Mar 6 15:13 /usr/lib64/libopenblas.so -> libopenblas_barcelona-r0.2.6.so*
On another, the result is
lrwxrwxrwx 1 root root 33 Mar 4 09:43 /usr/lib64/libopenblas.so -> libopenblas_sandybridge-r0.2.6.so*
There may be others, but these are the two I have checked. Both implement the same API, just use different optimizations.
The problem:
When I compile my own shared object file using these I use
gcc ... -lopenblas and it compiles just fine, and it runs on machines with similar architecture (i.e., those with 'libopenblas_barcelona-r0.2.6.so'), but on other architectures it fails to run.
ldd shows that it links against libopenblas_barcelona-r0.2.6.so rather than libopenblas.so.
Is there any way I can tell gcc to link against the symlink, rather than the result of that symlink, so that it will be "right" regardless of the architecture of the machine it's running on?
I forgot to enable building libs for profiling in my ~/.cabal/config before installing a bunch of packages on a new machine and now a --reinstall world to try to fix the situation has left everything broken (naturally).
I think it's a better use of my time to just purge everything. How do I do that correctly?
Cabal doesn't keep track of what it's installed, it just uses ghc's library mechanism (or that of some other compiler if you're not using ghc), so you can use rm -r ~/.ghc to remove all locally-installed libraries.
If you have multiple ghc's installed, and you only want to remove the libs for a specific ghc, delete the subdirectory corresponding to whichever ghc you want to remove.
For example, I could remove everything I've installed for ghc-7.6.0 with rm -r ~/.ghc/x86_64-linux-7.6.0.20120810
You can also use this to preserve your ghci_history if you like.
ll ~/.ghc/
total 24
-rw-r--r-- 1 johnl johnl 2300 Aug 21 11:47 ghci_history
drwxr-xr-x 3 johnl johnl 4096 Jun 17 19:09 x86_64-linux-6.12.3
drwxr-xr-x 3 johnl johnl 4096 May 17 08:17 x86_64-linux-7.2.1
drwxr-xr-x 3 johnl johnl 4096 May 16 17:34 x86_64-linux-7.4.1
drwxr-xr-x 3 johnl johnl 4096 Jun 15 08:21 x86_64-linux-7.4.2
drwxrwxr-x 3 johnl johnl 4096 Aug 15 12:37 x86_64-linux-7.6.0.20120810
Edit: ~/.cabal/world is a list of installed packages with version constraints as specified by the user. So in most cases, it would include e.g. mtl -any. If you've installed packages with specific versions, such as by issuing cabal install mtl-2.1.1, it will record that version. You should be able to either delete the world file and start over, or if you look it over and the dependencies are acceptable, you could try just running cabal install world. Or you could ignore it and not use cabal's world support (that's what I do).
This ghc-pkg-reset shell script cleans both ~/.ghc/ and ~/.cabal/, a little more selectively. (And the accompanying ghc-pkg-clean script can help recover from a confused package installation, perhaps avoiding a reset.) I find these useful.
When I do ls -l in /usr/lib I see lots of libs with "sameName.so.*.*" extension.
What is the significance of these extensions?
Why softlinks are created? what are their use?
One example will help a lot in understanding.
This is a trick used to version shared object files. It's a way of avoiding the dreaded DLL hell which came about because of lazy linking.
The advantage of lazy linking (or late binding) is that components of your executable can be changed without actually re linking those executables. This allows for bug fixes in third party components without having to ship a new executable, among other things.
The disadvantage is exactly the same as the advantage. Your executable can find that assumptions it made about the underlying libraries have been changed and this is likely to cause all sorts of issues.
Versioning of shared objects is one way to avoid this. Another would be to not share objects at all but that also has pros and cons which I won't get into here.
By way of example, let's say you have version 1 of xyz.so. You have a file and a symbolic link to that file:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.1
Now, when you create an executable file exe1, linking it with xyz.so, it will follow the symbolic link so that it stores xyz.so.1 in the executable as the thing it needs to load at runtime.
That way, when you upgrade the shared library thus:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1
-rw-r--r-- 1 pax paxgroup 67890 Nov 18 2009 xyz.so.2
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.2
your original executable exe1 will still load version 1 of the shared object.
However, any executables you create now (such as exe2) will be linked with version 2 of the shared object.
The actual implementation details may vary somewhat (I'm basing my answer on earlier UNIXes and Linux appears to do versioning a little more intelligently than just following symbolic links). IBM developerWorks has a nice article on how it's done here.
When you create a shared object, you give it both a real name and an soname. These are used to install the shared object (which creates both the object and a link to it).
So you can end up with the situation:
pax> ls -al xyz*
-rw-r--r-- 1 pax paxgroup 12345 Nov 18 2009 xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so.1 -> xyz.so.1.5
lrwxrwxrwx 1 pax paxgroup 0 Nov 18 2009 xyz.so -> xyz.so.1
with xyz.so.1.5 possessing the SONAME of xyz.so.1.
When the linker links in xyz.so, it follows the links all the way to xyz.so.1.5 and uses its SONAME of xyz.so.1 to store in the executable. Then, when you run the executable, it tries to load xyz.so.1 which will point to a specific xyz.so.1.N (not necessarily version 1.5).
So you could install xyz.so.1.6 and update the xyz.so.1 link to point to it instead and already-linked executables would use that instead.
One advantage of this multi-layer method is that you can have multiple potentially incompatible libraries of the same name (xyz.so.1.*, xyz.so.2.*) but, within each major version, you can freely upgrade them since they're supposed to be compatible.
When you link new executables:
Those linking with xyz.so will get the latest minor version of the latest major version.
Others linking with xyz.so.1 will get the latest minor version of a specific major version.
Still others linking with xyz.so.1.2 will get a specific minor version of a specific major version.
It's a versioning scheme for shared libraries. Every library should have 3 names:
Real name: the actual library name, libfoo.so.1.2.3
"SONAME": the name recorded in the executable, and the name dynamic linker looks for, libfoo.so.1.2. This name is actually written inside the library binary itself, and will be recorded in the executable at link time. It is usually a symlink to library's real name (usually latest version).
Linker name: the name you give to the linker when building your program. Usually links to the latest SONAME.
Example
Say you have libfoo version 1 installed: libfoo.so -> libfoo.so.1.0 -> libfoo.so.1.0.0. You build your program bar with -lfoo. it now links to libfoo and will load libfoo.so.1.0 at runtime due to SONAME. Then you upgrade to a patched but binary-compatible libfoo.so.1.0.1 by replacing real binary. bar still links to libfoo.so.1.0 and doesn't need to be rebuilt.
Now imagine you want to build a new program baz that takes advantage of incompatible changes in libfoo v1.1. You install new version and your system now have two versions installed in parallel:
libfoo.so.1.0 -> libfoo.so.1.0.1
libfoo.so -> libfoo.so.1.1 -> libfoo.so.1.1.0
Note linker name was updated to the latest version (this is the version corresponding to the headers you installed in /usr/include).
You build baz, and it links to libfoo.so and loads libfoo.so.1.1 at runtime. Not that bar still runs against libfoo.so.1.0 and doesn't need to be updated.