I am trying to make a Qt5 part of my source tree, so I haven't installed it on my machine, just copied it from source control. I am having a problem when I try to run uic.exe:
stiopa#stiopa-VirtualBox:~/ct/LinuxLibs/Qt/bin > ./uic
./uic: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory
I am still getting the same error even when I copy the libQt5Core library to bin directory. How is uic looking for shared libraries? Is there any environment variable I need to set to fix it?
This is yet another case of not putting the dependent shared libraries in a defined location that is supported by the program.
If you're planning on doing the 'copy the files to the same directory as the executable', the fast solution is to reference the directory in the library load path; e.g. if the binary is in $HOME/foo, you do:
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}$HOME/foo
This adds or makes $HOME/foo the run-time-linker's load path. As a result, any programs you run will look in this directory for libraries, as well as the default set for the OS (defined by the ld.so configuration), as well as the paths that are defined within the application itself (the rpath).
If you're going to follow this route, what you can do is to move the binary to target.bin, create a target bash script, which invokes the bin file automatically; e.g.
#!/bin/bash -p
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}$(dirname $0)
exec $0.bin "$*"
A secondary mechanism which will permit you to change the search location for a binary; without requiring an environment variable insert is to modify the binary so that it searches in different locations than it usually does; this takes advantage of some features in the run-time linker (which looks for libraries).
There is a program called chrpath, which can be added by various package managers, which allows you to edit the rpath directly. In this case; you can change the additional search path of the binary using:
chrpath -r '$ORIGIN' foo
This means that the program will look in the same directory as the binary for .so files, thus allowing it to run.
Related
I have an excutable and a shared library in same directory, the executable running well, however, after moving the shared library to other pos, I can not run the executable showing can not find libxxx.so
I want to know why it happens?
You can move the library to some standard library path. It is defined in /etc/ld.so.conf. The loader will only find libraries defined in there. In the other hand, you can also use LD_LIBRARY_PATH env variable to put your shared library path. It makes sure the path is searched first
For a set of programs written in most languages (C for instance) a script can normally run those programs without any sort of interference between dynamic link libraries and with no special hand holding so long as they are all found on PATH. That is, the following will work:
#!/bin/bash
prog1
prog2
prog3
However, if these three programs are written in Python and they import conflicting package versions then to run each one successfully it must either be installed into a virtualenv or each must have a separate site-packages directory which is referenced by PYTHONPATH. Either way they need a set up and possibly a tear down before running. That is, for virtualenv:
#!/bin/bash
source $PROG1_ROOT/bin/activate
prog1
deactivate
source $PROG2_ROOT/bin/activate
prog2
deactivate
source $PROG3_ROOT/bin/activate
prog3
deactivate
and for separate site-packages:
#!/bin/bash
export PYTHONPATH=$PROG1_ROOT/lib/python3.6/site-packages
prog1
export PYTHONPATH=$PROG2_ROOT/lib/python3.6/site-packages
prog2
export PYTHONPATH=$PROG3_ROOT/lib/python3.6/site-packages
prog3
This problem results because
import pkg_resources
(at least through Python3.6) cannot reliably import the proper versions when multiple versions of a package share the same site-package directory, even if __requires__ precedes it listing all the version restrictions.
It occurs to me that if PYTHONPATH, or some equivalent, could be specified relative to the program instead of the $PWD, and some consistency in directory layout was observed, then it would only have to be set once. That is, if prog1 is in $PROG1_ROOT/bin and its libraries are in $PROG1_ROOT/lib/python3.6/site-packages, then setting PYTHONPATH to "../lib/python3.6/site-packages" would work not only for prog1, but also for prog2, prog3, and for as many more as are needed through progN.
However, PYTHONPATH is normally provided as an absolute path, and relative paths are I believe with respect to $PWD, not to the python program (prog1). Is there some other Python path variable which has the desired property? Failing that, is there some type of file which could be dropped into $PROG1_ROOT/bin which would be normally picked up by a python program when it starts and which could direct it to use $PROG1_ROOT/lib/python3.6/site-packages? It would be OK to have either the relative or absolute path in that file, although the former would still be preferred because then one could move the entire PROG1_ROOT directory tree to another location in the file system without having to rewrite this special file. I really want to avoid solutions which would require modifying prog1 etc. themselves (ie, prog1 in the example).
Thanks.
EDITED:
I wrote this:
https://sourceforge.net/projects/python-devirtualizer/
to implement some of these ideas. At this point it is Linux (or at least POSIX) specific. It slightly modifies python scripts in a package's "bin" directory by changing the first line, and it "wraps" everything in that directory with a replacement native binary which injects a custom PYTHONPATH into the true target's environment. That binary looks up its location using a function from libSDL2 and then specifies the PYTHONPATH relative to that. So far it has worked pretty well, and the "programs" in installed python packages (the "bin" directory's contents) are run based on PATH just like any other program, no futzing about with PYTHONPATH in the shell.
Making search paths relative to the executable is a Very Bad Idea (TM). Move the executable or libraries around, all hell breaks loose. Some enterprising miscreant might notice the path settings and place a script just right to get their own doctored libraries (or just flawed old versions) to be used. And so on.
Clean up the misbehaving scripts. Chances are that by using old versions they are vulnerable to by now fixed security boo-boos, or other misbehaviours. Or find a way to load the stuff in the script itself.
So I am working on a project that is intended to run on a remote server. I develop the program on a local pc, compile it, then upload it to the remote server. Both the local pc and the remote server are run on CentOS 7.7.
The program is developed using the CLion IDE, configured with CMake. The program depends a few shared libraries, which are supposed to link to the executable according to what I wrote in CMake. At my local PC, I can compile and run the program perfectly. However, after I scp the whole directory of the project to the remote server, the executable fails to run. It cannot find any of the ".so" files, according to what ldd says.
This is my CMakeList.txt, with every path being relative path, instead of absolute path.
cmake_minimum_required(VERSION 3.15)
project(YS_Test)
set(CMAKE_CXX_STANDARD 11)
set(SOURCE_PATH_ src)
file(GLOB SOURCE_FILES_ ${SOURCE_PATH_}/*.*)
set(PROJECT_LIBS_ libTapQuoteAPI.so libTapTradeAPI.so libTapDataCollectAPI.so)
include_directories(api/include)
link_directories(api/lib/linux)
add_executable(YS_Test ${SOURCE_FILES_})
target_link_libraries(YS_Test ${PROJECT_LIBS_})
Please do not tell me to set LD_LIBRARY_PATH to fix my issue. The program worked fine on my local pc without LD_LIBRARY_PATH, so I expect it to run on the remote server without LD_LIBRARY_PATH. I would like to know what is really going on here, instead of a work around. Thanks!
If I understand your problem correctly, you want to ship your compiled YS_Test program along with some dependencies and have it run on a remote server. By default an executable will only look in the directories configured in /etc/ld.so, which will not include the deploy path.
Note: Typically you do not deploy your entire build directory but only the compiled artifacts and dependencies. For this answer I will assume you deploy the binary and its dependencies to the same directory.
You have two options:
Require users of your program to set LD_LIBRARY_PATH, either by themselves or by a wrapper script. This variable will instruct the dynamic linker to look in the specified directories as well. Even if you do not like this solution, it is by far the most common approach.
Add -Wl,-rpath='$ORIGIN' to your linker options. This will add a DT_RUNPATH attribute to the executable's dynamic section. As you are using CMake you can also set this using BUILD_RPATH and/or INSTALL_RPATH target properties.
The ld.so manpage describes this attribute as follows:
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
...
Using the directories specified in the DT_RUNPATH dynamic section
attribute of the binary if present.
The $ORIGIN part expands to the directory containing the program or shared
object.
If you really insist on shipping your build directory (eg during development), you can take a look at the CMake BUILD_RPATH_USE_ORIGIN property (and its usual global counterpart CMAKE_BUILD_RPATH_USE_ORIGIN), this will embed relative paths into binaries instead of absolute paths.
As you don't want a workaround (#Botje has given you two already), I will try an explanation instead. In your development machine, if you use this command:
ldd YS_Test
You will see all the shared libraries used by your program, with their corresponding paths. The libTapQuoteAPI.so libTapTradeAPI.so libTapDataCollectAPI.so are found at your 'api/lib/linux' directory, but resolved with full absolute paths. If you do the same at your server, some shared objects can't be resolved because they aren't at the same location.
If you use one of these commands (not sure which are available in Centos):
chrpath --list YS_Test
or
patchelf --print-rpath YS_Test
You will see the RPATH or RUNPATH tags embedded in your program. This is the path used by the Linux linker to locate dependencies that are outside the standard ld locations. You may find extended explanations on Internet about this, like this one or the Wikipedia article.
Breaking my promise, I give you a third workaround: use patchelf or chrpath at your server after scp to change the embedded RPATH tag, pointing it relative to $ORIGIN (which represents the program location).
Spack by default writes (on my system) into a directory spack-stage inside /tmp/. This directory can get full when building something big (e.g. GCC itself), at least in the shared system I am using:
fatal error: error writing to /tmp/ccPlIYSg.s: No space left on device
Is there any way to tell Spack to use a different location for spack-stage?
According to the spack documentation you just need to specify $TMP to point to a directory of your choice.
Builds can be faster in temporary directories on some (e.g., HPC) systems.
Specifying $tempdir will ensure use of the default temporary directory
(i.e., $TMP or $TMPDIR).
Another option that prevents conflicts and potential permission issues is
to specify ~/.spack/stage, which ensures each user builds in their home
directory.
I'm porting an SDK written in C++ from Windows to Linux. There are other binaries, but at its simplest, our SDK is this:
core.dll - implicitly loaded DLL ("libcore.so" shared library on Linux)
tests.exe - an app use to test the DLL (uses google test)
All of my binaries must live in one folder somewhere that apps can find. I've achieved that on Windows. I wanted to achieve the same thing in Linux. I'm failing miserably
To illustrate, Here's the basic project tree. We use CMake. After I build I've got
mysdk
|---CMakeLists.txt (has add_subdirectory() statements for "tests" and "core")
|---/tests (source code + CMakeLists.txt)
|---/core (source code + CMakeLists.txt)
|---/build (all build ouput, CMake output, etc)
|---tests (build output)
|---core (build output)
The goal is to "flatten" the "build" tree and put all the binary outputs of tests, core, etc into one folder.
I tried adding CMake's "install" command, to each of my CMakeLists.txt files (e.g. install(TARGETS core DESTINATION bin). I then then executed sudo make install after my normal build. This put all my binaries in /usr/local/bin with no errors. But when I ran tests from there, it failed to find libcore.so, even though it was sitting right there in the same folder
tests: error while loading shared libraries: libcore.so: Cannot open shared object file: No such file or directory
I read up on the LD_LIBRARY_PATH environment variable and so tried adding that folder (/usr/local/bin) into it and running. I can see I've properly altered LD_LIBRARY_PATH but it still doesn't work. "tests" still can't find libcore.so. I even tried changing the PATH environment variable as well. Same result.
In frustration, I tried brute-force copying the output binaries to a temporary subfolder (of /mysdk/build) and running tests from there. To my surprise it ran.
Then I realized why: Instead of loading the local copy of libcore.so it had loaded the one from the build output folder (as if the full path were "baked in" to the app at build time). Subsequently deleting that build-output copy of libcore.so made "tests" fail altogether as before, instead of loading the local copy. So maybe the path really was baked in.
I'm at a loss. I've read the CMake tutorial and reference. It makes this sound so easy. Aside from the obvious (What am I doing wrong?) I would appreciate if anyone could answer any of the following questions:
What is the correct way to control where my app looks for my shared libraries?
Is there a relationship between my project build structure and how my binaries must then appear when installed?
Am I even close to the right way of doing this?
Is it possible I've somehow inadvertently "baked" (into my app) full paths to my shared libraries? Is that a thing? I use all CMAKE variables in my CMakeLists files.
You can run ldd file to print the shared object dependencies for file. It will tell you where are its dependencies being read from.
You can export the environment variable LD_LIBRARY_PATH with the paths you want the linker to look for. If a dependency is not found, try adding the path where that dependency is located at to LD_LIBRARY_PATH and then run ldd again (make sure you export the variable).
Also, make sure the dependencies have the right permissions.
Updating LD_LIBRARY_PATH is an option. Another option is using RPATH. Please check the example.
https://github.com/mustafagonul/cmake-examples/blob/master/005-executable-with-shared-library/CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
# Project
project(005-executable-with-shared-library)
# Directories
set(example_BIN_DIR bin)
set(example_INC_DIR include)
set(example_LIB_DIR lib)
set(example_SRC_DIR src)
# Library files
set(library_SOURCES ${example_SRC_DIR}/library.cpp)
set(library_HEADERS ${example_INC_DIR}/library.h)
set(executable_SOURCES ${example_SRC_DIR}/main.cpp)
# Setting RPATH
# See https://cmake.org/Wiki/CMake_RPATH_handling
set(CMAKE_INSTALL_RPATH ${CMAKE_INSTALL_PREFIX}/${example_LIB_DIR})
# Add library to project
add_library(library SHARED ${library_SOURCES})
# Include directories
target_include_directories(library PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/${example_INC_DIR})
# Add executable to project
add_executable(executable ${executable_SOURCES})
# Linking
target_link_libraries(executable PRIVATE library)
# Install
install(TARGETS executable DESTINATION ${example_BIN_DIR})
install(TARGETS library DESTINATION ${example_LIB_DIR})
install(FILES ${library_HEADERS} DESTINATION ${example_INC_DIR})