I have perl installed on a cluster drive at /clusterhome/myperl, and the same /clusterhome directory mounted on a workstation computer at /home/chris/cluster
Running perl obviously works fine from the cluster, but when I run /home/chris/cluster/myperl/bin/perl from my workstation, it can't find any modules. The #INC is still set to
/clusterhome/myperl/lib/site_perl/5.16.3/x86_64-linux
/clusterhome/myperl/lib/site_perl/5.16.3
/clusterhome/myperl/lib/5.16.3/x86_64-linux
/clusterhome/myperl/lib/5.16.3
This happens even with the following environment variable values prepended on the workstation:
PATH /home/chris/cluster/myperl/bin
PERL5LIB /home/chris/cluster/myperl/lib
LD_LIBRARY_PATH /home/chris/cluster/myperl/lib
MANPATH /home/chris/cluster/myperl/man
Is there a way I can get this perl to work on both the cluster and the workstation? I reinstall it often (nightly), so if extra make flags are required, it's totally fine.
The exact installation location (where to look at for module inclusion) is compiled into the binaries of perl. There are other uses for the installation directory name (for example, when compiling new modules, a bunch of compilation options are provided from these compiled-in strings).
So, you have the following options:
you make sure that the files are available on every computer in the directory where they were designed to be (symlinks: ln -s, bind mounting: mount -o bind, or mounting there upfront),
you compile a new perl for every new location.
You may also disregard this compiled-in directory, and specify the directories to be used every time you want to use perl via some command-line or environment variable. For #INC, you can use command-line option -Idirectory.
Related
For a set of programs written in most languages (C for instance) a script can normally run those programs without any sort of interference between dynamic link libraries and with no special hand holding so long as they are all found on PATH. That is, the following will work:
#!/bin/bash
prog1
prog2
prog3
However, if these three programs are written in Python and they import conflicting package versions then to run each one successfully it must either be installed into a virtualenv or each must have a separate site-packages directory which is referenced by PYTHONPATH. Either way they need a set up and possibly a tear down before running. That is, for virtualenv:
#!/bin/bash
source $PROG1_ROOT/bin/activate
prog1
deactivate
source $PROG2_ROOT/bin/activate
prog2
deactivate
source $PROG3_ROOT/bin/activate
prog3
deactivate
and for separate site-packages:
#!/bin/bash
export PYTHONPATH=$PROG1_ROOT/lib/python3.6/site-packages
prog1
export PYTHONPATH=$PROG2_ROOT/lib/python3.6/site-packages
prog2
export PYTHONPATH=$PROG3_ROOT/lib/python3.6/site-packages
prog3
This problem results because
import pkg_resources
(at least through Python3.6) cannot reliably import the proper versions when multiple versions of a package share the same site-package directory, even if __requires__ precedes it listing all the version restrictions.
It occurs to me that if PYTHONPATH, or some equivalent, could be specified relative to the program instead of the $PWD, and some consistency in directory layout was observed, then it would only have to be set once. That is, if prog1 is in $PROG1_ROOT/bin and its libraries are in $PROG1_ROOT/lib/python3.6/site-packages, then setting PYTHONPATH to "../lib/python3.6/site-packages" would work not only for prog1, but also for prog2, prog3, and for as many more as are needed through progN.
However, PYTHONPATH is normally provided as an absolute path, and relative paths are I believe with respect to $PWD, not to the python program (prog1). Is there some other Python path variable which has the desired property? Failing that, is there some type of file which could be dropped into $PROG1_ROOT/bin which would be normally picked up by a python program when it starts and which could direct it to use $PROG1_ROOT/lib/python3.6/site-packages? It would be OK to have either the relative or absolute path in that file, although the former would still be preferred because then one could move the entire PROG1_ROOT directory tree to another location in the file system without having to rewrite this special file. I really want to avoid solutions which would require modifying prog1 etc. themselves (ie, prog1 in the example).
Thanks.
EDITED:
I wrote this:
https://sourceforge.net/projects/python-devirtualizer/
to implement some of these ideas. At this point it is Linux (or at least POSIX) specific. It slightly modifies python scripts in a package's "bin" directory by changing the first line, and it "wraps" everything in that directory with a replacement native binary which injects a custom PYTHONPATH into the true target's environment. That binary looks up its location using a function from libSDL2 and then specifies the PYTHONPATH relative to that. So far it has worked pretty well, and the "programs" in installed python packages (the "bin" directory's contents) are run based on PATH just like any other program, no futzing about with PYTHONPATH in the shell.
Making search paths relative to the executable is a Very Bad Idea (TM). Move the executable or libraries around, all hell breaks loose. Some enterprising miscreant might notice the path settings and place a script just right to get their own doctored libraries (or just flawed old versions) to be used. And so on.
Clean up the misbehaving scripts. Chances are that by using old versions they are vulnerable to by now fixed security boo-boos, or other misbehaviours. Or find a way to load the stuff in the script itself.
So I am working on a project that is intended to run on a remote server. I develop the program on a local pc, compile it, then upload it to the remote server. Both the local pc and the remote server are run on CentOS 7.7.
The program is developed using the CLion IDE, configured with CMake. The program depends a few shared libraries, which are supposed to link to the executable according to what I wrote in CMake. At my local PC, I can compile and run the program perfectly. However, after I scp the whole directory of the project to the remote server, the executable fails to run. It cannot find any of the ".so" files, according to what ldd says.
This is my CMakeList.txt, with every path being relative path, instead of absolute path.
cmake_minimum_required(VERSION 3.15)
project(YS_Test)
set(CMAKE_CXX_STANDARD 11)
set(SOURCE_PATH_ src)
file(GLOB SOURCE_FILES_ ${SOURCE_PATH_}/*.*)
set(PROJECT_LIBS_ libTapQuoteAPI.so libTapTradeAPI.so libTapDataCollectAPI.so)
include_directories(api/include)
link_directories(api/lib/linux)
add_executable(YS_Test ${SOURCE_FILES_})
target_link_libraries(YS_Test ${PROJECT_LIBS_})
Please do not tell me to set LD_LIBRARY_PATH to fix my issue. The program worked fine on my local pc without LD_LIBRARY_PATH, so I expect it to run on the remote server without LD_LIBRARY_PATH. I would like to know what is really going on here, instead of a work around. Thanks!
If I understand your problem correctly, you want to ship your compiled YS_Test program along with some dependencies and have it run on a remote server. By default an executable will only look in the directories configured in /etc/ld.so, which will not include the deploy path.
Note: Typically you do not deploy your entire build directory but only the compiled artifacts and dependencies. For this answer I will assume you deploy the binary and its dependencies to the same directory.
You have two options:
Require users of your program to set LD_LIBRARY_PATH, either by themselves or by a wrapper script. This variable will instruct the dynamic linker to look in the specified directories as well. Even if you do not like this solution, it is by far the most common approach.
Add -Wl,-rpath='$ORIGIN' to your linker options. This will add a DT_RUNPATH attribute to the executable's dynamic section. As you are using CMake you can also set this using BUILD_RPATH and/or INSTALL_RPATH target properties.
The ld.so manpage describes this attribute as follows:
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
...
Using the directories specified in the DT_RUNPATH dynamic section
attribute of the binary if present.
The $ORIGIN part expands to the directory containing the program or shared
object.
If you really insist on shipping your build directory (eg during development), you can take a look at the CMake BUILD_RPATH_USE_ORIGIN property (and its usual global counterpart CMAKE_BUILD_RPATH_USE_ORIGIN), this will embed relative paths into binaries instead of absolute paths.
As you don't want a workaround (#Botje has given you two already), I will try an explanation instead. In your development machine, if you use this command:
ldd YS_Test
You will see all the shared libraries used by your program, with their corresponding paths. The libTapQuoteAPI.so libTapTradeAPI.so libTapDataCollectAPI.so are found at your 'api/lib/linux' directory, but resolved with full absolute paths. If you do the same at your server, some shared objects can't be resolved because they aren't at the same location.
If you use one of these commands (not sure which are available in Centos):
chrpath --list YS_Test
or
patchelf --print-rpath YS_Test
You will see the RPATH or RUNPATH tags embedded in your program. This is the path used by the Linux linker to locate dependencies that are outside the standard ld locations. You may find extended explanations on Internet about this, like this one or the Wikipedia article.
Breaking my promise, I give you a third workaround: use patchelf or chrpath at your server after scp to change the embedded RPATH tag, pointing it relative to $ORIGIN (which represents the program location).
I’m not root for the linux server,
so I choose to install softwares in my $HOME/local/bin, I already added the $HOME/local/bin directory to the PATH environment variable, wrote in my .bashrc.
Some softwares install this way like:
tar xvzf ncurses-5.9.tar.gz
cd ncurses-5.9
./configure --prefix=$HOME/local
make
make install
cd ..
So it will directly install in my $HOME/local/bin.
But for some softwares, after download like sbt-1.2.1.zip (based on java), and decompression, shows just a file fold sbt, it contains three foldsbin conf lib, and in its bin, contains one executable file named sbt and java9-rt-export.jar sbt-launch-lib.bash sbt-launch.jar sbt.bat.
Here I wonder:
I should just soft link this executable sbt file path under my $HOME/local/bin, then source my .bashrc?
Or, after decompression, add one line in my .bashrc export PATH="downloadpath/sbt/bin:$PATH"?
Since just one executable downloadpath/sbt/bin, so I'm not sure it is right to add whole bin fold path, if software's bin fold contains executable files (one or many), I think this situation is more convenient for just add it's bin in .bashrc, but even so, I'm not sure its right?
I'm not familiar with installation software, now I usually know way
but not why. Here I shows two ways (more ways not be showed here) to
install, executable file always be written in bin or src? But some
softwares no bin just src but no executable files in it...
Slurm also can use modules to install software, conda also other way, but I want to
confirm these traditional ways I mentioned (that two) still can be
used on slurm or conda?
However, any suggestion even one aspect's reminding will be grateful!
For precompiled software, or, in general, software that does not offer configure scripts or (C)make files, it is ofter better to leave them in their target directory and adapt the *PATH (PATH to binaries, but also LD_LIBRARY_PATH, LIBRARY_PATH to libraries and CPATH to include files and MANPATH to the man page) environment variables.
The reason is that the software might be configured to read files with hardcoded paths, relative to the position of the executable, such as libraries, etc.
In your case, you might also need to setup the CLASSPATH env variable to the directory with the jar files.
To ease software installation, you can use tools such as easybuild that can help, and even create user modules just like the system module installed by the system administrators.
There is something wrong in my opinion with your setup. If you don`t have root account on your server, is not better to test what you have to test, in a more safe environment - for example a vm/container on your developement machine ?
However, in your situation maybe it can be better to start sbt by using a separate bash script than modifying your .bashrc
I am using Autotools for my current project. I'm using Ubuntu and Linux mint. With Autotools I can tell it to check a users's system to check for any required libraries my project needs in order to function properly. Now I would like to check if a user's system has enabled core dumps and if not, then execute the command ulimit -c unlimited to enable core dumps. How and where do I specify this?
Also, once the user has executed the make command to compile the source code, they execute sudo make install in order to move the binaries at /usr/local/bin/MYPROJECT. I want to add the location of my project's binaries into the path environmental variable, so that the user can execute any of the binaries in my project from a terminal without the need of typing the full path. How and where do I specify this in Autotools?
I'm thinking this is something I would add in the configure.ac file, but I haven't found any examples on how I can do this. Any help would be appreciated.
It sounds as if you basically misunderstand what installation of a software
package on Linux is about.
The job of autotools is to build a portable installation package of your
software. When I install your package, it does not become your decision
whether programs that crash will generate core dumps on my computer
when I run them. It does not become your decision what PATH I use to
invoke programs by unqualified name. These are my decisions or defaults
that I have accepted from my OS distribution.
If you execute ulimit -c unlimited, the command will in any case
only apply to the shell in which it is invoked. It doesn't
reconfigure the host system (!).
If you would like users to be able to invoke your program by unqualified
name, the normal procedure is make your package install it by default in the place,
/usr/local/bin, that unix-like OSes traditionally add to a
user's default PATH for finding locally installed programs. That is
where autotools will configure it to be installed, by default. Change it
only if you don't want your program to be in the user's default PATH.
And in any case, a user can decide where your software is installed by
passing --prefix=/path/of/my/choice to the ./configure command. Unless
you have some unavoidable reason not to, make your package installation
use the defaults that everybody expects and leave it up to the installing user
to change them.
Bottom line: You are asking how to do installation actions with autotools that
are not meant to be done with autotools, because they are not meant to be
done by package installations.
Some Perl modules, such as DBI, need to be downloaded, compiled and installed.
I'm connecting to a remote production testing computer, for which I have only my local user password (no root, for obvious reasons). I've used wget to download some external modules that I need, such as DBI, and unpacked these resulting in directories like ~/modules/DBI-<version>.
Normally, when compiling something for Linux, you run configure to pre-configure everything before installation; and one of its switches is --prefix=<some_dir>, which controls where the compiled executable and all compiled dependencies will ultimately end up.
But for Perl modules, you don't run configure, so my first question is:
Can I control where the compiled modules (e.g. DBI.pm) go when I run make? If so, how?
Failing that, I at least need to update #INC, so I can refer to the module; so my second question is:
How can I find out where the compiled modules went when I ran make?
I can't issue make install after compiling, and moreover, I've been asked not to. (I've been asked to design the script so that it doesn't rely on external modules being in the standard system path.)
perl modules should either be installed with the distributions's system, like you did with gentoo or pkg_add on BSD, etc. or by using CPAN. Don't do what you're doing, that is going to confuse you & the system.
perl -MCPAN -e "install DBI"
You can use local::lib to install Perl modules in a custom directory. Modules so installed can be used from Perl scripts:
use local::lib '/path/to/custom/directory'; # Custom modules can be `use`d from hereon
cpanm uses local::lib internally when you use the -l or -L flag. To install a module in the current directory:
cpanm -l. DBI
The installation directory is set when the makefile for the module is built. Each module comes with a Makefile.PL which must be run to build the makefile, taking into account the current Perl configuration. Makefile.PL has the option PREFIX that says where the build is going to be installed, so after unpacking the module's distribution and cding to the unpacked directory you can say
perl Makefile.PL PREFIX=/module/directory/path
make test
make install
This process is described in the Perl documentation - read perldoc perlmodinstall. You could go into the CPAN shell and use the 'o' (lower-case) oprion that allows you to change the options passed to makefile.PL, but I think the manual build/test/install is more straightforward and gives you more control over the process.
Remember to add
use lib qw(/module/directory/path);
to the start of your program to make sure Perl searches the new directory for modules.