--extra-lib-dirs should ostensibly function like LD_LIBRARY_PATH, but when used with stack exec I see different results:
$ stack --extra-lib-dirs=$MATLAB_PATH/bin/glnxa64 --extra-include-dirs=$MATLAB_PATH/extern/include --nix exec matlab-engine-test
/home/bebarker/workspace/haskell-matlab/.stack-work/install/x86_64-linux-nix/24c5769e9013838d87aa76fb4cdd10a09798b6904a6faa380de6fe6949e2c952/8.6.5/bin/matlab-engine-test: error while loading shared libraries: libmx.so: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=$MATLAB_PATH/bin/glnxa64 /home/bebarker/workspace/haskell-matlab/.stack-work/install/x86_64-linux-nix/24c5769e9013838d87aa76fb4cdd10a09798b6904a6faa380de6fe6949e2c952/8.6.5/bin/matlab-engine-test
matlab-engine-test: user error (engOpen)
LD_LIBRARY_PATH behaves as expected in this case (though clearly I have another unrelated issue to deal with there). Should I be looking for another option or am I using --extra-lib-dirs incorrectly?
I don't tend to use Stack, but in build systems the library path provided by command line flags is usually only passed to the linker. Observe (with main = print =<< lookupEnv "LD_LIBRARY_PATH"):
$ stack --extra-lib-dirs=/foopath/bin/glnxa64 --extra-include-dirs=/foopath/extern/include exec ./foo
Just ""
$
Compare with what you did second:
$ LD_LIBRARY_PATH=/foopath/bin/glnxa64 ./foo
Just "/foopath/bin/glnxa64"
Related
I'm fairly new to NixOS, and am trying to invoke emacs from a Haskell program using the following function:
ediff :: String -> String -> String -> IO ()
ediff testName a b = do
a' <- writeSystemTempFile (testName ++ ".expected") a
b' <- writeSystemTempFile (testName ++ ".received") b
let quote s = "\"" ++ s ++ "\""
callCommand $ "emacs --eval \'(ediff-files " ++ quote a' ++ quote b' ++ ")\'"
When I run the program that invokes this command using stack test, I get the following result (interspersed with unit test results):
/bin/sh: emacs: command not found
Exception: callCommand: emacs --eval '(ediff-files "/run/user/1000/ast1780695788709393584.expected" "/run/user/1000/ast4917054031918502651.received")'
When I run the command that failed to run above from my shell, it works flawlessly. How can I run processes from Haskell in NixOS, as though I had invoked them directly, so that they can access the same commands and configurations as my user?
Both your shell and callCommand use the PATH environment variable, so it seems like stack is changing that. It turns out that stack uses a pure nix shell by default, but you also want to access your user environment, which is 'impure'.
To quote the stack documenation
By default, stack will run the build in a pure Nix build environment (or shell), which means the build should fail if you haven't specified all the dependencies in the packages: section of the stack.yaml file, even if these dependencies are installed elsewhere on your system. This behaviour enforces a complete description of the build environment to facilitate reproducibility. To override this behaviour, add pure: false to your stack.yaml or pass the --no-nix-pure option to the command line.
Another solution is to add Emacs to nix.dependencies in stack.yaml (thanks #chepner). It has the benefit that some version of Emacs will always be available when a developer runs the tests, but that Emacs may not be the Emacs they want to use. You may be able to work around that using something like ~/.config/nixpkgs/config.nix, unless they have configured their Emacs elsewhere, like the system configuration or perhaps a home manager. I'd prefer the simple but impure $PATH solution.
I wish to write some Haskell that calls an executable as part of its work; and install this on a nixOS host. I don't want the executable to be in my PATH (and to rely on that would disrupt the beautiful dependency model of nix).
If this were, say, a Perl script, I would have a simple builder that looked for strings of a certain format, and replaced them with the executable names, based upon dependencies declared in the .nix file. But that seems somewhat harder with the cabal-based building common to haskell.
Is there a standard idiom for encoding the paths to executables at build time (including during development, as well as at install time) within Haskell code on nix?
For the sake of a concrete example, here is a trivial "script":
import System.Process ( readProcess )
main = do
stdout <- readProcess "hostname" [] ""
putStrLn $ "Hostname: " ++ stdout
I would like to be able to compile run this (in principle) without relying on hostname being in the PATH, but rather replacing hostname with the full /nix/store/-inetutils-/bin/hostname path, and thus also gaining the benefits of dependency management under nix.
This could possibly be managed by using a shell (or similar) script, built using a replacement scheme as defined above, that sets up an environment that the haskell executable expects; but still that would need some bootstrapping via the cabal.mkDerivation, and since I'm a lover of OptParse-Applicative's bash completion, I'm loathe to slow that down with another script to fire up every time I hit the tab key. But if that's what's needed, fair enough.
I did look through cabal.mkDerivation for some sort of pre-build step, but if it's there I'm not seeing it.
Thanks,
Assuming you're building the Haskell app in Nix, you can patch a configuration file via your Nix expression. For an example of how to do this, have a look at this small project.
The crux is that you can define a postConfigure hook like this:
pkgs.haskell.lib.overrideCabal yourProject (old: {
postConfigure = ''
substituteInPlace src/Configuration.hs --replace 'helloPrefix = Nothing' 'helloPrefix = Just "${pkgs.hello}"'
'';
})
What I do with my xmonad build in nix1 is refer to executable paths as things like ##compton##/bin/compton. Then I use a script like this to generate my default.nix file:
#!/usr/bin/env bash
set -eu
packages=($(grep '##[^#]*##' src/Main.hs | sed -e 's/.*##\(.*\)##.*/\1/' | sort -u))
extra_args=()
for p in "${packages[#]}"; do
extra_args+=(--extra-arguments "$p")
done
cabal2nix . "${extra_args[#]}" \
| head -n-1
echo " patchPhase = ''";
echo " substituteInPlace src/Main.hs \\"
for p in "${packages[#]}"; do
echo " --replace '##$p##' '\${$p}' \\"
done
echo " '';"
echo "}"
What it does is grep through src/Main.hs (could easily be changed to find all haskell files, or to some specific configuration module) and pick out all the tags surrounded by## like ##some-package-name##. It then does 2 things with them:
passes them to cabal2nix as extra arguments for the nix expression it generates
post-processes nix expression output from cabal2nix to add a patch phase, which replaces the ##some-package-name## tag in the Haskell source file with the actual path to the derivation.2
This generates a nix-expression like this:
{ mkDerivation, base, compton, networkmanagerapplet, notify-osd
, powerline, setxkbmap, stdenv, synapse, system-config-printer
, taffybar, udiskie, unix, X11, xmonad, xmonad-contrib
}:
mkDerivation {
pname = "xmonad-custom";
version = "0.0.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base taffybar unix X11 xmonad xmonad-contrib
];
description = "My XMonad build";
license = stdenv.lib.licenses.bsd3;
patchPhase = ''
substituteInPlace src/Main.hs \
--replace '##compton##' '${compton}' \
--replace '##networkmanagerapplet##' '${networkmanagerapplet}' \
--replace '##notify-osd##' '${notify-osd}' \
--replace '##powerline##' '${powerline}' \
--replace '##setxkbmap##' '${setxkbmap}' \
--replace '##synapse##' '${synapse}' \
--replace '##system-config-printer##' '${system-config-printer}' \
--replace '##udiskie##' '${udiskie}' \
'';
}
The net result is I can just write Haskell code and a cabal package file; I don't have to worry much about maintaining the nix package file as well, only re-running my generate-nix script if my dependencies change.
In my Haskell code I just write paths to executables as if ##the-nix-package-name## was an absolute path to a folder where that package is installed, and everything magically works.
The installed xmonad binary ends up containing hardcoded references to the absolute paths to the executables I call, which is how nix likes to work (this means it automatically knows about the dependency during garbage collection, for example). And I don't have to worry about keeping the things I called in my interactive environment's PATH, or maintaining a wrapper that sets up PATH just for this executable.
1 I have it set up as a cabal project that gets built and installed into the nix store, rather than having it dynamically recompile itself from ~/.xmonad/xmonad.hs
2 Step 2 is a little meta, since I'm using a bash script to generate nix code with an embedded bash script in it
This is not indented to be the answer but if I post this in comment section it would turn out to be ugly formatted.
Also I am not sure if this hack is the right way to do the job.
I notice that if I use nix-shell I can get full path to nix store
Assume hash is always the same, AFAIK I believe it is, you can use it to hard-coded in build recipe.
$ which bash
/run/current-system/sw/bin/bash
[wizzup# ~]
$ nix-shell -p bash
[nix-shell:~]$ which bash
/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12/bin/bash
Lastly, I doubt if you have to to any of this if you use buildInput, it should be the same path.
I'm trying to run the elm-reactor project, which is written in Haskell. It fails because it's trying to proc out to the elm command like this:
createProcess (proc "elm" $ args fileName)
My elm executable is sitting in ~/.cabal/bin, which is in my PATH.
The System.Process.proc command searches the $PATH for its command argument, but it doesn't do tilde (~) expansion, so it doesn't find elm.
System.Process.shell has the opposite problem. It does tilde expansion, but it doesn't search the $PATH, apparently.
From the source of the System.Process command, it looks like most everything rests on a foreign ccall to "runInteractiveProcess", which I assume is doing whatever $PATH searching is being done. I don't know where the source for runInteractiveProcess would be, and my C is about 15 years worth of rusty.
I can work around this issue by
a) adding the fully-expanded cabal/bin path to my PATH or
b) symlinking an elm from the working directory to its location in cabal/bin.
However, I'd like to offer a suggested fix to the elm project, to save future adopters the trouble I've gone through. Is there a System.Process call that they should be making here that I haven't tried? Or is there a different method they should be using? I suppose at worst they could getEnv for the PATH and HOME, and implement their own file search using that before calling proc - but that breaks cross-platform compatibility. Any other suggestions?
Try using shell instead of proc, i.e.:
createProcess (shell "elm")
This should invoke elm via a shell, which hopefully will interpret tildes in $PATH as desired.
Update: Here is the experiment I performed to test what shell does...
Compile the following program (I called it run-foofoo):
import System.Process
main = do
(,,_,h) <- createProcess $ shell "foofoo"
ec <- waitForProcess h
print ec
Create a new directory ~/new-bin and place the following perl script there as the file foofoo:
#!/usr/bin/perl
print "Got here and PATH is $ENV{PATH}\n";
Run: chmod a+rx ~/new-bin/foofoo
Test with:
PATH="/bin:/usr/bin:/sbin" ./run-foofoo # should fail
PATH="$HOME/new-bin:/bin:/usr/bin:/sbin" ./run-foofoo # should succeed
PATH="~/new-bin:/bin:/usr/bin:/sbin" ./run-foofoo # ???
On my OSX system, the third test reports:
Got here and PATH is ~/new-bin:/bin:/usr/bin:/sbin
ExitSuccess
I have a problem with ghci and I need an advice on how to debug it. The problem is that when I execute a function from my imported project I have a duplicate definition error and ghci exits because it can't continue:
> ghci -v0 --interactive -ignore-dot-ghci -isrc -idist/build/autogen tests/System/Console/Hawk/PreludeTests.hs -no-user-package-db -package-db /mnt/git/hawk/.cabal-sandbox/x86_64-linux-ghc-7.6.3-packages.conf.d
*System.Console.Hawk.PreludeTests> test [] "1" ""
GHCi runtime linker: fatal error: I found a duplicate definition for symbol
__stginit_stringsearchzm0zi3zi6zi5_DataziByteStringziSearch
whilst processing object file
/mnt/git/hawk/.cabal-sandbox/lib/x86_64-linux-ghc-7.6.3/stringsearch-0.3.6.5/libHSstringsearch-0.3.6.5.a
This could be caused by:
* Loading two different object files which export the same symbol
* Specifying the same object file twice on the GHCi command line
* An incorrect `package.conf' entry, causing some object to be
loaded twice.
GHCi cannot safely continue in this situation. Exiting now. Sorry.
The problem is that I can't find where this is happening. The symbol is unique in my cabal sandbox:
> for f in `find .cabal-sandbox -type f -iname "*.a"`; do nm $f | grep '__stginit_stringsearchzm0zi3zi6zi5_DataziByteStringziSearch$'; done
0000000000000000 D __stginit_stringsearchzm0zi3zi6zi5_DataziByteStringziSearch
so probably the stringsearch library is somehow loaded two times, but ghci is vague about it.
I would like to know if there is a way to debug this or, at least, to get more informations on the error before ghci kills itself. I already tried to change verbosity but I still get no informations.
Often this can occur when you are indirectly depending on two different versions of a library that both export the same symbol. This could occur, for example, if you also had a library from outside your sandbox (e.g. in the global package db) that depended on a stringsearch from there.
Additionally, to get more debug info, you should pass a flag indicating a higher verbosity.
I have a Linux binary, without sources, that works on one machine, and I'd like to make a self-contained package that would run on a different machine of the same architecture. What is a way of achieving this?
In my case, both machines have the same architecture, same Ubuntu kernel, but target machine doesn't have make and has wrong version of files under /lib and /usr
One idea I had was to use chroot and recreate a subset of the filesystem that the binary uses, possibly using strace to figure out what it needs. Is there a tool that does this already?
For posterity, here's how I figure out which files a process opens
#!/usr/bin/python
# source of trace_fileopen.py
# Runs command and prints all files that have been successfully opened with mode O_RDONLY
# example: trace_fileopen.py ls -l
import re, sys, subprocess, os
if __name__=='__main__':
strace_fn = '/tmp/strace.out'
strace_re = re.compile(r'([^(]+?)\((.*)\)\s*=\s*(\S+?)\s+(.*)$')
cmd = sys.argv[1]
nowhere = open('/dev/null','w')#
p = subprocess.Popen(['strace','-o', strace_fn]+sys.argv[1:], stdout=nowhere, stderr=nowhere)
sts = os.waitpid(p.pid, 0)[1]
output = []
for line in open(strace_fn):
# ignore lines like --- SIGCHLD (Child exited) # 0 (0) ---
if not strace_re.match(line):
continue
(function,args,returnval,msg) = strace_re.findall(line)[0]
if function=='open' and returnval!='-1':
(fname,mode)=args.split(',',1)
if mode.strip()=='O_RDONLY':
if fname.startswith('"') and fname.endswith('"') and len(fname)>=2:
fname = fname[1:-1]
output.append(fname)
prev_line = ""
for line in sorted(output):
if line==prev_line:
continue
print line
prev_line = line
Update
The problem with LD_LIBRARY_PATH solutions is that /lib is hardcoded into interpreter and takes precedence over LD_LIBRARY_PATH, so native versions will get loaded first. The interpreter is hardcoded into the binary. One approach might be to patch the interpreter and run the binary as patched_interpreter mycommandline Problem is that when mycommandline is starts with java, this doesn't work because Java sets-up LD_LIBRARY_PATH and restarts itself, which resorts to the old interpreter. A solution that worked for me was to open the binary in the text editor, find the interpreter (/lib/ld-linux-x86-64.so.2), and replace it with same-length path to the patched interpreter
As others have mentioned, static linking is one option. Except static linking with glibc gets a little more broken with every release (sorry, no reference; just my experience).
Your chroot idea is probably overkill.
The solution most commercial products use, as far as I can tell, is to make their "application" a shell script that sets LD_LIBRARY_PATH and then runs the actual executable. Something along these lines:
#!/bin/sh
here=`dirname "$0"`
export LD_LIBRARY_PATH="$here"/lib
exec "$here"/bin/my_app "$#"
Then you just dump a copy of all the relevant .so files under lib/, put your executable under bin/, put the script in ., and ship the whole tree.
(To be production-worthy, properly prepend "$here"/lib to LD_LIBRARY_PATH if it is non-empty, etc.)
[edit, to go with your update]
I think you may be confused about what is hard-coded and what is not. ld-linux-x86-64.so.2 is the dynamic linker itself; and you are correct that its path is hard-coded into the ELF header. But the other libraries are not hard-coded; they are searched for by the dynamic linker, which will honor LD_LIBRARY_PATH.
If you really need a different ld-linux.so, instead of patching the ELF header, simply run the dynamic linker itself:
/path/to/my-ld-linux.so my_program <args>
This will use your linker instead of the one listed in the ELF header.
Patching the executable itself is evil. Please consider the poor person who has to maintain your stuff after you move on... Nobody is going to expect you to have hacked the ELF header by hand. Anybody can read what a shell script is doing.
Just my $0.02.
There's CDE a bit of software designed to do exactly what you want. Here's a google tech talk about it
http://www.youtube.com/watch?v=6XdwHo1BWwY
There are almost certainly better answers, but you can find out what libraries the binary needs with the ldd command (example for the ls binary):
$ ldd /bin/ls
linux-vdso.so.1 => (0x00007ffffff18000)
librt.so.1 => /lib/librt.so.1 (0x00007f5ae565c000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00007f5ae543e000)
libacl.so.1 => /lib/libacl.so.1 (0x00007f5ae5235000)
libc.so.6 => /lib/libc.so.6 (0x00007f5ae4eb2000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f5ae4c95000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5ae588b000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f5ae4a90000)
libattr.so.1 => /lib/libattr.so.1 (0x00007f5ae488b000)
Once you have this, you could make copies and put them in the proper locations on the target machine.