How to replace paths to executables in source code with Nix that are not in PATH - haskell

I wish to write some Haskell that calls an executable as part of its work; and install this on a nixOS host. I don't want the executable to be in my PATH (and to rely on that would disrupt the beautiful dependency model of nix).
If this were, say, a Perl script, I would have a simple builder that looked for strings of a certain format, and replaced them with the executable names, based upon dependencies declared in the .nix file. But that seems somewhat harder with the cabal-based building common to haskell.
Is there a standard idiom for encoding the paths to executables at build time (including during development, as well as at install time) within Haskell code on nix?
For the sake of a concrete example, here is a trivial "script":
import System.Process ( readProcess )
main = do
stdout <- readProcess "hostname" [] ""
putStrLn $ "Hostname: " ++ stdout
I would like to be able to compile run this (in principle) without relying on hostname being in the PATH, but rather replacing hostname with the full /nix/store/-inetutils-/bin/hostname path, and thus also gaining the benefits of dependency management under nix.
This could possibly be managed by using a shell (or similar) script, built using a replacement scheme as defined above, that sets up an environment that the haskell executable expects; but still that would need some bootstrapping via the cabal.mkDerivation, and since I'm a lover of OptParse-Applicative's bash completion, I'm loathe to slow that down with another script to fire up every time I hit the tab key. But if that's what's needed, fair enough.
I did look through cabal.mkDerivation for some sort of pre-build step, but if it's there I'm not seeing it.
Thanks,

Assuming you're building the Haskell app in Nix, you can patch a configuration file via your Nix expression. For an example of how to do this, have a look at this small project.
The crux is that you can define a postConfigure hook like this:
pkgs.haskell.lib.overrideCabal yourProject (old: {
postConfigure = ''
substituteInPlace src/Configuration.hs --replace 'helloPrefix = Nothing' 'helloPrefix = Just "${pkgs.hello}"'
'';
})

What I do with my xmonad build in nix1 is refer to executable paths as things like ##compton##/bin/compton. Then I use a script like this to generate my default.nix file:
#!/usr/bin/env bash
set -eu
packages=($(grep '##[^#]*##' src/Main.hs | sed -e 's/.*##\(.*\)##.*/\1/' | sort -u))
extra_args=()
for p in "${packages[#]}"; do
extra_args+=(--extra-arguments "$p")
done
cabal2nix . "${extra_args[#]}" \
| head -n-1
echo " patchPhase = ''";
echo " substituteInPlace src/Main.hs \\"
for p in "${packages[#]}"; do
echo " --replace '##$p##' '\${$p}' \\"
done
echo " '';"
echo "}"
What it does is grep through src/Main.hs (could easily be changed to find all haskell files, or to some specific configuration module) and pick out all the tags surrounded by## like ##some-package-name##. It then does 2 things with them:
passes them to cabal2nix as extra arguments for the nix expression it generates
post-processes nix expression output from cabal2nix to add a patch phase, which replaces the ##some-package-name## tag in the Haskell source file with the actual path to the derivation.2
This generates a nix-expression like this:
{ mkDerivation, base, compton, networkmanagerapplet, notify-osd
, powerline, setxkbmap, stdenv, synapse, system-config-printer
, taffybar, udiskie, unix, X11, xmonad, xmonad-contrib
}:
mkDerivation {
pname = "xmonad-custom";
version = "0.0.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base taffybar unix X11 xmonad xmonad-contrib
];
description = "My XMonad build";
license = stdenv.lib.licenses.bsd3;
patchPhase = ''
substituteInPlace src/Main.hs \
--replace '##compton##' '${compton}' \
--replace '##networkmanagerapplet##' '${networkmanagerapplet}' \
--replace '##notify-osd##' '${notify-osd}' \
--replace '##powerline##' '${powerline}' \
--replace '##setxkbmap##' '${setxkbmap}' \
--replace '##synapse##' '${synapse}' \
--replace '##system-config-printer##' '${system-config-printer}' \
--replace '##udiskie##' '${udiskie}' \
'';
}
The net result is I can just write Haskell code and a cabal package file; I don't have to worry much about maintaining the nix package file as well, only re-running my generate-nix script if my dependencies change.
In my Haskell code I just write paths to executables as if ##the-nix-package-name## was an absolute path to a folder where that package is installed, and everything magically works.
The installed xmonad binary ends up containing hardcoded references to the absolute paths to the executables I call, which is how nix likes to work (this means it automatically knows about the dependency during garbage collection, for example). And I don't have to worry about keeping the things I called in my interactive environment's PATH, or maintaining a wrapper that sets up PATH just for this executable.
1 I have it set up as a cabal project that gets built and installed into the nix store, rather than having it dynamically recompile itself from ~/.xmonad/xmonad.hs
2 Step 2 is a little meta, since I'm using a bash script to generate nix code with an embedded bash script in it

This is not indented to be the answer but if I post this in comment section it would turn out to be ugly formatted.
Also I am not sure if this hack is the right way to do the job.
I notice that if I use nix-shell I can get full path to nix store
Assume hash is always the same, AFAIK I believe it is, you can use it to hard-coded in build recipe.
$ which bash
/run/current-system/sw/bin/bash
[wizzup# ~]
$ nix-shell -p bash
[nix-shell:~]$ which bash
/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12/bin/bash
Lastly, I doubt if you have to to any of this if you use buildInput, it should be the same path.

Related

Is there a way to define custom implicit GNU Make rules?

I'm often creating png files out of dot (graphviz format) files. The command to do so is the following:
$ dot my_graph.dot -o my_graph.png -Tpng
However, I would like to be able to have a shorter command format like $ make my_graph.dot to automatically generate my png file.
For the moment, I'm using a Makefile in which I've defined the following rule, but the recipe is only available in the directory containing the Makefile
%.eps: %.dot
dot $< -o $# -Teps
Is it possible to define custom implicit GNU Make recipes ? Which would allow the above recipe to be available system-wide
If not, what solution do you use to solve those kind of problem ?
Setup:
Fedora Linux with ZSH/Bash
You could define shell functions in your shell's startup files, e.g.
dotpng()
{
echo dot ${1%.dot}.dot -o ${1%.dot}.png -Tpng;
}
This function can be called like
dotpng my_graph.dot
or
dotpng my_graph
The code ${1%.dot}.dot strips .dot from the file name if present and appends it (again) to allow both my_graph.dot and my_graph as function argument.
Is it possible to define custom implicit GNU Make recipes ?
Not without modifying the source code of GNU Make.
If not, what solution do you use to solve those kind of problem ?
I wouldn't be a fan o modyfying the system globally, but you could do:
Create a file /usr/local/lib/make/myimplicitrules.make with the content
%.eps: %.dot
dot $< -o $# -Teps
Use include /usr/local/lib/make/myimplicitrules.make in your Makefile.
I would rather use a git submodule or similar to share common configuration between projects, rather than depending on global configuration. Depending on global environment will make your program hard to test and non-portable.
I would rather go with a shell function, something along:
mymake() {
make -f <(cat <<'EOF'
%.eps: %.dot
dot $< -o $# -Teps
EOF
) "$#"
}
mymake my_graph.dot
GNU Make lets you specify extra makefiles to read using the MAKEFILES
environment variable. Quoting from info '(make)MAKEFILES Variable':
the default goal is never taken from one of these makefiles (or any
makefile included by them) and it is not an error if the files listed
in 'MAKEFILES' are not found
if you are running 'make' without a specific makefile, a makefile
in 'MAKEFILES' can do useful things to help the built-in implicit
rules work better
As an example, with no makefile in the current directory and the
following .mk files in make's include path (e.g. via
MAKEFLAGS=--include-dir="$HOME"/.local/lib/make/) you can create
subdir gen/ and convert my_graph.dot or dot/my_graph.dot by
running:
MAKEFILES=dot.mk make gen/my_graph.png
To further save some typing it's tempting to add MAKEFILES=dot.mk
to a session environment but defining MAKEFILES in startup files
can make things completely nontransparent. For that reason I prefer
seeing MAKEFILES=… on the command line.
File: dot.mk
include common.mk
genDir ?= gen/
dotDir ?= dot/
dotFlags ?= $(if $(DEBUG),-v)
Tvariant ?= :cairo:cairo
vpath %.dot $(dotDir)
$(genDir)%.png $(genDir)%.svg $(genDir)%.eps : %.dot | $(genDir).
dot $(dotFlags) $< -o $# -T'$(patsubst .%,%,$(suffix $#))$(Tvariant)'
The included common.mk is where you'd store general definitions to
manage directory creation, diagnostics etc., e.g.
.PRECIOUS: %/. ## preempt 'unlink: ...: Is a directory'
%/. : ; $(if $(wildcard $#),,mkdir -p -- $(#D))
References:
?= = := … - info '(make)Reading Makefiles'
vpath - info '(make)Selective Search'
order-only prerequisites (e.g. | $(genDir).) - info '(make)Prerequisite Types'
.PRECIOUS - info '(make)Chained Rules'

Cmake add command line argument to binary

I create a binary myBinary via cmake/CMakeLists.txt.
I would like to "include" default options on my binary.
In other words, I want my binary to be called with myBinary --option myopt even when I just run ./myBinary
How can I do that?
CMake does not have built-in support for you you want to do.
One solution is to do as #Youka said - change the source code of your program.
Another solution that I have used sometimes is to autogenerate a script that executes an executable:
# Create startup script
MACRO(GEN_START_SCRIPT binName)
# Generate content
SET(fileContent
"#!/bin/bash\n"
"\n"
"# This startup script is auto generated - do not modify!\n"
"\n"
"${binName} -a 23 -b 34 -c 976\n"
"\n"
)
# Write to file
SET(fileName ${CMAKE_CURRENT_BINARY_DIR}/${binName}.sh)
FILE(WRITE ${fileName} ${fileContent})
ENDMACRO()
Then call the macro after defining your executable:
ADD_EXECUTABLE(myBinary file1.c file.2)
GEN_START_SCRIPT(myBinary)
You can of course add other stuff to the script, like environment variables etc.
If you're in control of the sources and you want different default behavior... change the sources!
This is in no way a build system issue (CMake or otherwise).

Read environment variable in make file

I have a environment variable set with name $MY_ENV_VARIABLE.
How do I use this variable inside my makefile to (for example) include some source files?
LOCAL_SRC_FILES = $(MY_ENV_VARIABLE)/libDEMO.so
Something like above doesn't seem to work.
Note: in my case this is needed for building with the Android NDK but I guess this applies to make in general.
Just to add some information...
The syntax to access the environment variable in make is like other variables in make...
#export the variable. e.g. in the terminal,
export MY_ENV_VARIABLE="hello world"
...
#in the makefile (replace before call)
echo $(MY_ENV_VARIABLE)
This performs the substitution before executing the commmand. If you instead, want the substitution to happen during the command execution, you need to escape the $ (For example, echo $MY_ENV_VARIABLE is incorrect and will attempt to substitute the variable M in make, and append it to Y_ENV_VARIABLE)...
#in the makefile (replace during call)
echo $$MY_ENV_VARIABLE
Make sure you exported the variable from your shell. Running:
echo $MY_ENV_VARIABLE
shows you whether it's set in your shell. But to know whether you've exported it so that subshells and other sub-commands (like make) can see it try running:
env | grep MY_ENV_VARIABLE
If it's not there, be sure to run export MY_ENV_VARIABLE before running make.
That's all you need to do: make automatically imports all environment variables as make variables when it starts up.
I just had a similar issue (under Cygwin):
Running echo $OSTYPE on the shell prints the value, but
running env | grep OSTYPE doesn't give any output.
As I can't guarantee that this variable is exported on all machines I want to run that makefile on, I used the following to get the variable from within the makefile:
OSTYPE = $(shell echo $$OSTYPE)
Which of course can also be used within a condition like the following:
ifeq ($(shell echo $$OSTYPE),cygwin)
# ...do something...
else
# ...do something else...
endif
EDIT:
Some things I found after experimenting with the info from jozxyqk's answer, all from within the makefile:
If I run #echo $$OSTYPE or #echo "$$OSTYPE" in a recipe, the variable is successfully expanded into cygwin.
However, using that in a condition like ifeq ($$OSTYPE,cygwin) or ifeq ("$$OSTYPE","cygwin") doesn't expand it.
Thus it is logical that first setting a variable like TEST = "$$OSTYPE" will lead to echo $(TEST) printing cygwin (the expansion is done by the echo call) but that doesn't work in a condition - ifeq ($(TEST),cygwin) is false.

Compiling static library for Google Native Client using SCons

I'm working on a few multi platform projects that all depend on common framework.
I want to add support for Google Native-Client (NaCl). The way I aproached the problem is first to compile the framework as static library (this is how I've been doing it on all other platforms).
I have to say that I have never used SCons before. I think I start grasping it. Starting from a build.scons from a tutorial I can get some code compiling and linking. Now I would want to skip the linking process but seems like the nacl_env was never intended to compile static libraries.
Reading the SCons help didn't help me much since the Library node is missing from the nacl_env.
I don't think I understand SCons enough to write the whole build process from scratch so I was hopping to not have to do so.
1. Am I approaching the problem correctly?
2. Any tips or sample nacl static libs, build using SCons?
Ok, what I did is way more trickery than what you probably need.
I wanted my static library to handle the initialization steps of the NaCl module, and then call some project-specific function.
I ended up turning my whole framework and the contents of the built-in libppapi_cpp.a into a single .o file, and then that into a single .a file, a static library.
I needed a single .o file, because otherwise I would run into dependency problems releated to initialization, I could not solve.
build_lib.sh (framework):
#!/bin/bash -e
SDK="/home/kalmi/ik/nacl_sdk/pepper_15"
function create_allIn_a {
TMPDIR="`mktemp -d`"
echo $TMPDIR
cp $O_FILES $TMPDIR
pushd $TMPDIR &> /dev/null
$AR x $LIBPPAPI_CPP_A
$LD -Ur * -o ALL.o
$AR rvs $OUTPUT_NAME ALL.o
$RANLIB $OUTPUT_NAME
popd &> /dev/null
}
./scons
BIN_BASE="$SDK/toolchain/linux_x86/bin"
LD="$BIN_BASE/i686-nacl-ld"
AR="$BIN_BASE/i686-nacl-ar"
RANLIB="$BIN_BASE/i686-nacl-ranlib"
LIBPPAPI_CPP_A="$SDK/toolchain/linux_x86_newlib/x86_64-nacl/lib32/libppapi_cpp.a"
O_FILES="`find $(pwd)/opt_x86_32 | grep .o$ | grep --invert-match my_main.o | tr "\n" " "`"
LIBDIR="../../../bin/lib/lib32"
mkdir -p $LIBDIR
if [ -f $LIBDIR/libweb2grid_framework.a ]; then
rm $LIBDIR/libweb2grid_framework.a
fi
OUTPUT_NAME="`readlink -m $LIBDIR/libweb2grid_framework.a`"
create_allIn_a
BIN_BASE="$SDK/toolchain/linux_x86/bin"
LD="$BIN_BASE/x86_64-nacl-ld"
AR="$BIN_BASE/x86_64-nacl-ar"
RANLIB="$BIN_BASE/x86_64-nacl-ranlib"
LIBPPAPI_CPP_A="$SDK/toolchain/linux_x86_newlib/x86_64-nacl/lib64/libppapi_cpp.a"
O_FILES="`find $(pwd)/opt_x86_64 | grep .o$ | grep --invert-match my_main.o | tr "\n" " "`"
LIBDIR="../../../bin/lib/lib64"
mkdir -p $LIBDIR
if [ -f $LIBDIR/libweb2grid_framework.a ]; then
rm $LIBDIR/libweb2grid_framework.a
fi
OUTPUT_NAME="`readlink -m $LIBDIR/libweb2grid_framework.a`"
create_allIn_a
./scons -c
The my_main.o file is excluded from the static library, because that file contains the function that is to be provided by the project that uses this framework.
The build.scons file for the framework is truly ordinary.
build.scons (for some project that uses this framework):
#! -*- python -*-
#What to compile:
sources = [ 'src/something.cpp', 'src/something_helper.cpp' ]
###############################################################x
import make_nacl_env
import nacl_utils
import os
nacl_env = make_nacl_env.NaClEnvironment(
use_c_plus_plus_libs=False,
nacl_platform=os.getenv('NACL_TARGET_PLATFORM'))
nacl_env.Append(
# Add a CPPPATH that enables the full-path #include directives, such as
# #include "examples/sine_synth/sine_synth.h"
CPPPATH=[os.path.dirname(os.path.dirname(os.path.dirname(os.getcwd())))],
LIBS=['web2grid_framework','srpc'],
LIBPATH=['../../../bin/lib/lib32','../../../bin/lib/lib64'],
LINKFLAGS=['-pthread']
)
nacl_env.AllNaClModules(sources, 'client')
Some lines worth highlighting:
use_c_plus_plus_libs=False,
LIBS=['web2grid_framework','srpc'],
LIBPATH=['../../../bin/lib/lib32','../../../bin/lib/lib64'],
LINKFLAGS=['-pthread']
I am not saying that this is a clean method, but it gets the job done.
So, there's two questions here
1. Using SCONS:
NaCl uses SCONS for it's examples, simply to help compiling of the examples easier. In reality, SCONS simply directs to the GCC/G++ compilers in the SDK build directories. (SCONS will take the input scripts, and create the final param string to send to GCC)
GCC is a common compiler, and is well documented on the net : http://gcc.gnu.org/
How you integrate NaCl compilation into your work-flow is up to you (ie you're not forced to use SCONS).
For instance, if you'd like to go to GCC directly, you can simply call :
<path to bin>/x86_64-nacl-gcc -m64 -o test.nexe main.c
For a more detailed look into how to compile NaCl modules, please read the documentation # gonacl.com on compiling which will detail how to compile with and without SCONS.
2.Compilng Static libs with GCC
Here is an example : http://www.adp-gmbh.ch/cpp/gcc/create_lib.html
~Main

Autoconf check for program and fail if not found

I'm creating a project and using GNU Autoconf tools to do the configuring and making. I've set up all my library checking and header file checking but can't seem to figure out how to check if an executable exists on the system and fail if it doesn't exist.
I've tried:
AC_CHECK_PROG(TEST,testprogram,testprogram,AC_MSG_ERROR(Cannot find testprogram.))
When I configure it runs and outputs:
Checking for testprogram... find: `testprogram. 15426 5 ': No such file or directory
but does not fail.
I found this to be the shortest approach.
AC_CHECK_PROG(FFMPEG_CHECK,ffmpeg,yes)
AS_IF([test x"$FFMPEG_CHECK" != x"yes"], [AC_MSG_ERROR([Please install ffmpeg before configuring.])])
Try this which is what I just lifted from a project of mine, it looks for something called quantlib-config in the path:
# borrowed from a check for gnome in GNU gretl: def. a check for quantlib-config
AC_DEFUN(AC_PROG_QUANTLIB, [AC_CHECK_PROG(QUANTLIB,quantlib-config,yes)])
AC_PROG_QUANTLIB
if test x"${QUANTLIB}" == x"yes" ; then
# use quantlib-config for QL settings
[.... more stuff omitted here ...]
else
AC_MSG_ERROR([Please install QuantLib before trying to build RQuantLib.])
fi
Similar to the above, but has the advantage of also being able to interact with automake by exporting the condition variable
AC_CHECK_PROG([ffmpeg],[ffmpeg],[yes],[no])
AM_CONDITIONAL([FOUND_FFMPEG], [test "x$ffmpeg" = xyes])
AM_COND_IF([FOUND_FFMPEG],,[AC_MSG_ERROR([required program 'ffmpeg' not found.])])
When using AC_CHECK_PROG, this is the most concise version that I've run across is:
AC_CHECK_PROG(BOGUS,[bogus],[bogus],[no])
test "$BOGUS" == "no" && AC_MSG_ERROR([Required program 'bogus' not found.])
When the program is missing, this output will be generated:
./configure
...cut...
checking for bogus... no
configure: error: Required program 'bogus' not found.
Or when coupled with the built-in autoconf program checks, use this instead:
AC_PROG_YACC
AC_PROG_LEX
test "$YACC" == ":" && AC_MSG_ERROR([Required program 'bison' not found.])
test "$LEX" == ":" && AC_MSG_ERROR([Required program 'flex' not found.])
Stumbled here while looking for this issue, I should note that if you want to have your program just looked in pathm a runtime test is enough:
if ! which programname >/dev/null ; then
AC_MSG_ERROR([Missing programname]
fi
This is not exactly a short approach, it's rather a general purporse approach (although when there are dozens of programs to check it might be also the shortest approach). It's taken from a project of mine (the prefix NA_ stands for “Not Autotools”).
A general purpose macro
dnl ***************************************************************************
dnl NA_REQ_PROGS(prog1, [descr1][, prog2, [descr2][, etc., [...]]])
dnl
dnl Checks whether one or more programs have been provided by the user or can
dnl be retrieved automatically. For each program `progx` an uppercase variable
dnl named `PROGX` containing the path where `progx` is located will be created.
dnl If a program is not reachable and the user has not provided any path for it
dnl an error will be generated. The program names given to this function will
dnl be advertised among the `influential environment variables` visible when
dnl launching `./configure --help`.
dnl ***************************************************************************
AC_DEFUN([NA_REQ_PROGS], [
m4_if([$#], [0], [], [
AC_ARG_VAR(m4_translit([$1], [a-z], [A-Z]), [$2])
AS_IF([test "x#S|#{]m4_translit([$1], [a-z], [A-Z])[}" = x], [
AC_PATH_PROG(m4_translit([$1], [a-z], [A-Z]), [$1])
AS_IF([test "x#S|#{]m4_translit([$1], [a-z], [A-Z])[}" = x], [
AC_MSG_ERROR([$1 utility not found])
])
])
m4_if(m4_eval([$# + 1 >> 1]), [1], [], [NA_REQ_PROGS(m4_shift2($*))])
])
])
Sample usage
NA_REQ_PROGS(
[find], [Unix find utility],
[xargs], [Unix xargs utility],
[customprogram], [Some custom program],
[etcetera], [Et cetera]
)
So that within Makefile.am you can do
$(XARGS)
or
$(CUSTOMPROGRAM)
and so on.
Features
It advertises the programs among the “influential environment variables” visible when the final user launches ./configure --help, so that an alternative path to the program can be provided
A bash variable named with the same name of the program, but upper case, containing the path where the program is located, is created
En error is thrown if any of the programs given have not been found and the user has not provided any alternative path for them
The macro can take infinite (couples of) arguments
When you should use it
When the programs to be tested are vital for compiling your project, so that the user must be able to provide an alternative path for them and an error must be thrown if at least one program is not available at all
When condition #1 applies to more than one single program, in which case there is no need to write a general purpose macro and you should just use your own customized code

Resources