Freezing all dependencies when building executables with `cabal v2-install` - haskell

I am building a Docker image in which I want to bundle multiple executables. Each executable is defined in a different package, in my case pandoc, pandoc-citeproc, and pandoc-crossref. The build should be as reproducible as reasonably possible on a Debian/Ubuntu based system.
What I'd like to do is use (something like) a cabal.project.freeze file to ensure that all subsequent builds will use the same packages.
I'm aware that I can fix the version of the executables:
cabal v2-install pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1
But this will not fix the versions of transitive dependencies, so rebuilding at different times may lead to subtly different build results. Can I somehow create and use a freeze file in this setup? Using v2-freeze seems to be of no use here:
$ cabal new-freeze pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1
cabal: 'freeze' doesn't take any extra arguments: pandoc-2.7.3
pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1

Okay, there might be a better built-in way to do this kind of thing, but here's a hacky workaround that might be suitable for you until a real cabal expert comes along.
The basic plan will be this: temporarily create a project with the three packages you care about -- just long enough to get a freeze file -- then use some simple text-editor macros to turn the freeze file into a v2-install command. So:
% cabal unpack pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1
% echo >cabal.project packages: pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1
% cabal v2-freeze
% sed "s/^constraints: /cabal v2-install pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1 --constraint '/;s/^ \+/--constraint '/;s/,\$/' \\\\/;\$s/\$/'/" cabal.project.freeze >cabal-v2-install.sh
Woof, that last one is a mouthful. It says:
# Replace the starting "constraints" stanza with the v2-install command we want to
# run. The first line of the stanza includes a constraint, so prefix it with
# --constraint and start a quote.
s/^constraints: /cabal v2-install pandoc-2.7.3 pandoc-citeproc-0.16.2 pandoc-crossref-0.3.4.1 --constraint '/
# The line we just produced doesn't start with spaces, so this only fires on the
# remaining lines. On those lines, it prefixes --constraint and starts a quote.
s/^ \+/--constraint '/
# Close the quote begun on each line, and replace cabal's line-continuation
# character (,) with a shell's line-continuation character (\). The $ and \ are
# escaped because we are inside the current shell's ""-quoted string.
s/,\$/' \\\\/
# The last line doesn't have a line-continuation character, but still needs its
# quote closed. The two occurrences of $ are escaped because we are inside the
# current shell's ""-quoted string.
\$s/\$/'/
You could also do these manually in an editor if you wanted. At the end of this process, which you can run in a temporary directory to ease cleanup afterwards, you should have a file named cabal-v2-install.sh with a command that will select the exact same versions and flags for all packages involved, including dependencies.

Related

Getting a list of `PackageDescriptions` from the local database

Is there anyway, presumably using the Cabal package, to get a list of PackageDesciptions in the current local database?
I don't know if there's a "built-in"-way of doing this, but I've used this
ghc-pkg list | grep -v "\(^/\|^$\|(\)" | xargs cabal info | grep "\(^\*\|License\)"
to extract "License" info for all installed packages.
If you use stack or cabal-sandbox - the first command needs to be replaced by either stack exec -- ghc-pkg list or cabal sandbox hc-pkg list.
the (first) regex removes the path-lines like /opt/ghc/8.0.2/lib/ghc-8.0.2/package.conf.d, empty lines and lines starting with ( - somehow the colored lines produced by ghc-pkg like ghc-8.0.2 end up with starting ( in my terminal when grepping.
"map cabal info over all packages"
[optional] extracting the package information for the package itself (starting with * and the License field.
I hope this is what you were looking for. Another way, I think, would be writing a haskell program and use Cabal as a library.

How to replace paths to executables in source code with Nix that are not in PATH

I wish to write some Haskell that calls an executable as part of its work; and install this on a nixOS host. I don't want the executable to be in my PATH (and to rely on that would disrupt the beautiful dependency model of nix).
If this were, say, a Perl script, I would have a simple builder that looked for strings of a certain format, and replaced them with the executable names, based upon dependencies declared in the .nix file. But that seems somewhat harder with the cabal-based building common to haskell.
Is there a standard idiom for encoding the paths to executables at build time (including during development, as well as at install time) within Haskell code on nix?
For the sake of a concrete example, here is a trivial "script":
import System.Process ( readProcess )
main = do
stdout <- readProcess "hostname" [] ""
putStrLn $ "Hostname: " ++ stdout
I would like to be able to compile run this (in principle) without relying on hostname being in the PATH, but rather replacing hostname with the full /nix/store/-inetutils-/bin/hostname path, and thus also gaining the benefits of dependency management under nix.
This could possibly be managed by using a shell (or similar) script, built using a replacement scheme as defined above, that sets up an environment that the haskell executable expects; but still that would need some bootstrapping via the cabal.mkDerivation, and since I'm a lover of OptParse-Applicative's bash completion, I'm loathe to slow that down with another script to fire up every time I hit the tab key. But if that's what's needed, fair enough.
I did look through cabal.mkDerivation for some sort of pre-build step, but if it's there I'm not seeing it.
Thanks,
Assuming you're building the Haskell app in Nix, you can patch a configuration file via your Nix expression. For an example of how to do this, have a look at this small project.
The crux is that you can define a postConfigure hook like this:
pkgs.haskell.lib.overrideCabal yourProject (old: {
postConfigure = ''
substituteInPlace src/Configuration.hs --replace 'helloPrefix = Nothing' 'helloPrefix = Just "${pkgs.hello}"'
'';
})
What I do with my xmonad build in nix1 is refer to executable paths as things like ##compton##/bin/compton. Then I use a script like this to generate my default.nix file:
#!/usr/bin/env bash
set -eu
packages=($(grep '##[^#]*##' src/Main.hs | sed -e 's/.*##\(.*\)##.*/\1/' | sort -u))
extra_args=()
for p in "${packages[#]}"; do
extra_args+=(--extra-arguments "$p")
done
cabal2nix . "${extra_args[#]}" \
| head -n-1
echo " patchPhase = ''";
echo " substituteInPlace src/Main.hs \\"
for p in "${packages[#]}"; do
echo " --replace '##$p##' '\${$p}' \\"
done
echo " '';"
echo "}"
What it does is grep through src/Main.hs (could easily be changed to find all haskell files, or to some specific configuration module) and pick out all the tags surrounded by## like ##some-package-name##. It then does 2 things with them:
passes them to cabal2nix as extra arguments for the nix expression it generates
post-processes nix expression output from cabal2nix to add a patch phase, which replaces the ##some-package-name## tag in the Haskell source file with the actual path to the derivation.2
This generates a nix-expression like this:
{ mkDerivation, base, compton, networkmanagerapplet, notify-osd
, powerline, setxkbmap, stdenv, synapse, system-config-printer
, taffybar, udiskie, unix, X11, xmonad, xmonad-contrib
}:
mkDerivation {
pname = "xmonad-custom";
version = "0.0.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base taffybar unix X11 xmonad xmonad-contrib
];
description = "My XMonad build";
license = stdenv.lib.licenses.bsd3;
patchPhase = ''
substituteInPlace src/Main.hs \
--replace '##compton##' '${compton}' \
--replace '##networkmanagerapplet##' '${networkmanagerapplet}' \
--replace '##notify-osd##' '${notify-osd}' \
--replace '##powerline##' '${powerline}' \
--replace '##setxkbmap##' '${setxkbmap}' \
--replace '##synapse##' '${synapse}' \
--replace '##system-config-printer##' '${system-config-printer}' \
--replace '##udiskie##' '${udiskie}' \
'';
}
The net result is I can just write Haskell code and a cabal package file; I don't have to worry much about maintaining the nix package file as well, only re-running my generate-nix script if my dependencies change.
In my Haskell code I just write paths to executables as if ##the-nix-package-name## was an absolute path to a folder where that package is installed, and everything magically works.
The installed xmonad binary ends up containing hardcoded references to the absolute paths to the executables I call, which is how nix likes to work (this means it automatically knows about the dependency during garbage collection, for example). And I don't have to worry about keeping the things I called in my interactive environment's PATH, or maintaining a wrapper that sets up PATH just for this executable.
1 I have it set up as a cabal project that gets built and installed into the nix store, rather than having it dynamically recompile itself from ~/.xmonad/xmonad.hs
2 Step 2 is a little meta, since I'm using a bash script to generate nix code with an embedded bash script in it
This is not indented to be the answer but if I post this in comment section it would turn out to be ugly formatted.
Also I am not sure if this hack is the right way to do the job.
I notice that if I use nix-shell I can get full path to nix store
Assume hash is always the same, AFAIK I believe it is, you can use it to hard-coded in build recipe.
$ which bash
/run/current-system/sw/bin/bash
[wizzup# ~]
$ nix-shell -p bash
[nix-shell:~]$ which bash
/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12/bin/bash
Lastly, I doubt if you have to to any of this if you use buildInput, it should be the same path.

scons surrounds option with double quotes

I use scons (V1.1.0) for a project that contains a build step that involves the flex tool.
The definition for the flex command in the scons default rules is:
env["LEX"] = env.Detect("flex") or "lex"
env["LEXFLAGS"] = SCons.Util.CLVar("")
env["LEXCOM"] = "$LEX $LEXFLAGS -t $SOURCES > $TARGET"
which I don't want to change.
However, since -t causes #line directives to be created in the output file that refer to the file "<stdout>", this confuses the subsequent gcov processing.
As a solution, I found that -o can be used to override the file name flex produces into the #line directives (it still produces its output on stdout due to the -t option which apparently has precedence).
To achieve that, I added this in the project's SConscript file:
env.AppendUnique(LEXFLAGS = ['-o $TARGET','-c'],delete_existing=1)
I added the -c option (which does nothing) only to show the difference between how it is treated compared to -o.
An according debug print in the SConscript file results in the following (as expected):
repr(env["LEXFLAGS"]) = ['-o $TARGET', '-c']
This results in the following command line, according to the scons log:
flex "-o build/myfile.cpp" -c -t src/myfile.ll > build/myfile.cpp
So the -c option gets into the command line as desired, but the -o option and its filename parameter has double quotes around it, that must have been created by scons when expanding the LEXFLAGS variable.
When I use this definition for LEXFLAGS instead:
env.AppendUnique(LEXFLAGS = ['--outfile=$TARGET','-c'],delete_existing=1)
the resulting command line works as desired:
flex --outfile=build/myfile.cpp -c -t src/myfile.ll > build/myfile.cpp
So one could speculate that the blank in the -o case caused the double quotes to be used, maybe in an attempt to bind the content together into one logical parameter for the command.
So while my immediate problem is solved by using --outfile, my question is still is it possible to rid of the double quotes in the -o case?
Thanks,
Andy
SCons 1.1.0 is extremely old at this point. I'd recommend trying 2.3.0. But your analysis is correct; if an option (a single option, that is) has a space in it, SCons will quote it so it stays a single option. But you don't have a single option; you really have two, '-o' and '$TARGET'. Just break it up like that and it'll work.

emacs syntax highlighting for jags / bugs

Are there packages to color-highlight jags amd bugs model files? I have ESS installed, but it doesn't seem to recognize .bug files or jags/bugs syntax out of the box.
Syntax highlighting
I'm using ESS 5.14 (from ELPA) and syntax highlighting or smart underscore works fine for me with GNU Emacs 24.1.1. If you want to highlight a given file, you can try M-x ess-jags-mode or add a hook to highlight JAGS file each time, e.g.
(add-to-list 'auto-mode-alist '("\\.jag\\'" . jags-mode))
However, that is not really needed since you can simply
(require 'ess-jags-d)
in your .emacs. There's a corresponding mode for BUGS file. This file was already included in earlier release (at least 5.13), and it comes with the corresponding auto-mode-alist (for "\\.[jJ][aA][gG]\\'" extension).
(Please note that there seems to exist subtle issue with using both JAGS and BUGS, but I can't tell more because I only use JAGS.)
Running command file
If you want to stick with Emacs for running JAGS (i.e., instead of rjags or other R interfaces to JAGS/BUGS), there's only one command to know:
As described in the ESS manual, when working on a command file, C-c C-c should create a .jmd file, and then C-c C-c'ing again should submit this command file to Emacs *shell* (in a new buffer), and call jags in batch mode. Internally, this command is binded to a 'Next Action' instruction (ess-*-next-action). For example, using the mice data that comes with JAGS sample files, you should get a mice.jmd that looks like that:
model in "mice.jag"
data in "mice.jdt"
compile, nchains(1)
parameters in "mice.in1", chain(1)
initialize
update 10000
update 10000
#
parameters to "mice.to1", chain(1)
coda \*, stem("mice")
system rm -f mice.ind
system ln -s miceindex.txt mice.ind
system rm -f mice1.out
system ln -s micechain1.txt mice1.out
exit
Local Variables:
ess-jags-chains:1
ess-jags-command:"jags"
End:
Be careful with default filenames! Here, data are assumed to be in file mice.jdt and initial values for parameters in mice.in1. You can change this in the Emacs buffer if you want, as well as modify the number of chains to use.

Bash script to copy from one line and replace another line with the copy

I am looking to write a bash script for something slightly more complicated than the usual find/replace via sed. I have a book called bash Cookbook that I have been trying to glean some inspiration from but I am not getting very far.
Basically I am trying to write a script to update the version numbers in a bunch of maven pom.xml files automatically. Here is the general setup I am looking at:
<!-- TEMPLATE:BEGIN
<version>##VERSION##</version>
-->
<version>1.0.0</version>
<!-- TEMPLATE:END -->
After running the script (with the new version number 1.0.1) I'd like the file to read this instead:
<!-- TEMPLATE:BEGIN
<version>##VERSION##</version>
-->
<version>1.0.1</version>
<!-- TEMPLATE:END -->
So this would be in the actual release pom file, with 1.0.0 being the current version (and I am trying to replace it with 1.0.1 or something). Obviously the version number will be changing so there isn't a good way to do a find/replace (since the thing you want to find is variable). I am hoping to be able to write a bash script which can
replace ##VERSION## with the actual version number
delete the current version line
write the updated version line on the line before the TEMPLATE:END (while preserving the ##VERSION## in the file - possibly do this by writing template out to a temp file, doing replacement, then back in?)
I can sort of do some of this (writing out to a new file, doing replacement) using an ant script a la
<replace file="pom.xml">
<replacefilter
token="##VERSION##"
value="${version}"/>
</replace>
But I am not sure what the best ways to a.) delete the line with the old version or b.) tell it to copy the new line in the correct place are. Anyone know how to do this or have any advice?
Assuming the new version number is in a shell variable $VERSION, then you should be able to use:
sed -e '/<!-- TEMPLATE:BEGIN/,/<!-- TEMPLATE:END -->/{
s/<version>[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*<\/version>/<version>'$VERSION'<\/version>/
}'
Note that this ignores the template version line with ##VERSION##, but only matches a three-part version number that appears between the lines containing TEMPLATE:BEGIN and TEMPLATE:END, leaving everything else (including other lines containing a <version>...</version> element) alone.
You can decide how to do file overwriting (maybe your version of sed is from GNU and it does that automatically on request with the -i option), etc. You might also be able to use more powerful regular expression notations that lead to more compact matches. However, that should work on most versions of sed without change.
The steps you outlined (1-3) read as if you do not actually care to perform the replacement in accordance to the templated rules defined within the comments.
As such, here is some code that behaves verbosely as you outlined:
#!/bin/bash
file=$1
newversion=$2
sed -i $file -e "s|<version>\([^#]*\)</version>|<version>$newversion</version>|"
Run it:
chmod +x yourscript.sh
./yourscript.sh filetoupdate.xml 1.0.1
use 5.010;
use strictures;
use Perl::Version qw();
use XML::LibXML qw();
my $dom = XML::LibXML->load_xml(location => 'pox.xml');
for my $node ($dom->findnodes('//version')) {
my $version = Perl::Version->new($node->textContent);
$version->inc_subversion;
$version->stringify;
$node->removeChildNodes;
$node->appendText($version);
};
say $dom->toString;

Resources