Dear StackOverflow Haskellers:
I tried asking this same question in reddit r/haskell but sadly I got no answers at all. I hope it does better here.
Maybe the answer is that llvm-hs isn't used that much. If you have experience using llvm with haskell, but don't use llvm-hs, I'd love to see how you use it.
I'm trying to run basic llvm-hs ORC JIT functionality, which has taken me to try and run the orc example of llvm-hs-examples. So far I've been unsuccessful in multiple ways (out of the four examples only the first one, basic, runs):
Trying to run examples with the shell.nix in https://github.com/llvm-hs/llvm-hs:
$ git clone https://github.com/llvm-hs/llvm-hs.git
$ git clone https://github.com/llvm-hs/llvm-hs-examples.git
$ cd llvm-hs
$ nix-shell shell.nix
$ cd ../llvm-hs-examples
$ cabal new-build
$ cabal run orc
which produces:
$ cabal run orc
Up to date
; ModuleID = 'basic'
source_filename = "<string>"
define i32 #add() {
entry:
ret i32 42
}
JITSymbolError ""
Eager JIT Result:
()
doing a default.nix a la haskell.nix and running with nix-build
llvm-hs-examples/default.nix:
let
examplesOverlays = [ (self: super: {
llvm-config = self.llvm_9;
}) ];
in
{ # Fetch the latest haskell.nix and import its default.nix
haskellNix ? import (builtins.fetchTarball "https://github.com/input-output-hk/haskell.nix/archive/ef6ca0f431fe3830c25cb2d185367245c1cce894.tar.gz") {}
# haskellNix ? import (builtins.fetchTarball "https://github.com/input-output-hk/haskell.nix/archive/c88f9eccc975b21ae1e6a6b8057a712b91e374f2.tar.gz") {}
# haskellNix ? import (builtins.fetchTarball "https://github.com/input-output-hk/haskell.nix/archive/master.tar.gz") {}
# haskell.nix provides access to the nixpkgs pins which are used by our CI,
# hence you will be more likely to get cache hits when using these.
# But you can also just use your own, e.g. '<nixpkgs>'.
, nixpkgsSrc ? haskellNix.sources.nixpkgs-2003
# haskell.nix provides some arguments to be passed to nixpkgs, including some
# patches and also the haskell.nix functionality itself as an overlay.
, nixpkgsArgs ? haskellNix.nixpkgsArgs
# import nixpkgs with overlays
, pkgs ? (import nixpkgsSrc (nixpkgsArgs // { overlays = nixpkgsArgs.overlays ++ examplesOverlays;}))
# , pkgs ? import nixpkgsSrc nixpkgsArgs
}: pkgs.haskell-nix.project {
# 'cleanGit' cleans a source directory based on the files known by git
src = pkgs.haskell-nix.haskellLib.cleanGit {
name = "examples";
src = ./.;
};
# For `cabal.project` based projects specify the GHC version to use.
# compiler-nix-name = "ghc884"; # Not used for `stack.yaml` based projects.
}
in llvm-hs-examples's directory ran:
$ nix-build -A examples.components.exes.orc
$ ./result/bin/orc
; ModuleID = 'basic'
source_filename = "<string>"
define i32 #add() {
entry:
ret i32 42
}
JITSymbolError ""
Eager JIT Result:
()
which is the same output as before.
I believe the last (2) approach uses stack to build, but also tried manually with stack:
a) first failed approach:
In a nix-shell with llvm-config (I used the one in llvm-hs and the one I created with haskell.nix, but same result):
$ llvm-config --version
9.0.1
$ stack build examples:orc
No packages found in snapshot which provide a "llvm-config" executable, which is a build-tool dependency of llvm-hs
llvm-hs > configure
llvm-hs > [1 of 2] Compiling Main ( /run/user/1000/stack-2e6b46f4d38b8260/llvm-hs-9.0.1/Setup.hs, /run/user/1000/stack-2e6b46f4d38b8260/llvm-hs-9.0.1/.stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/setup/Main.o )
llvm-hs > [2 of 2] Compiling StackSetupShim ( /home/hhefesto/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /run/user/1000/stack-2e6b46f4d38b8260/llvm-hs-9.0.1/.stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/setup/StackSetupShim.o )
llvm-hs > Linking /run/user/1000/stack-2e6b46f4d38b8260/llvm-hs-9.0.1/.stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/setup/setup ...
llvm-hs > setup: The program 'llvm-config' version ==9.0.* is required but it could not
llvm-hs > be found.
llvm-hs >
b) build-successful stack approach I also tried with stack's nix integration explicitly specifying llvm-config as part of buildInputs:
stack.yaml
# resolver: nightly-2020-01-30
resolver: lts-14.0
packages:
- '.'
extra-deps:
- llvm-hs-9.0.1
- llvm-hs-pure-9.0.0
- llvm-hs-pretty-0.9.0.0
flags:
llvm-hs:
shared-llvm: true
nix:
enable: true
shell-file: stackShell.nix
stackShell.nix:
let
examplesOverlays = [ (self: super: {
llvm-config = self.llvm_9;
}) ];
in
{ haskellNix ? import (builtins.fetchTarball "https://github.com/input-output-hk/haskell.nix/archive/ef6ca0f431fe3830c25cb2d185367245c1cce894.tar.gz") {}
, nixpkgsSrc ? haskellNix.sources.nixpkgs-2003
, nixpkgsArgs ? haskellNix.nixpkgsArgs
, pkgs ? (import nixpkgsSrc (nixpkgsArgs // { overlays = nixpkgsArgs.overlays ++ examplesOverlays;}))
}:
with pkgs;
haskell.lib.buildStackProject {
name = "llvm-hs";
buildInputs = [ llvm-config
];
inherit ghc;
}
Which was able to build successfully, but same error:
$ stack build examples:orc ghc-shell-for-examples
examples> configure (exe)
Configuring examples-1.0.0.0...
examples> build (exe)
Preprocessing executable 'orc' for examples-1.0.0.0..
Building executable 'orc' for examples-1.0.0.0..
examples> copy/register
Installing executable orc in /home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin
Installing executable basic in /home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin
Installing executable arith in /home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin
Installing executable irbuilder in /home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin
$ /home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin/orc ghc-shell-for-examples
; ModuleID = 'basic'
source_filename = "<string>"
define i32 #add() {
entry:
ret i32 42
}
JITSymbolError ""
Eager JIT Result:
()
A weird thing worth noting is that stack was unable to find what it had just built with stack exec examples:orc:
$ stack exec examples:orc
Executable named examples:orc not found on path: ["/home/hhefesto/src/llvm-hs-examples/.stack-work/install/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin","/home/hhefesto/.stack/snapshots/x86_64-linux-nix/2bab12248a943811dbc5aa23a88b887ce7aef1939551d21b69d96e3f64bfcbd7/8.6.5/bin","/home/hhefesto/.stack/compiler-tools/x86_64-linux-nix/ghc-8.6.5/bin","/nix/store/9wvsbqr57k9n6d8vv6b10d04j51f9ims-ghc-8.6.5/bin","/nix/store/4xb9z8vvk3fk2ciwqh53hzp72d0hx1da-bash-interactive-4.4-p23/bin","/nix/store/9wvsbqr57k9n6d8vv6b10d04j51f9ims-ghc-8.6.5/bin","/nix/store/m6h7zh8w6s52clnyskffj5lbkakqgywn-gcc-wrapper-9.2.0/bin","/nix/store/b3zsk4ihlpiimv3vff86bb5bxghgdzb9-gcc-9.2.0/bin","/nix/store/0k65d30z9xsixil10yw3bwajbdk4yskv-glibc-2.30-bin/bin","/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin","/nix/store/n48b8n251dwwb04q7f3fwxdmirsakllz-binutils-wrapper-2.31.1/bin","/nix/store/hrkc2sf2883l16d5yq3zg0y339kfw4xv-binutils-2.31.1/bin","/nix/store/0k65d30z9xsixil10yw3bwajbdk4yskv-glibc-2.30-bin/bin","/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin","/nix/store/6dacwd7ldb2jazc218d11v2w2g55hba8-pkg-config-0.29.2/bin","/nix/store/lb61dshvvqy1rgjhhlzaiiv2fv157lr5-stack-2.1.3.1/bin","/nix/store/71n1xcigc00w3z7yc836jqcx9cb2dys8-patchelf-0.9/bin","/nix/store/xhhkr936b9q5sz88jp4l29wljbbcg39k-ncurses-6.1-20190112/bin","/nix/store/khqyxflp8wbq038wdyv5sr8sjsfwlr72-llvm-9.0.1/bin","/nix/store/84g84bg47xxg01ba3nv0h418v5v3969n-ncurses-6.1-20190112-dev/bin","/nix/store/xhhkr936b9q5sz88jp4l29wljbbcg39k-ncurses-6.1-20190112/bin","/nix/store/x0jla3hpxrwz76hy9yckg1iyc9hns81k-coreutils-8.31/bin","/nix/store/97vambzyvpvrd9wgrrw7i7svi0s8vny5-findutils-4.7.0/bin","/nix/store/dqq1bvpi3g0h4v05111b3i0ymqj4v5x1-diffutils-3.7/bin","/nix/store/p34p7ysy84579lndk7rbrz6zsfr03y71-gnused-4.8/bin","/nix/store/b0vjq4r4sp9z4l2gbkc5dyyw5qfgyi3r-gnugrep-3.4/bin","/nix/store/c8balm59sxfkw9ik1fqbkadsvjqhmbx4-gawk-5.0.1/bin","/nix/store/g7dr83wnkx4gxa5ykcljc5jg04416z60-gnutar-1.32/bin","/nix/store/kkvgr3avpp7yd5hzmc4syh43jqj03sgb-gzip-1.10/bin","/nix/store/rw96psqzgyqrcd12qr6ivk9yiskjm3ab-bzip2-1.0.6.0.1-bin/bin","/nix/store/dp6y0n9cba79wwc54n1brg7xbjsq5hka-gnumake-4.2.1/bin","/nix/store/hrpvwkjz04s9i4nmli843hyw9z4pwhww-bash-4.4-p23/bin","/nix/store/xac1zfclx1xxgcd84vqb6hy3apl171n8-patch-2.7.6/bin","/nix/store/mm0w8jc58rn01c4kz2n9jvwd6bibcihs-xz-5.2.4-bin/bin"]
Other things worth noting: I tried everything stack related with two resolvers: lts-14.0 which was already there and nightly-2020-01-30 which was recently added in a commit to llvm-hs, but same result.
Also tried changing llvm-hs version restrictions to include the latest 9.0.1 in llvm-hs-examples' cabal file, but same result.
I'm on NixOS channel 20.03
If you want me to share or try something else, please let me know, and thank you for your help.
Lastly, thank you very much for your time!
I am trying to make a python37 development environment containing the openai pypi package using nix.
This question was originally on reddit, but I couldn't find an answer and the activity in the thread was very low.
using the tipps i got there and the language frameworks documentation i managed to come up with the following expressions:
default.nix:
with import<nixpkgs>{};
( let
openai = pkgs.callPackage ./release.nix {
inherit pkgs;
buildPythonPackage = pkgs.python37Packages.buildPythonPackage;
};
in pkgs.python37.buildEnv.override rec {
extraLibs = [ pkgs.python37Packages.requests openai ];
}
).env
release.nix
{ pkgs, buildPythonPackage }:
buildPythonPackage rec{
pname="openai";
version="0.2.6";
src=fetchTarball{
url="https://files.pythonhosted.org/packages/59/2d/b3bc2aa663b2c376f073fd141e128ecfb47f3aff95ccee284a74d85a1ef8/openai-0.2.6.tar.gz";
sha256="0cplrzfw3i6yxcd35ijfjkx9jbcvkvzn5jn5b8s657a8myhm6kav";
};
propagateBuildInputs = [ pkgs.python37Packages.requests ];
doCheck=false;
meta = { # only for testing the env right now
homepage="...";
description="...";
license = "...";
maintainers= [];
};
}
however, this leaves me with the (i guess its pip-)error
Processing ./openai-0.2.6-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement requests>=2.20; python_version >= "3.0" (from openai==0.2.6) (from versions: none)
ERROR: No matching distribution found for requests>=2.20; python_version >= "3.0" (from openai==0.2.6)
builder for '/nix/store/ncnga4fcxl15xyvv3f325f9g0q45mqvr-python3.7-openai-0.2.6.drv' failed with exit code 1
which surprises me, because propagateBuildInputs = [ pkgs.python37Packages.requests ]; clearly states that the package requests (version 2.22.0 in nixpkgs) should be present at build time.
what am I misunderstanding about the buildPythonPackage function that prevents this from working?
Changing progagatedBuildInputs to another name like progagateBuildInputs causes it to be ignored, so any dependencies it contains (i.e. requests) are not found. For example:
yubico-client/default.nix
propagateBuildInputs = [ requests ];
ERROR: Could not find a version that satisfies the requirement requests<3.0,>=2.7 (from yubico-client==1.13.0) (from versions: none)
ERROR: No matching distribution found for requests<3.0,>=2.7 (from yubico-client==1.13.0)
propagatedBuildInputs = [ requests ];
$ nix-build -I nixpkgs=~/git/nixpkgs '<nixpkgs>' -k -A python37Packages.yubico-client
/nix/store/0yjz8smgmjr0006nmka6wliy01z8av7m-python3.7-yubico-client-1.13.0
I have a problem which boils down to this simplified example. Say I have the following shell.nix file:
let
base = import <nixpkgs> {};
pinnedVersion = base.pkgs.lib.importJSON ./nixpkgs.json;
nixpkgs = base.pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs";
inherit (pinnedVersion) rev sha256;
};
pkgs = import nixpkgs { };
p = ({ mkDerivation, base }:
mkDerivation {
pname = "foo";
version = "0.0.0.1";
src = ./.;
isLibrary = true;
license = "GPL";
isExecutable = true;
libraryHaskellDepends = [ base ];
libraryToolDepends = [ ];
executableHaskellDepends = [ base ];
testHaskellDepends = [ base ];
});
in
pkgs.haskellPackages.callPackage p {}
this is just a dummy derivation. I fire off 'nix-shell' and inside the shell:
$ ghc-pkg list
/nix/store/f520j7wfgp9h2mmq6rk36r6vgsb279sf-ghc-8.8.3/lib/ghc-8.8.3/package.conf.d
Cabal-3.0.1.0
...
So version 3.0.1.0 of the Cabal library is in the ghc package set.. ok but the thing is, the version of Nixpkgs I am using above (i.e. the one pinned in the following JSON) ...
{
"url": "https://github.com/nixos/nixpkgs.git",
"rev": "6cf0ed0c04fb3aa7f81b46d0c90bae6457964b3f",
"date": "2020-07-02T07:24:47+02:00",
"path": "/nix/store/gkilcm4l6qsp31lapirjxd1mn1dnn100-nixpkgs",
"sha256": "0sp8108j70f3kz3r8hagiyq21sjwr46n0dqg16gb27k7761bcvpg",
"fetchSubmodules": false,
"deepClone": false,
"leaveDotGit": false
}
... only has Cabal library versions 3.2.0.0, 2.4.1.0 and 2.2.0.1 available. See below:
$ nix repl
Welcome to Nix version 2.3.1. Type :? for help.
nix-repl> base = import <nixpkgs> {}
nix-repl> pinnedVersion = base.pkgs.lib.importJSON ./nixpkgs.json
nix-repl> nixpkgs = base.pkgs.fetchFromGitHub { owner = "NixOS"; repo = "nixpkgs"; inherit (pinnedVersion) rev sha256; }
nix-repl> pkgs=import nixpkgs { }
nix-repl> pkgs.haskellPackages.Cabal *<tab> <tab>*
pkgs.haskellPackages.Cabal pkgs.haskellPackages.Cabal_2_4_1_0
pkgs.haskellPackages.Cabal-ide-backend pkgs.haskellPackages.Cabal_3_2_0_0
pkgs.haskellPackages.CabalSearch
pkgs.haskellPackages.Cabal_2_2_0_1
nix-repl> pkgs.haskellPackages.Cabal
pkgs.haskellPackages.Cabal pkgs.haskellPackages.Cabal_2_4_1_0
pkgs.haskellPackages.Cabal-ide-backend pkgs.haskellPackages.Cabal_3_2_0_0
pkgs.haskellPackages.CabalSearch
pkgs.haskellPackages.Cabal_2_2_0_1
nix-repl> pkgs.haskell.packages.ghc883.Cabal *<tab> <tab>*
pkgs.haskell.packages.ghc883.Cabal
pkgs.haskell.packages.ghc883.Cabal-ide-backend
pkgs.haskell.packages.ghc883.CabalSearch
pkgs.haskell.packages.ghc883.Cabal_2_2_0_1
pkgs.haskell.packages.ghc883.Cabal_2_4_1_0
pkgs.haskell.packages.ghc883.Cabal_3_2_0_0
So this is getting pulled into my nix-shell environment "somehow" and I think I am simply hitting the limit of my understanding on how Haskell nix packages are put together and in particular the GHC package set etc. Can someone shed some light into the apparently missing gap? Thanks!
EDIT: I am not sure if the question is actually clear.. the actual question is: why is there a Cabal_3_0_1_0 library in my GHC packages when everything in this Nixpkgs version suggests it should have a Cabal_3_2_0_0 and furthermore, how can I override this? Thanks!
I need to build the rustc compiler using a modified libc crate. I cloned the libc directory and made the changes, now how do I include the modified libc in my build?
This is my Cargo.toml
[patch.crates-io]
# Similar to Cargo above we want the RLS to use a vendored version of `rustfmt`
# that we're shipping as well (to ensure that the rustfmt in RLS and the
# `rustfmt` executable are the same exact version).
rustfmt-nightly = { path = "src/tools/rustfmt" }
# See comments in `src/tools/rustc-workspace-hack/README.md` for what's going on
# here
rustc-workspace-hack = { path = 'src/tools/rustc-workspace-hack' }
# See comments in `tools/rustc-std-workspace-core/README.md` for what's going on
# here
rustc-std-workspace-core = { path = 'src/tools/rustc-std-workspace-core' }
rustc-std-workspace-alloc = { path = 'src/tools/rustc-std-workspace-alloc' }
rustc-std-workspace-std = { path = 'src/tools/rustc-std-workspace-std' }
libc = {path = "../libc"}
[patch."https://github.com/rust-lang/rust-clippy"]
clippy_lints = { path = "src/tools/clippy/clippy_lints" }
[dependencies]
# libc = {verion = "0.2", default-features= false, path = "../libc"}
This is the error I get:
mahto#hydlnxeng27:/local/mnt/workspace/mahto/rust$ ./x.py build --config config.toml src/libstd 2>&1 | tee build.log
Updating only changed submodules
Submodules updated in 0.04 seconds
error: failed to parse manifest at `/local/mnt/workspace/mahto/rust/Cargo.toml`
Caused by:
virtual manifests do not specify [dependencies]
failed to run: /local/mnt/workspace/mahto/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /local/mnt/workspace/mahto/rust/src/bootstrap/Cargo.toml
Build completed unsuccessfully in 0:00:00
After commenting-out the dependencies section in Cargo.toml, I get this new error:
error[E0433]: failed to resolve: unresolved import
error: aborting due to previous error
For more information about this error, try `rustc --explain E0433`.
error: could not compile `libc`.
I am trying to build https://github.com/SaschaWillems/Vulkan. I wrote a small default.nix file:
{ stdenv, libxcb, pkgconfig, cmake, vulkan-loader, assimp }:
stdenv.mkDerivation rec {
name = "VulkanExamples";
buildDepends = [ cmake libxcb.dev pkgconfig vulkan-loader assimp ];
}
which I call with
nix-shell -E 'with import <nixpkgs> {}; callPackage ./default.nix {}'
But when I call cmake . I get
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Could NOT find xcb (missing: XCB_INCLUDE_DIR XCB_LIBRARY
What do I need to specify so that the cmake script can find the correct header files?
I finally got it working, it seems I had to use buildInputs
{ stdenv, libxcb, pkgconfig, cmake, vulkan-loader, assimp }:
with import <nixpkgs> {}; {
vulkanEnv = stdenv.mkDerivation {
name = "vulkan";
buildInputs = [ stdenv cmake libxcb pkgconfig vulkan-loader assimp ];
};
}