Does anyone know how to get cabal install to exploit parallelism? I'm compiling with GHC, and while I don't know if GHC itself can do parallel builds, surely cabal install could run multiple compilations in parallel, no? At least for distinct, independent packages?
Does anyone know if it is possible and how to do it?
I'm the one who was working on this Summer of Code project. The patches have been sent to Duncan, but he hasn't reviewed them yet. Note that my code works at the package granularity, so you won't get any speedup when building a single package. I'm currently working on a parallel wrapper around ghc --make, which will solve this problem (I hope to get it merged into the mainline cabal-install eventually).
Update (Feb. 2012): Duncan has reviewed my patches, I need to incorporate his feedback and resubmit them. I hope to get this done before the end of this month.
Update (Apr. 2012): I've updated my patches in response to Duncan's comments. The new code is a bit slower, but requires much less changes to the Cabal library.
Update (Jun. 2012): Duncan Coutts just merged the parallel branch into Cabal HEAD. Parallel install will be available in the next cabal-install release.
Update (Oct. 2012): cabal-install1.16.0 has just been released. This is the first official release that includes my parallel patches.
Completing Mikhail Glushenkov's answer to document the usage a bit:
As of cabal 1.16.0, you can now use
cabal install -j [pkgs…]
This defaults to one job per core. It also gives much cleaner output.
You can make parallel installs the default with:
echo "jobs: $(getconf _NPROCESSORS_ONLN)" >> ~/.cabal/config
Or (cabal-install 1.18+):
echo 'jobs: $ncpus' >> ~/.cabal/config
Get the latest cabal-install with:
cabal update
cabal install cabal-install --bindir ~/bin --upgrade-dependencies
There was a Google Summer of Code project this summer to parallelize cabal-install. While it hasn't been merged into the mainline yet, the linked article provides instructions for grabbing the source and building it yourself.
Related
I have a project that I want to compile with LTS 2, but I'm stuck trying to work around this well-known bug in OS X 10.11.
Can I somehow specify a newer version of Cabal to use with ghc-7.8.4? Note that the project only really compiles with GHC 7.8.*.
Edit:
Really meant Cabal, not cabal-install.
Edit 2:
Actually, the answer to this question doesn't really solve my concrete problem (compiling helium, which requires GHC <= 7.8.4), because the bug is not specific to Cabal, but rather to unix, which GHC < 7.10 depends upon.
I think the problem is related not just to Cabal library version (btw, stack doesn't use or need cabal-install), but also to GHC version. Stack's FAQ mentions an item about this error, but I think it didn't help for me and I just upgraded to 7.10.2 at the time.
Here's a recipe for installing ghc-7.8.4 into the directory of your choice:
Download the "bindist" from https://www.haskell.org/ghc/download_ghc_7_8_4#macosx_x86_64
Untar it into a scratch directory
Create the destination directory, e.g. $HOME/my-prefix.
In the scratch directory run:
$ ./configure --prefix=$HOME/my-prefix
$ make install
Now ghc may be found in $HOME/my-prefix/bin.
As noted in Michael Snoyman's comment, there is stack setup --upgrade-cabal to do just that. That's it for the answer to this question.
In my specific case (XY problem), for that to work I had to do the usual work-around for that El Capitano related bug, which is explained here. So I enabled rootless, ran the upgrade command and re-enabled rootless.
To no avail I'm afraid: The bug is specific to unix < 2.7.1.0, which GHC 7.8.4 also depends on. So even after upgrading cabal, I can't compile anything in lts-2 unless I disable rootless, which is pretty annoying.
Actually it's kinda depressing that the GHC folks decided not to file a new minor release with the updated unix package. This means that no realistic/recent OS X configuration can build anything which involves GHC < 7.10.
I'm going to install GHC on a fresh copy of Ubuntu and I'm wondering: How can I ensure that profiling libraries are installed for the core libraries (e.g., text, unordered-containers)?
I'm aware of the changing the profiling option in cabal's .config file but my understanding is that this only ensures that profiling libraries are installed for those packages that I install AFTER setting up cabal (see italicized text in update below).
(I inadvertently blew up my Ubuntu vbox last night as a result of trying to retroactively install profiling libraries for installed GHC packages. It led to the existing packages not working, which led to me trying to uninstall GHC, which led to...KABOOM!)
UPDATE:
I've installed GHC and am now at the point where I'm about to install cabal. I've confirmed my suspicion that I'm facing a "chicken-and-an-egg" dilemma: In order to get the initial cabal config file (in which I can set profiling option as True), I need to install cabal. However, installing cabal results in the installation of core packages (e.g., text, unordered-containers) BEFORE I get a chance to make the change in the cabal config file.
SOLVED:
As per Daniel Wagner's suggestion (thanks!), I made a couple of modifications to the bootstrap.sh script file (I unfortunately didn't have my old cabal or I would've followed his other suggestion). As reference for future readers, the beginning of my bootstrap.sh file looked like this (after the changes):
#VERBOSE
DEFAULT_CONFIGURE_OPTS="--enable-library-profiling --enable-shared"
EXTRA_CONFIGURE_OPTS=${EXTRA_CONFIGURE_OPTS-$DEFAULT_CONFIGURE_OPTS}
#EXTRA_CONFIGURE_OPTS
#EXTRA_BUILD_OPTS
#EXTRA_INSTALL_OPTS
The preferred solution is to install cabal-install via your package manager. If you have an old version of cabal-install in your package manager, you can then use the old version to install the new version with a config in place, or even specify profiling directly on the command line via cabal install cabal-install --enable-library-profiling.
An alternate solution if you plan to install cabal-install via its bootstrap.sh script is to use the environment variables it provides for configuration. There are four, notated at the top of bootstrap.sh; the relevant one is EXTRA_CONFIGURE_OPTS, which contents get passed to the configure step of each package's Setup command. So something like this ought to do the trick (though I haven't tested it):
EXTRA_CONFIGURE_OPTS=--enable-library-profiling ./bootstrap.sh
I'm experimenting a problem with the interaction between the ghc-mod plugin in emacs, and NixOS 14.04. Basically, once packages are installed via nix-env -i, they are visible from ghc and ghci, recognised by haskell-mode, but not found by ghc-mod.
To avoid information duplication, you can find all details, and the exact replication of the problem in a VM, in the bug ticket https://github.com/kazu-yamamoto/ghc-mod/issues/269
The current, default, package management set up for Haskell on NixOS does work will with packages that use the ghc-api, or similar (ghc-mod, hint, plugins, hell, ...) run time resources. It takes a little more work to create a Nix expression that integrates them well into the rest of the environment. It is called making a wrapper expression for the package, for an example look at how GHC is installed an operates on NixOS.
It is reasonable that this is difficult since you are trying to make a install procedure that is atomic, but interacts with an unknown number of other system packages with their own atomic installs and updates. It is doable, but there is a quicker work around.
Look at this example on the install page on the wiki. Instead of trying to create a ghc-mod package that works atomically you weld it on to ghc so ghc+ghc-mod is an atomic update.
I installed ghc+ghc-mod with the below install script added to my ~/.nixpkgs/nixpkgs.nix file.
hsEnv = haskellPackages.ghcWithPackages (self : [
self.ghc
self.ghcMod
# add more packages here
]);
Install package with something like:
nix-env -i hsEnv
or better most of the time:
nix-env -iA nixpkgs.haskellPackages.hsEnv
I have an alias for the above so I do not have to type it out every time. It is just:
nixh hsEnv
The down side of this method is that other Haskell packages installed with nix-env -i[A] will not work with the above installation. If I wanted to get everything working with the lens package then I would have to alter the install script to include lens like:
hsEnv = haskellPackages.ghcWithPackages (self : [
self.ghc
self.ghcMod
self.lens
# add more packages here
]);
and re-install. Nix does not seem to use a different installation for lens or ghc-mod in hsEnv and with the ghc from nix-env -i ghc so apparently only a little more needs to happen behind the scenes most of the time to combine existing packages in the above fashion.
ghc-mod installed fine with the above script but I have not tested out its integration with Emacs as of yet.
Additional notes added to the github thread
DanielG:
I'm having a bit of trouble working with this environment, I can't even get cabal install to behave properly :/ I'm just getting lots of errors like:
With Nix and NixOS you pretty much never use Cabal to install at the global level
Make sure to use sandboxes, if you are going to use cabal-install. You probably do not need it but its there and it works.
Use ghcWithPackages when installing packages like ghc-mod, hint, or anything needs heavy runtime awareness of existing package (They are hard to make atomic and ghcWithPackages gets around this for GHC).
If you are developing install the standard suite of posix tools with nix-env -i stdenv. NixOS does not force you to have your command line and PATH cultured with tools you do not necessarily need.
cabal assumes the existence a few standard tools such as ar, patch(I think), and a few others as well if memory services me right.
If you use the standard install method and/or ghcWithPackages when needed then NixOS will dedup, on a package level (If you plot a dependency tree they will point to the same package in /nix/store, nix-store --optimise can always dedup the store at a file level.), many packages automatically unlike cabal sandboxes.
Response to comment
[carlo#nixos:~]$ nix-env -iA nixos.pkgs.hsEnv
installing `haskell-env-ghc-7.6.3'
these derivations will be built:
/nix/store/39dn9h2gnp1pyv2zwwcq3bvck2ydyg28-haskell-env-ghc-7.6.3.drv
building path(s) `/nix/store/minf4s4libap8i02yhci83b54fvi1l2r-haskell-env-ghc-7.6.3'
building /nix/store/minf4s4libap8i02yhci83b54fvi1l2r-haskell-env-ghc-7.6.3
collision between `/nix/store/1jp3vsjcl8ydiy92lzyjclwr943vh5lx-ghc-7.6.3/bin/haddock' and `/nix/store/2dfv2pd0i5kcbbc3hb0ywdbik925c8p9-haskell-haddock-ghc7.6.3-2.13.2/bin/haddock' at /nix/store/9z6d76pz8rr7gci2n3igh5dqi7ac5xqj-builder.pl line 72.
builder for `/nix/store/39dn9h2gnp1pyv2zwwcq3bvck2ydyg28-haskell-env-ghc-7.6.3.drv' failed with exit code 2
error: build of `/nix/store/39dn9h2gnp1pyv2zwwcq3bvck2ydyg28-haskell-env-ghc-7.6.3.drv' failed
It is the line that starts with collision that tells you what is going wrong:
collision between `/nix/store/1jp3vsjcl8ydiy92lzyjclwr943vh5lx-ghc-7.6.3/bin/haddock' and `/nix/store/2dfv2pd0i5kcbbc3hb0ywdbik925c8p9-haskell-haddock-ghc7.6.3-2.13.2/bin/haddock' at /nix/store/9z6d76pz8rr7gci2n3igh5dqi7ac5xqj-builder.pl line 72.
It is a conflict between two different haddocks. Switch to a new profile and try again. Since this is a welding together ghc+packages it should not be installed in a profile with other Haskell packages. That does not stop you from running binaries and interrupters from both packages at once, they just need to be in their own name space so when you call haddock, cabal, ghc, there is only one choice per profile.
If you are not familiar with profiles yet you can use:
nix-env -S /nix/var/nix/profiles/per-user/<user>/<New profile name>
The default profile is either default or channels do not which one it will be for your set up. But check for it so you can switch back to it later. There are some tricks so that you do not have to use the /nix/var/nix/profiles/ directory to store you profiles to cut down on typing but that is the default location.
I'm currently trying out web development frameworks for haskell and I recently came across yesod. It seemed pretty interesting so I installed it using cabal, however I'm not able to run the development server. Following their getting started instructions here's the result:
$ yesod init
$ cd mysite
$ yesod devel
Configuring mysite-0.0.0...
Testing files...
Rebuilding app
yesod: bind: resource busy (Address already in use)
Preprocessing library mysite-0.0.0...
Preprocessing executables for mysite-0.0.0...
Building mysite-0.0.0...
Controller.hs:16:7:
Could not find module `Data.Dynamic':
It is a member of the hidden package `base'.
Perhaps you need to add `base' to the build-depends in your .cabal file.
It is a member of the hidden package `base-3.0.3.2'.
Perhaps you need to add `base' to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
Testing files...
Testing files...
^^ above line just keeps repeating...
I'm assuming it has something to do with the Data.Dynamic module but I don't know how to go about fixing it.
Additional Info
Running Ubuntu 10.10 Maverick
ghc version:
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 6.12.1
I haven't run into this specific issue, but the error message looks like it's a simple question of GHC being unable to find version 3.0.3.2 the package "base." This version has been buildable since GHC 6.9, so you should have it. Try running the following command:
ghc-pkg check
This will tell you if there is something wrong with your packages. Cabal can be a bit of a nightmare for dependencies -- partly, it seems, because a lot of Haskell developers underestimate the extent to which their underlying libraries will shift in the future. So they will define a dependency as ">= [version of package x]" without limiting the max version to the one presently available. Or they just leave out version-limiting altogether.
Yesod, I'm happy to say, doesn't fall into this trap. But several of the libraries it depends on do. When you start developing in Haskell, learn this lesson: never assume that future versions of a library won't break your code. They will. A lot.
If ghc-pkg comes up with broken packages, you may need to clean up/uninstall/reinstall these packages until they are either cleaned up or hidden. (Just do ghc-pkg hide [package name] to tell ghc to ignore that package.
Your next problem is that hidden base package. Try the following:
ghc-pkg list | grep base
If you see brackets around the library, that means it's hidden. The package base-3.0.3.2 might show up as hidden (although that's a bit unlikely, as that's where the backward-compatible Prelude lives). If it is hidden, try to unhide it with the following command:
ghc-pkg expose base-3.0.3.2
Now try re-running yesod devel and see how it goes. Best case scenario is that it works. If not, let us know.
According to the Yesod in Five Minutes guide, you appear to be missing the a call to the command "cabal install" between your "cd mysite" and "yesod devel". It may need to install further packages based on what your responses were during "yesod init", such as which database you want to use.
Also, you may want to check that the port is not currently being used, as you have the "Address already in use" shown in your transcript.
First, it would be a good idea to resolve any broken packages reported by ghc-pkg check, by removing/reinstalling/upgrading them.
Next, can you manually cabal build the mysite app without trouble ? If your mysite.cabal actually does not specify base in the build-depends list, you should follow the suggestion to add that.
The repeating Testing Files message is normal for current yesod devel, it is polling your source files.
So I recently installed cabal (from the default binary of ArchLinux).
I then tried to upgrade cabal as a user:
cabal upgrade Cabal --user --prefix=$USER
Resolving dependencies...
cabal: fromFlag NoFlag. Use fromFlagOrDefault
What I've already done:
Googled the error message. Turned up the cabal source and little else.
Looked at haskell-wiki on cabal-install.
Looked through this guide.
So basically I'm wondering:
What's up with the error message?
Could anyone point me in the direction of a cabal tutorial?
I think that's a bug in Arch's package. I'm running Arch as well, and I got the same error.
I then built cabal-install (0.8.2) from Hackage, which didn't complain when I run cabal upgrade Cabal. In fact, it started building right away, although I didn't let it finish because I didn't feel comfortable upgrading a core package.
May I ask why you wanted to upgrade Cabal? The version that comes with ghc is 1.8.0.2, and the latest version is 1.8.0.4 -- not too distant IMO. Besides, if you're running cabal-install as normal user, why not simply cabal install? It doesn't have permission to overwrite the system files anyway.
I'm guessing wildly here, but can you see if
You have a nonstandard Setup.hs?
You have flags: set in ~/.cabal/config?
The most helpful tutorial I have found is not ostensibly about Cabal; it is called How to Write a Haskell Program.
I have looked through the guide you mention (the official Cabal documentation), and the information I wanted is a tiny bit of flotsam swimming in a sea of complexity. I assume that the complexity serves some useful purpose, since I know that good people have been working very hard on Cabal for several years, but as an outsider, I find the state of play discouraging. I was just barely able to build, install, and upload a simple package of my own—with a little help from my friends on SO.
I tentatively suggest you might want to avoid cabal upgrade and maybe try cabal install instead, and see if you can move forward with your original goals.
From user tlo:
The global configuration file for cabal-install is ~/.cabal/config. If
you do not have this file, cabal will create it for you on the first
call to cabal update.
I deleted my config, ran cabal update, and all was good.