I'm creating a monorepo using Nx, Rust and TS. The Rust code is divided in crates, and since I'm using Nx, I've not used Cargo Workspaces. The problem I'm facing now is that every re-install the dependencies of the crates it depends on, needlessly increasing the size of target/debug/deps.
I'm looking for a way to install/store the dependencies in a unified – probably global – location, that can reduce this dependency duplication, like pnpm does for javascript. I know that cargo already caches the packages' source code globally, which is great, but it still stores the compiled objects in the target/debug/deps.
Its not possible but as #drewtato said you can use sccache to cache the crates to speed up build times. You can also try using a faster linker like mold to speed up linking
To use sccache (after installed):
(adapted from sccache readme)
Add this to your global cargo config $HOME/.cargo/config.toml or project cargo config project/.cargo/config.toml
[build]
rustc-wrapper = "/path/to/sccache"
You can also use the env var export RUSTC_WRAPPER=/path/to/sccache
Related
I work on a Rust project that has a lot of packages as explicit or implicit dependencies (~420). When I want to rebuild the target after changing the .env file (that configures things like IP to download files from), I would like to rebuild only the packages that I authored, not all the dependencies.
How can I tell cargo build to use the previously compiled dependencies, but not use the previously compiled package that uses the .env file as input?
Ideally, cargo build would realize that the .env file has changed and automatically decide to rebuild only the parts that use the .env file, but unfortunately this doesn't seem to be the case.
So the second best solution is to manually tell cargo build at which point in the build graph to start off again.
We're using the dotenv crate https://crates.io/crates/dotenv crate to read the .env file.
I tried cargo clean -p nextclade to tell it to clean only the package in question that I'm working on - but that still cleans up all the dependencies which cause my build to take 5 minutes rather than 2 minutes (if using compiled dependencies).
There's a question that seems to ask a similar question, but that question is actually a different use case/set up, so it's not a duplicate: How does cargo decide whether to rebuild the deps or not?
I have a rust project where I want to use protobuf definitions that exist in another, non-rust repository.
I'm planning to download the protobuf repository, create a src/common_protobuf module in my main repository, use cargo build to generate all the rust implementations of each protobuf into the common_protobuf module, then selectively re-export the generated structures into their intended modules with pub use.
I can't seem to find the best way to specify the dependency. Using
[build-dependencies]
pbrepo = { git="https://github.com/username/pbrepo" }
results in
Caused by:
Could not find Cargo.toml in `/Users/username/.cargo/git/checkouts/pbrepo-33abcde7dddd6356/fdefgd9`
I can't commit a Cargo.toml into the external repository. I considered using a git submodule or making a sys-crate with the submodule and Cargo.toml but I prefer to have my build dependencies in a single place and not have to synchronize multiple repos during a build.
I could download the repository manually in the the build scripts and set the commit to pull in my build.rs but again I would like to have all my build dependencies in one place.
Is there a mechanism to accomplish this or is this a better way to use remote protobuf definitions?
Cargo does not support downloading non-Rust dependencies. The only things that you can have Cargo download for you are Cargo packages (loosely “crates”).
Even if you added a Cargo.toml to your non-Rust repository, that would not help, since there is no way to ask for the location of that dependency to read from it. Cargo's dependencies mechanism lets you depend on a built library only, not its source files (as far as I know; I could have missed something).
You will have to use some separate procedure that downloads the dependency (custom script, git submodule…) before running cargo build.
I'd use cargo-make
It allows for more complex dependency tracking and building things from other languages.
Since you can't use Cargo directly, you have to resort to a higher-level mechanism, such as Git submodules, which can be downloaded automatically before building via build.rs. This way cargo build will both fetch the submodule and build your project in a single command.
I followed the tutorialenter link description here to this step，
make build
WASM_BUILD_TOOLCHAIN=nightly-2020-10-05 cargo build --release
Compiling node-template-runtime v2.0.0 (/home/wangliqiu2021/CLionProjects/substrate-node-template/runtime)
Building [=====================================================> ] 857/861: node-template-runtime(build)
Cargo has been executed for a long time(almost one+ hour) without ending,It seems to be stuck, does anyone know the reason？help me
OS：Ubuntu 20.04
CPU: AMD Ryzen 7 1700 Eight-Core Processor
The compile is not stuck in compilation, it is just taking a while due to the 800+ dependencies. From #gnunicorn on this github issue:
Rust isn't particular quick with the compiling at the moment and opaque to the person in front, at this step (when compiling node-template-runtime) we are actually building the project twice: once natively and once in wasm. So at some step in between it appears as if nothing happens and that can take up to half the total build time – if the other part took e.g. 10minutes then this process might take another 10min without any indicator of process (other than the CPU pumping hard).
You are doing a release build (cargo build --release) which enables optimizations. For development purposes, a regular build or just a cargo check will be substantially faster.
Some comments in the linked GitHub issue mentioned that running a cargo clean and rebuilding helped speed up compile times, so you can try that as well.
From your username. I think you are in China, same as me.
node-template-runtime(build) means you're compiling the runtime into a wasm file. During this, it might need a download (so try to use a VPN).
The download only happened in 1.0.0 https://docs.rs/substrate-wasm-builder/1.0.0/substrate_wasm_builder/?search=
Also, the wasm compiling will take a long time too (depend on your hardware).
In the 2016 MacBookPro, the whole compiling takes 30mins.
Moreover, there might be a bug in that build.rs. Sometimes I've to run cargo clean. If I interrupt the compiling while node-template-runtime(build).
I found the reason.
$CARGO_HOME/config.toml:
[build]
target-dir = "target"
remove it.
In my case the issue was, that the WASM runtime build expects to find the WASM files inside the local target directory.
If the build can't find the local WASM files in the target directory - it just hangs or deadlocks (e.g. GitHub issue).
I had to change the target-dir setting in ~/.cargo/config.toml.
You can also override the global setting with a local env var:
CARGO_TARGET_DIR=target cargo build
Or you can use the cargo CLI flags:
cargo build --target-dir=target
Note that some of the wasm-builder based build scripts ignore the CARGO_TARGET_DIR env variable (e.g. wasm_project.rs).
I am currently building some docker images.
I found that the Linux distribution I was using was hard to adapt to Docker multi-stage builds until I found Nix.
With Nix, I can copy files among images (COPY --from=source/image /nix/store /nix/store) without worrying about conflicts and breaking things.
But I found that it installed too many things after running nix-env -i curl command.
warning: there are multiple derivations named 'curl-7.60.0'; using the first one
installing 'curl-7.60.0'
these paths will be fetched (49.44 MiB download, 203.64 MiB unpacked):
/nix/store/0yaiablzxhd8ki5qan156ydz78grlav7-nghttp2-1.32.0-bin
/nix/store/0zvcf4dnlcd4bk84qmxcxm1pbc534chv-openssl-1.0.2o-bin
/nix/store/3xvnr0y2mx7g8b796rb9p77bjfbaw03h-linux-headers-4.15
/nix/store/4bikvz91b83sycavf35lmby65m6zxgch-libssh2-1.8.0-dev
/nix/store/504vcw350rp1yh31razv0mq2vsgp0izh-libkrb5-1.15.2-dev
/nix/store/5gzy6cacylfb0lha2yd0i0as0k1d0d5v-libev-4.24
/nix/store/5xnniwzazzlg6qinhrwammxxwsq5c1di-nghttp2-1.32.0-dev
/nix/store/7l1smzwil1kxyyfayzl6lg1hw9m4iwmw-nghttp2-1.32.0
/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23
/nix/store/93ljbaqhsipwamcn1acrv94jm6rjpcnd-acl-2.2.52
/nix/store/dgp8mnf40pmwh8ghpcfda1vcwcy34w6z-curl-7.60.0-devdoc
/nix/store/gbddfvxzjjqpgkr17whn8ynh9z8afz8l-curl-7.60.0-debug
/nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0
/nix/store/jg9yh6cm4iwcpl4l18g7mr9y7sdwav5q-curl-7.60.0-dev
/nix/store/jsmnk16iwb9xrm3c6jv2fyxkh7xr7q3j-curl-7.60.0-man
/nix/store/lyd89mv72m8a0aw1a4idfimyi0rb2b13-glibc-2.27-dev
/nix/store/n7qp8pffvcb5ff52l2nrc3g2wvxfrk75-coreutils-8.29
/nix/store/pa4q0szxz23bd6srry91gmw08fmwgfw2-libkrb5-1.15.2
/nix/store/q239yikz665n4a5rff7rg2vc7jpay6xb-openssl-1.0.2o-dev
/nix/store/rmq6gnybmxxzpssj3s63sfjivlq4inrm-attr-2.4.47
/nix/store/szdi35clpzj13c8dhfzh55fj6hk0z8j6-glibc-2.27-bin
/nix/store/v5xh3glylamhfg586hcykn6hlk4n41dh-nghttp2-1.32.0-lib
/nix/store/vawc9a89l53mf05yq0k1910q7dakd99w-perl-5.24.3
/nix/store/vl5k9m1pjkd6cm9125afic1kj06y4i6b-curl-7.60.0-bin
/nix/store/y8cfvcvya61l260jil989lcmkia5b5gh-zlib-1.2.11-dev
/nix/store/z4k2pbdd8pz9mjc0p5394j0zp435fcc5-curl-7.60.0
It is important to keep docker images slim and I do not think curl need dependencies like gcc or linux-headers at runtime.
Is there a way for Nix to exclude the dependencies of these source or dev libraries?
Build dependencies become runtime dependencies whenever a path name to the build dependency is included in the package. This is necessary because there is no general way to tell whether such a reference is actually used by a program.
The best way to avoid having build dependencies in your closures is by not referencing them in the first place. The next best thing is to understand why the reference is there and, if safe, modify the package build script to remove the reference.
In order to figure out where these references come from, you can make use of the Nix 2.0 nix why-depends command. It will tell you the shortest path, or all paths that lead from the first argument package to the second argument package. You can also use store paths instead of the attribute paths in the examples of nix why-depends --help.
The method for removing the dependency depends on the referencing package, so there's no general formula for that. General hacks to remove the reference in unsafe ways exist, but they are probably not worth the risk.
I am trying to create Rust bindings for the C++ library cryptominisat. The actual code works, but I'm not sure how to properly package it up with Cargo.
The git repository looks like
src/
c++ code here
.gitignore
readme, etc.
I added a rust directory, and created my Cargo project inside of it like so
rust/
cryptominisat/
Cargo.toml
build.rs
src/
rust code here
src/
c++ code here
.gitignore
readme, etc.
Unfortunately, cargo package doesn't seem to want to package up anything outside of the rust/cryptominisat directory, which means it doesn't include the C++ code needed to actually build the library. What can I do? I don't want to move the entire repository into the rust directory if I can avoid it, since that would make it impossible to merge upstream.
The way it's generally solved:
Use a git submodule (or a script run before publishing) to embed a copy of the C++ repo inside the Rust repo (e.g. in rust/cryptominisat/vendor/). During development you could use a symlink instead to avoid having two copies of the C++ code.
Use build.rs to download a tarball/clone/rsync the code at build time. You can dump it into OUT_DIR env var specified by Cargo to avoid polluting user-visible directories.
Make the C++ code a system-level library. The Rust package would not build it, but expect it's already installed, and only search for it and specify link flags for it. That's how most *-sys crates work.