I work on a Rust project that has a lot of packages as explicit or implicit dependencies (~420). When I want to rebuild the target after changing the .env file (that configures things like IP to download files from), I would like to rebuild only the packages that I authored, not all the dependencies.
How can I tell cargo build to use the previously compiled dependencies, but not use the previously compiled package that uses the .env file as input?
Ideally, cargo build would realize that the .env file has changed and automatically decide to rebuild only the parts that use the .env file, but unfortunately this doesn't seem to be the case.
So the second best solution is to manually tell cargo build at which point in the build graph to start off again.
We're using the dotenv crate https://crates.io/crates/dotenv crate to read the .env file.
I tried cargo clean -p nextclade to tell it to clean only the package in question that I'm working on - but that still cleans up all the dependencies which cause my build to take 5 minutes rather than 2 minutes (if using compiled dependencies).
There's a question that seems to ask a similar question, but that question is actually a different use case/set up, so it's not a duplicate: How does cargo decide whether to rebuild the deps or not?
Related
I'm creating a monorepo using Nx, Rust and TS. The Rust code is divided in crates, and since I'm using Nx, I've not used Cargo Workspaces. The problem I'm facing now is that every re-install the dependencies of the crates it depends on, needlessly increasing the size of target/debug/deps.
I'm looking for a way to install/store the dependencies in a unified – probably global – location, that can reduce this dependency duplication, like pnpm does for javascript. I know that cargo already caches the packages' source code globally, which is great, but it still stores the compiled objects in the target/debug/deps.
Its not possible but as #drewtato said you can use sccache to cache the crates to speed up build times. You can also try using a faster linker like mold to speed up linking
To use sccache (after installed):
(adapted from sccache readme)
Add this to your global cargo config $HOME/.cargo/config.toml or project cargo config project/.cargo/config.toml
[build]
rustc-wrapper = "/path/to/sccache"
You can also use the env var export RUSTC_WRAPPER=/path/to/sccache
I had the exact same question as Do I need both package-lock.json and package.json? (tldr; "what's the difference between package.json and package-lock.json?") and found some really great answers in there. However it leaves me with a few other very similar-related questions that I don't see answered elsewhere.
For instance, what if package.json and package-lock.json conflict with one another? Say package.json says to use some-lib-2.* (any 2.x version of some-lib) but package-lock.json is configured to use some-lib-1.18.4? Is there an error? Is preference given to either file as the "source of dependency truth"?
I like the idea of one file to manage my specific dependencies, and so I feel like I'm leaning towards:
Not specifying libraries or version in package.json at all; and
Using package-lock.json to specify the exact versions of each module/library my project uses
Is this possible to do? If so are there any special configurations that I need to make? Do I track both files in version control, or is there ever any reasons why I would not want to track these in git/VCS?
You use the the command line (npm install [optional args]) to update both files
NPM -- and your command line invocation -- decide what the acceptable ranges of dependency versions there are for module and define those ranges in package.json. It then picks a version within that range -- uses it for buildtime/runtime -- and writes that exact version in package-lock.json
So you want to place both files in version control so you have repeatable builds and any developers checking out your project will immediately be able to build the project with the same versions of the same dependencies
And the only time you edit package.json directly is if you don't want to allow a range of versions for a particular dependency and want to cherry pick the exact version to use. You make the edit, you save, you run npm install [options] and package-lock.json will be updated to use that version as well
For what it's worth, this is terribly confusing and advocates the anti-pattern of not managing your dependencies. It allows developers to think its OK to just pull in the latest version of a given dependency, even if that version changes from build to build. That leads to bug creep in your application, non-repeatable builds and all sorts of headaches.
I would strongly advocate for always specifying the exact version you want for all your direct dependencies: no more ranges or wildcards please.
I am currently building some docker images.
I found that the Linux distribution I was using was hard to adapt to Docker multi-stage builds until I found Nix.
With Nix, I can copy files among images (COPY --from=source/image /nix/store /nix/store) without worrying about conflicts and breaking things.
But I found that it installed too many things after running nix-env -i curl command.
warning: there are multiple derivations named 'curl-7.60.0'; using the first one
installing 'curl-7.60.0'
these paths will be fetched (49.44 MiB download, 203.64 MiB unpacked):
/nix/store/0yaiablzxhd8ki5qan156ydz78grlav7-nghttp2-1.32.0-bin
/nix/store/0zvcf4dnlcd4bk84qmxcxm1pbc534chv-openssl-1.0.2o-bin
/nix/store/3xvnr0y2mx7g8b796rb9p77bjfbaw03h-linux-headers-4.15
/nix/store/4bikvz91b83sycavf35lmby65m6zxgch-libssh2-1.8.0-dev
/nix/store/504vcw350rp1yh31razv0mq2vsgp0izh-libkrb5-1.15.2-dev
/nix/store/5gzy6cacylfb0lha2yd0i0as0k1d0d5v-libev-4.24
/nix/store/5xnniwzazzlg6qinhrwammxxwsq5c1di-nghttp2-1.32.0-dev
/nix/store/7l1smzwil1kxyyfayzl6lg1hw9m4iwmw-nghttp2-1.32.0
/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23
/nix/store/93ljbaqhsipwamcn1acrv94jm6rjpcnd-acl-2.2.52
/nix/store/dgp8mnf40pmwh8ghpcfda1vcwcy34w6z-curl-7.60.0-devdoc
/nix/store/gbddfvxzjjqpgkr17whn8ynh9z8afz8l-curl-7.60.0-debug
/nix/store/imfm3gk3qchmyv7684pjpm8irvkdrrkk-gcc-7.3.0
/nix/store/jg9yh6cm4iwcpl4l18g7mr9y7sdwav5q-curl-7.60.0-dev
/nix/store/jsmnk16iwb9xrm3c6jv2fyxkh7xr7q3j-curl-7.60.0-man
/nix/store/lyd89mv72m8a0aw1a4idfimyi0rb2b13-glibc-2.27-dev
/nix/store/n7qp8pffvcb5ff52l2nrc3g2wvxfrk75-coreutils-8.29
/nix/store/pa4q0szxz23bd6srry91gmw08fmwgfw2-libkrb5-1.15.2
/nix/store/q239yikz665n4a5rff7rg2vc7jpay6xb-openssl-1.0.2o-dev
/nix/store/rmq6gnybmxxzpssj3s63sfjivlq4inrm-attr-2.4.47
/nix/store/szdi35clpzj13c8dhfzh55fj6hk0z8j6-glibc-2.27-bin
/nix/store/v5xh3glylamhfg586hcykn6hlk4n41dh-nghttp2-1.32.0-lib
/nix/store/vawc9a89l53mf05yq0k1910q7dakd99w-perl-5.24.3
/nix/store/vl5k9m1pjkd6cm9125afic1kj06y4i6b-curl-7.60.0-bin
/nix/store/y8cfvcvya61l260jil989lcmkia5b5gh-zlib-1.2.11-dev
/nix/store/z4k2pbdd8pz9mjc0p5394j0zp435fcc5-curl-7.60.0
It is important to keep docker images slim and I do not think curl need dependencies like gcc or linux-headers at runtime.
Is there a way for Nix to exclude the dependencies of these source or dev libraries?
Build dependencies become runtime dependencies whenever a path name to the build dependency is included in the package. This is necessary because there is no general way to tell whether such a reference is actually used by a program.
The best way to avoid having build dependencies in your closures is by not referencing them in the first place. The next best thing is to understand why the reference is there and, if safe, modify the package build script to remove the reference.
In order to figure out where these references come from, you can make use of the Nix 2.0 nix why-depends command. It will tell you the shortest path, or all paths that lead from the first argument package to the second argument package. You can also use store paths instead of the attribute paths in the examples of nix why-depends --help.
The method for removing the dependency depends on the referencing package, so there's no general formula for that. General hacks to remove the reference in unsafe ways exist, but they are probably not worth the risk.
My project contains third-party library sources that located in separate directory:
/prj
/src
/app
/lib1
/lib2
/third-party-lib
SConscript
...
SConstruct
Compiling of the third-party-lib is quite long because of large library size. I'm never change the sources of the library and the only case when the library needs to be rebuild is changes of the build options (compiler flags, for example).
To rebuild the project I issue commands:
scons -c && scons
In this case SCons removes all build products including the third-party-lib and subsequent build consumes a significant time due to third-party-lib compiling which, as said above, never changed. Method:
lib = env.StaticLibrary(Target, obj)
env.NoClean(lib)
does not give desired result - this preserves only final library file (lib.a) from clean. I've tried to preserve object files:
obj = env.Object(Sources)
env.NoClean(obj)
but this solves the problem only partially, because some object files compiled implicitly - the library code contains Qt code which processed by Qt meta-object compiler (MOC), therefore these object files do not included in 'obj' list.
Is there a way to prevent such third-part library rebuilding every time when the project rebuild carried out?
The SCons way of doing things would be to not call "scons -c", but only
scons
if you want to rebuild your project.
It is SCons main strength to get all the dependencies (implicit and explicit) right, even for an iterative rebuild where only a handful of files have changed. By using the "-c" option, like you may be used from other build tools like "make", you're shortcutting this feature and creating problems where there would be none usually.
I assume that you're using your "make clean; make all" approach because you haven't properly defined all the dependencies in your project yet. Please do that first, it will help your build in the long run.
And no, there is no method that will prevent "cleaning" for a whole folder and its subdirs.
Use env.Glob(), it will see files which SCons knows about but are not yet created when that logic is run.
I am trying to create Rust bindings for the C++ library cryptominisat. The actual code works, but I'm not sure how to properly package it up with Cargo.
The git repository looks like
src/
c++ code here
.gitignore
readme, etc.
I added a rust directory, and created my Cargo project inside of it like so
rust/
cryptominisat/
Cargo.toml
build.rs
src/
rust code here
src/
c++ code here
.gitignore
readme, etc.
Unfortunately, cargo package doesn't seem to want to package up anything outside of the rust/cryptominisat directory, which means it doesn't include the C++ code needed to actually build the library. What can I do? I don't want to move the entire repository into the rust directory if I can avoid it, since that would make it impossible to merge upstream.
The way it's generally solved:
Use a git submodule (or a script run before publishing) to embed a copy of the C++ repo inside the Rust repo (e.g. in rust/cryptominisat/vendor/). During development you could use a symlink instead to avoid having two copies of the C++ code.
Use build.rs to download a tarball/clone/rsync the code at build time. You can dump it into OUT_DIR env var specified by Cargo to avoid polluting user-visible directories.
Make the C++ code a system-level library. The Rust package would not build it, but expect it's already installed, and only search for it and specify link flags for it. That's how most *-sys crates work.