I have a workspace project with multiple packages. The two important ones are:
flowc - which is a lib and a binary
flowstdlib
flowc is a kind of compiler that I build as part of the project.
flowstdlib has a build script, that uses the flowc built binary to build that package (generate "code", files etc), so I need the flowc compiler ready when flowstdlib is to be built.
In cargo.toml of flowstdlib I define flowc as a build dependency:
[build-dependencies]
flowc = {path = "../flowc", version = "0.31.0" }`
(I've tried making it also a dependency, but no change)
in the build.rs of flowstdlib I look for it in the path, and if not found in the ../target/debug/flowc location:
let flowc = if Path::new(env!("CARGO_MANIFEST_DIR")).join("../target/debug/flowc").exists() {
"../target/debug/flowc"
} else if Simpath::new("PATH").find_type("flowc", FileType::File).is_ok() {
"flowc"
} else {
""
};
When I run the build, it looks like it's trying to build multiple packages at the same time in parallel:
Compiling flowstdlib v0.31.0 (/Users/andrew/workspace/flow/flowstdlib)
Compiling flowsamples v0.31.1 (/Users/andrew/workspace/flow/samples)
warning: Could not find `flowc` in $PATH or `target/debug`, so cannot build flowstdlib
error: failed to run custom build command for `flowsamples v0.31.1 (/Users/andrew/workspace/flow/samples)`
and the flowstdlib build fails as flowc binary is not built yet.
Since the build continues and eventually finishes building flowc, if I re-run the build, it will work the second time around (as flowc binary is now found).
So:
it looks like a build-dependency does wait for the depended-on binary to be built (maybe it waits for the library to be built, hard to tell)
Question
How I can make the build of flowstdlib wait for the completion of the flowc binary?
(without forcing a non-parallel build)
Pending the RFC landing, my workaround is to split the build into two commands (I'm using a Makefile to invoke them currently):
* cargo build -p flowc # will complete the build of the flowc binary
* cargo build # will build the entire workspace, including flowstdlib
Related
I'm trying to only build a library without having all tests dependencies available,
my tests directory have it's own SConscript file that run env.ParseConfig('pkg-config --libs --cflags libfuzzertestdependonthis')
And if I'm building the library by specifying the lib target only, the command ParseConfig will fail because the lib is not available in my build environment
The only solutions i found are really bad:
enclose env.ParseConfig in try expect block
checking the command line build target content to exclude some part of the SConstruct file
I wonder if there a smart way to do this, it would be great if ParseConfig could be a handled as a source node for a specific target instead of being run immediately.
Edit:my question don't seems to be clear enough, so I will try with a better example.
When I'm building in release mode, I don't have(don't want) the libcunit required to build the tests, the issue I'm facing is that ParseConfig command is always executed regarless of the target, and in this example ParsConfig will execute pkg-config --libs libcunit, which will fail because this lib is not installed.
Ok. From your update.
If you're building in your release mode, don't call ParseConfig()..
I'm specifying the -Cpanic=abort and -Zbuild-std=panic_abort when compiling. Why does the linker still say it needs libunwind to compile a program?
I'm experimenting with various ways to cross-compile Rust programs as small as possible (using the min-sized-rust repo as a reference). Right now I'm trying to compile the powerpc64-unknown-linux-musl target and I'm stuck on trying to remove a dependency on libunwind.
Here's my setup:
# 1. Install the Rust std source code
rustup component add rust-src --toolchain nightly
# 2. Setup a simple rust repo
cargo init testing
cd testing
# 3. Download a musl toolchain
wget https://musl.cc/powerpc64-linux-musl-cross.tgz
tar xzf powerpc64-linux-musl-cross.tgz
# 4. Try to compile the project (options on the command line instead of in files for
# maximum obviousness).
# RUSTFLAGS:
# -Cpanic=abort - abort immediately on panic
# -Clink-self-contained=no - don't use rustc's builtin libraries and objects (this
# is needed because powerpc64-unknown-linux-musl is a tier 3 target)
# -Clink-arg=--sysroot and -Clink-arg=/path/to/sysroot - pass the option to the linker
# to specify the sysroot of cross-compilation toolchain
# Cargo options:
# --config target.<triple>.linker - specify the linker to use
# -Zbuild-std=std,panic_abort - build the standard library from source. Specify
# panic_abort to make the abort on panic work
RUSTFLAGS="-Cpanic=abort -Clink-self-contained=no -Clink-arg=--sysroot -Clink-arg=powerpc64-linux-musl-cross/powerpc64-linux-musl/" \
cargo +nightly build \
--config "target.powerpc64-unknown-linux-musl.linker=\"powerpc64-linux-musl-cross/bin/powerpc64-linux-musl-gcc\"" \
--target powerpc64-unknown-linux-musl -Zbuild-std=panic_abort,std --release
This fails with the following error:
error: linking with `/home/user/Projects/testing/powerpc64-linux-musl-cross/bin/powerpc64-linux-musl-gcc` failed: exit status: 1
<output snipped>
= note: /home/user/Projects/testing/powerpc64-linux-musl-cross/bin/../lib/gcc/powerpc64-linux-musl/11.2.1/../../../../powerpc64-linux-musl/bin/ld: cannot find -lunwind
From min-size-rust repository:
"Even if panic = "abort" is specified in Cargo.toml, rustc will still include panic strings and formatting code in final binary by default. An unstable panic_immediate_abort feature has been merged into the nightly rustc compiler to address this.
To use this, repeat the instructions above to use build-std, but also pass the following -Z build-std-features=panic_immediate_abort option."
Still, you will get "cannot find -lunwind", because the linker still uses libunwind, even though it's truly unneeded,why! I do not know, maybe it's a bug.(Maybe someone with fair knowledge about linkers can easily solve that.I tried a naive solution which is "cargo .... --verbose", copy , remove "libunwind" then relinking which failed)
I verified that is indeed the missing piece by build from source(--target=x86_64-unknown-linux-musl) AND using an old simple trick which is "touch libunwind.a" in the "self-contained" directory inside a target lib folder.(because the linker would still use it even though it's now truly unneeded, then I gave him a dummy libunwind.a)
In your case, I really tried to build it to your target until I got a headache, but couldn't and stopped, but here is possible solutions:
Giving that you're using "-Z build-std-features=panic_immediate_abort"
-If you can custom the linking process, then solve it (until what seems to be a bug is solved)
-Create a dummy(empty) libunwind.a where it should be in your toolchain
I have a basic rust/cargo project with a single main file and some basic dependencies. The cargo build command works fine when the target is not specified (I am using windows so it builds to windows), but when I try to cross compile the program to linux using cargo build --target=x86_64-unknown-linux-gnu or cargo build --target=x86_64-unknown-linux-musl, the process fails with the following error: linker 'cc' not found.
Does anyone have an idea how to get around this? Is there a specific linker I need to install?
Thanks.
I've just figured it out.
It turns out you need to tell cargo to use the LLVM linker instead. You do this by creating a new directory called .cargo in your base directory, and then a new file called config.toml in this directory. Here you can add the lines:
[target.x86_64-unknown-linux-musl]
rustflags = ["-C", "linker-flavor=ld.lld"]
Then building with the command cargo build --target=x86_64-unknown-linux-musl should work!
I have a build script that is unnecessary for the library but required to build some examples. How do I specify a build file for an example with Cargo? I tried the following:
[[example]]
name = "basic"
build = "examples/build.rs"
but Cargo tells me the build key is unused.
Suppose I execute cargo new one --bin and cargo new two --bin then add the same dependency to each project's Cargo.toml and build them.
Now there are two absolutely identical sets of libraries:
/one/target/debug/deps/ *.rlib
/two/target/debug/deps/ *.rlib
They are same files and waste storage space, but really the problem is that I have to compile these libraries again for every project. It takes a very much time. There is the same problem with cargo install.
Can I specify a place to store compiled libraries to avoid recompilation?
Several Cargo projects might share the libraries by using the same target dir.
.cargo/config
Place a ".cargo" folder in a project and create a "config" file there containing:
[build]
target-dir = "/path/to/your/shared/target/dir"
On Unix this might look like:
mkdir ~/shared_rust_target
mkdir .cargo
echo "[build]" > .cargo/config
echo "target-dir = \"$HOME/shared_rust_target\"" >> .cargo/config
CARGO_TARGET_DIR
Set the CARGO_TARGET_DIR environment variable.
On Unix this might look like:
export CARGO_TARGET_DIR = "$HOME/shared_rust_target"
See this commit for some extra target-dir documentation.
In particular, prior to Cargo 1.9 you shouldn't build several projects into the same target dir concurrently. (Here's more on how Cargo 1.9 supports concurrent builds).
target-dir is also mentioned in the Cargo docs.
Should work, according to this issue.
P.S. It is now also possible to achieve crate reuse with workspaces.
P.S. https://docs.rs/cargo-hakari/latest/cargo_hakari/ helps with keeping some of dependencies compatible between projects.
Even if there is a way to do it, you probably don't want to. Just because you happen to be using the same libraries doesn't mean that they were compiled the same. For example, Cargo supports the concept of features, compilation time configuration that changes how the crate was compiled.
Likewise, you may need to support multiple versions of Rust, such as nightly and stable. Or perhaps you need to cross-compile for a different architecture. Each of those will produce different code.
Cargo will cache the build products of a single project, so I've found the overhead not really noticeable, and I compile a lot of projects from people asking questions on Stack Overflow! :-)
I managed to piece together code from a few answers, but mainly this one.
This Dockerfile should not just cache Cargo dependancy downloads, but also their compilations and the crates.io index. All the other answers I could find only cached the downloads or the index, not both.
FROM arm64v8/rust as builder
# Capture dependencies
COPY Cargo.toml Cargo.lock /app/
# We create a dummy main.rs to build deps
WORKDIR /app
RUN mkdir src && echo "fn main() {}" > src/main.rs
# RUN rustup install nightly && rustup default nightly
# This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
# Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release && rm src/main.rs
# Copy our sources
COPY ./src /app/src
# A bit of magic here!
# * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
# * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
RUN --mount=type=cache,target=/usr/local/cargo/registry \
set -e && \
# update timestamps to force a new build &&
touch /app/src/main.rs && \
cargo build --release
# Again, our final image is the same - a slim base and just our app
FROM debian:buster-slim as app
COPY --from=builder /app/target/release/app/app
CMD ["/app"]
Take note of the FROM arm64v8, if you are targeting x86, replace the builder and app FROMs with their respective x86 versions.