My package has a binary target that uses include_bytes!(…) to bundle a copy of some precomputed values into the compiled binary. This is an optimization, but isn't strictly necessary: the program is capable of calculating these values at run time if the bundled data slice .is_empty().
The program needs to be able to build without this data. However, include_bytes!("data/computed.bin") causes a build error if the target file does not exist.
error: couldn't read src/data/computed.bin: No such file or directory (os error 2)
Currently, I have a Bash build script that uses touch data/computed.bin to ensure the file exists before building. However, I don't want to depend on platform-specific solutions like Bash; I want to be able to build this project on any supported platform using cargo build.
How can my Rust program include_bytes!(…) or include_str!(…) from a file if it exits, but gracefully fall back to an alternative value or behaviour if the file doesn't exist, while only using the standard Cargo build tools?
We can use a build script to ensure that the included file exists before out package tries to include it. However, build scripts can only write to the current build's unique output directory, so we can't just create the missing input files in the source directory directly.
error: failed to verify package tarball
Caused by:
Source directory was modified by build.rs during cargo publish. Build scripts should not modify anything outside of OUT_DIR.
Instead, our build script can create the file-to-include in the build directory, copying the source data if it exists, and we can update our package code to include this data from the build directory instead of from the source directory. The build path will be available in the OUT_DIR environment variable during the build, so we can access it from std::env::var("OUT_DIR") in our build script and from env!("OUT_DIR") in the rest of our package.
//! build.rs
use std::{fs, io};
fn main() {
let out_dir = std::env::var("OUT_DIR").unwrap();
fs::create_dir_all(&format!("{}/src/data", out_dir))
.expect("unable to create data directory");
let path = format!("src/data/computed.bin", name);
let out_path = format!("{}/{}", out_dir, path);
let mut out_file = fs::OpenOptions::new()
.append(true)
.create(true)
.open(&out_path)
.expect("unable to open/create data file");
if let Ok(mut source_file) = fs::File::open(&path) {
io::copy(&mut source_file, &mut out_file).expect("failed to copy data after opening");
}
}
//! src/foo.rs
fn precomputed_data() -> Option<&'static [u8]> {
let data = include_bytes!(concat!(env!("OUT_DIR"), "/src/data/computed.bin")).as_ref();
if !data.is_empty() {
Some(data)
} else {
None
}
}
While using a build script (like in this answer) would work, I am not a fan of that solution:
The build script copies the file – depending on the file size, that might be prohibitively expensive. Though one could probably solve this problem using hardlinks instead.
An empty file might be perfectly fine data – the solution would misdetect an empty file as missing. However, depending on the use case, an empty file might actually be perfectly valid.
It is very verbose – this turns a simple include_bytes! into a build script of approximately 20 lines and additionally a few more lines when including to handle the data.is_empty() case.
It is hard to grasp what is happening here for a casual reader – why is this script including something from $OUT_DIR? It would probably take a moment for the reader to get the idea a build script might be involved here.
It does not scale well – most of those problems would get even worse if there were multiple files that needed to be included optionally.
I therefore decided to write the procedural macro crate include_optional to solve this problem (currently only works on nightly Rust because it depends on some unstable features).
With this, the solution to this problem is a one liner:
use include_optional::include_bytes_optional;
fn precomputed_data() -> Option<&'static [u8]> {
include_bytes_optional!("./computed.bin")
}
There are also macros wrapping include_str! and include!.
Related
I created a library that in order to work needs to parse a txt file every time you call the main method. The problem is that when importing into another project the txt file can not be found because I'm using env::current_dir(), when I call the method from the library folder current folder is the crate's root, and I can access root/src/my_file.txt. When importing and using the library the root is different and there isn't any my_file.txt.
How can resolve this? Here is the Crate
Here is how I access the file.
fn parse(&mut self, name_to_find: &str) -> () {
let p = env::current_dir().unwrap();
println!("{}", p.display());
let file = File::open(format!("{}/src/nam_dict.txt", p.display())).unwrap();
let lines = BufReader::new(file).lines();
...
Rust is a compiled language, so one can't assume access to the sourcecode and adjacent files at runtime.
So your options are
Include it at compile time with the include_str! macro. This means changes to the file won't be picked up until the library and its dependents are rebuilt
Locate the file at runtime, e.g. through a specified location (such as somewhere in the config hierarchy in the user's home directory), an environment variable, a commandline option or from the current directory. This way the file can be changed without recompiling the program but the user has to know that he must provide it.
More complicated approaches such as including a default configuration and letting the user override it are also possible.
In my library I have few examples -- (normally) each one is represented by a single x<N>.rs file living in examples directory.
One example uses a .proto file -- this file needs to be compiled during build (of said example) and it's generated output is used by example itself.
I've tried this in my Cargo.toml:
[[example]]
name = "x1"
path = "examples/x1/main.rs"
build = "examples/x1/build.rs"
but build key gets ignored when I run cargo build --example x1
Is it possible to have example-specific build.rs file?
If not -- what is the correct way to deal with this situation?
Edit: I ended up processing that .proto file in crate's build.rs (even though it is not required to build that crate) and using artefacts in the example like this:
pub mod my_proto {
include!(concat!(env!("OUT_DIR"), "/my_proto.rs"));
}
This is not possible. This issue explains why, but in a nutshell build scripts are used for whole crate. So you could move your example into separate crate.
I'm coming to Rust from the JavaScript world, where you could have a .env file (git-ignored), and environmental values from this file could be made available at runtime via an npm package (say, dotenv).
I was able to use a dotenv crate to make this work similarly when I just cargo run the project, but after I build a release version, the value of the environment variable is lost and the app panics.
I understand that cargo doesn't evaluate env variables at build time, and using env!-like macros won't work either, at least to my best understanding.
What I'm asking is this: how do I store a variable in a file (that I can gitignore), make Rust pick up values from this file, and use them at build time so that these values are available to the release-built app?
I'm sure that there's a well-established way of doing this, but I struggle to figure it out.
There is the dotenv-codegen crate that provides a dotenv! macro that works like the env! macro except it will load from a .env file.
I'm not so sure this is well-established, but you can use a build script that will read the file and use println!("cargo:rustc-env=VAR=VALUE") to send the environment variables to Cargo, allowing you to retrieve them in the code with env!() or option_env!().
For example, to use a .env file, add dotenv to build-dependencies, and use it like so in build.rs:
fn main() {
let dotenv_path = dotenv::dotenv().expect("failed to find .env file");
println!("cargo:rerun-if-changed={}", dotenv_path.display());
// Warning: `dotenv_iter()` is deprecated! Roll your own or use a maintained fork such as `dotenvy`.
for env_var in dotenv::dotenv_iter().unwrap() {
let (key, value) = env_var.unwrap();
println!("cargo:rustc-env={key}={value}");
}
}
The Rust language supports conditional compilation using attributes like #[cfg(test)].
Rust also supports build scripts using a build.rs file to run code as part of the build process to prepare for compilation.
I would like to use conditional compilation in Rust code to conditionally compile depending on whether we're compiling for a build script, similar to how that is possible for test builds.
Imagine the following:
#[cfg(build)]
fn main() {
// Part of build script
}
#[cfg(not(build))]
fn main() {
// Not part of build script, probably regular build
}
This does not work, because build is not a valid identifier here.
Is it possible to do this using a different attribute, or could some other trick be used to achieve something similar?
For some context on this issue:
My goal is to generate shell completion scripts through clap at compile time. I've quite a comprehensive App definition across multiple files in the application. I'd like to use this in build.rs by including these parts using the include!(...) macro (as suggested by clap), so I don't have to define App a second time. This pulls some dependencies with it, which I'd like to exclude when used by the build.rs file as they aren't needed in that case. This is what I'm trying to make available in my build.rs script.
You can just put the build code in build.rs (or presumably have build.rs declare a mod xyz to pull in another file).
I wonder if the question you are trying to ask is whether you can reference the same code from build.rs and main.rs, and if so can that code tell if it's being called by one or the other. It seems you could switch on an environment variable set when using build.rs (using something like option_env, but possibly a nicer way might be to enable a feature in the main code from within build.rs.
(Have a read of the documentation for build scripts if you haven't already.)
This is what works for me:
fn main() {
if std::env::var("PROFILE").unwrap() == "debug" {
// Here I do something that is needed for tests
// only, not for 'release'
}
}
I'm writing a Cargo helper command that needs to know the default target triple used by Rust/Cargo (which I presume is the same as host's target triple). Ideally it should be a compile-time constant.
There's ARCH constant, but it's not a full triple. For example, it doesn't distinguish between soft float and hard float ARM ABIs.
env!("TARGET") would be ideal, but it's set only for build scripts, and not the lib/bin targets. I could pass it on to the lib with build.rs and dynamic source code generation (writing the value to an .rs file in OUT_DIR), but it seems like a heavy hack just to get one string that the compiler has to know anyway.
Is there a more straightforward way to get the current target triple in lib/bin target built with Cargo?
Build scripts print some additional output to a file so you can not be sure that build script only printed output of $TARGET.
Instead, try something like this in build.rs:
fn main() {
println!(
"cargo:rustc-env=TARGET={}",
std::env::var("TARGET").unwrap()
);
}
This will fetch the value of the $TARGET environment variable in the build script and set it so it will be accessible when the program is started.
In my main.rs:
const TARGET: &str = env!("TARGET");
Now I can access the target triplet in my program. If you are using this technique, you'll only read the value of theTARGET environment variable and nothing else.
I don't think this is exposed other than through a build script. A concise way to get the target triple without "dynamic source code generation" would be, in build.rs:
fn main() {
print!("{}", std::env::var("TARGET").unwrap());
}
and in src/main.rs:
const TARGET: &str = include_str!(concat!(env!("OUT_DIR"), "/../output"));
fn main() {
println!("target = {:?}", TARGET);
}
If you just want to know which target is installed, run: rustup target list --installed