I am working on some code where a buffer is backed by a statically sized array. Since Rust and the build tools provided by it offer the possibilities to compile conditionally, I can do something like this:
struct Buffer {
// default case, if none is set
#[cfg(not(buffersize))]
buffer: [f32; 16],
#[cfg(buffersize = "32")]
buffer: [f32; 32],
#[cfg(buffersize = "64")]
buffer: [f32; 64],
}
impl Buffer {
fn new() -> Buffer {
Buffer {
#[cfg(not(buffersize))]
buffer: [0.0; 16],
#[cfg(buffersize = "32")]
buffer: [0.0; 32],
#[cfg(buffersize = "64")]
buffer: [0.0; 64],
}
}
}
There is another question that uses features to compile the code conditionally. Using features alone, I would have to combine buffersize and the actual value e.g. buffersize16. Is it possible to provide the cfg flags to Cargo, or would I need to provide them directly to rustc?
You can set the environnment variable RUSTFLAGS or set rustflags variable in .cargo/config.
From environment-variables
RUSTFLAGS — A space-separated list of custom flags to pass to all
compiler invocations that Cargo performs. In contrast with cargo
rustc, this is useful for passing a flag to all compiler instances.
In your example, you could use :
RUSTFLAGS='--cfg buffersize="32"' cargo build
I want to post an update to my question as an additional option on how to pass (numeric) configuration values at compile time, that's possible through a build script.
Suppose you have following build script inside your project:
use std::env;
use std::fs::File;
use std::io::Write;
use std::path::Path;
fn main() {
println!("cargo:rerun-if-env-changed=SIZE");
let out_dir = env::var("OUT_DIR").unwrap();
let dest = Path::new(&out_dir).join("consts.rs");
let mut out_file = File::create(&dest).expect("Cannot create file");
let size: usize = env!("SIZE").parse().unwrap();
write!(&out_file, "pub const S : usize = {};", size);
}
It reads an environment variable at compile time, parses it as usize and writes a rust file (consts.rs) containing only the constant. Now, in your application code you can include this file and use the constant to eg. allocate memory on the stack:
include!(concat!(env!("OUT_DIR"), "/consts.rs"));
fn main() {
let array = [0.0f32; S];
println!("array len= {:?}", array.len());
}
The downside of this trick is, that you have to recompile the whole project (or parts of it) whenever the value of the environment variable changes, since cargo:rerun-if-env-changed=SIZE doesn't get captured. It also implies to always keep knowledge of this configuration option, but this could be wrapped in an additional build script like a makefile. Even if this is not the most elegant path to choose, it might be an option on certain occasions.
It would be nice to have this as macro option, though.
Related
Is it possible to define a constant in source code that can be overridden by a compiler flag? That is, something like setting a #define value in the C preprocessor with the -D key=val option to the compiler.
I have read about conditional compilation via the #[cfg(...)] attribute, but that only seems to support booleans. I want to allow the user to set the value of a constant during compilation.
Something like this:
#[from_cfg("max_dimensions")]
const MAX_DIMENSIONS: usize = 100_000;
Building on Lukas Kalbertodt's answer, you can get the environment variable as a constant number with some extra indirection, namely by using a build script.
build.rs
use std::{env, fs::File, io::Write, path::Path};
fn main() {
let out_dir = env::var("OUT_DIR").expect("No out dir");
let dest_path = Path::new(&out_dir).join("constants.rs");
let mut f = File::create(&dest_path).expect("Could not create file");
let max_dimensions = option_env!("MAX_DIMENSIONS");
let max_dimensions = max_dimensions
.map_or(Ok(10_000), str::parse)
.expect("Could not parse MAX_DIMENSIONS");
write!(&mut f, "const MAX_DIMENSIONS: usize = {};", max_dimensions)
.expect("Could not write file");
println!("cargo:rerun-if-env-changed=MAX_DIMENSIONS");
}
main.rs
include!(concat!(env!("OUT_DIR"), "/constants.rs"));
fn main() {
println!("The value is {} ({})", MAX_DIMENSIONS, MAX_DIMENSIONS + 1);
}
$ cargo run
The value is 10000 (10001)
$ MAX_DIMENSIONS=17 cargo run
The value is 17 (18)
$ MAX_DIMENSIONS=1 cargo run
The value is 1 (2)
No, you can't define constants (read: const bindings) with a compiler flag. But you can use the env! macro for something similar. It reads some environment variable at compile time.
const MAX_DIMENSIONS_RAW: &'static str = env!("MAX_DIMENSIONS");
Sadly, this returns a string and not an integer. Furthermore we can't yet call arbitrary functions (like parse) at compile time to calculate a constant. You could use lazy_static to achieve something similar:
lazy_static! {
static ref MAX_DIMENSIONS: usize = MAX_DIMENSIONS_RAW.parse().unwrap();
}
Of course you should add proper error handling. If your user doesn't need to define the environment variable, you can use option_env!.
With this approach, you can pass the setting at build time:
$ MAX_DIMENSIONS=1000 cargo build
Is it possible to define a constant in source code that can be overridden by a compiler flag? That is, something like setting a #define value in the C preprocessor with the -D key=val option to the compiler.
I have read about conditional compilation via the #[cfg(...)] attribute, but that only seems to support booleans. I want to allow the user to set the value of a constant during compilation.
Something like this:
#[from_cfg("max_dimensions")]
const MAX_DIMENSIONS: usize = 100_000;
Building on Lukas Kalbertodt's answer, you can get the environment variable as a constant number with some extra indirection, namely by using a build script.
build.rs
use std::{env, fs::File, io::Write, path::Path};
fn main() {
let out_dir = env::var("OUT_DIR").expect("No out dir");
let dest_path = Path::new(&out_dir).join("constants.rs");
let mut f = File::create(&dest_path).expect("Could not create file");
let max_dimensions = option_env!("MAX_DIMENSIONS");
let max_dimensions = max_dimensions
.map_or(Ok(10_000), str::parse)
.expect("Could not parse MAX_DIMENSIONS");
write!(&mut f, "const MAX_DIMENSIONS: usize = {};", max_dimensions)
.expect("Could not write file");
println!("cargo:rerun-if-env-changed=MAX_DIMENSIONS");
}
main.rs
include!(concat!(env!("OUT_DIR"), "/constants.rs"));
fn main() {
println!("The value is {} ({})", MAX_DIMENSIONS, MAX_DIMENSIONS + 1);
}
$ cargo run
The value is 10000 (10001)
$ MAX_DIMENSIONS=17 cargo run
The value is 17 (18)
$ MAX_DIMENSIONS=1 cargo run
The value is 1 (2)
No, you can't define constants (read: const bindings) with a compiler flag. But you can use the env! macro for something similar. It reads some environment variable at compile time.
const MAX_DIMENSIONS_RAW: &'static str = env!("MAX_DIMENSIONS");
Sadly, this returns a string and not an integer. Furthermore we can't yet call arbitrary functions (like parse) at compile time to calculate a constant. You could use lazy_static to achieve something similar:
lazy_static! {
static ref MAX_DIMENSIONS: usize = MAX_DIMENSIONS_RAW.parse().unwrap();
}
Of course you should add proper error handling. If your user doesn't need to define the environment variable, you can use option_env!.
With this approach, you can pass the setting at build time:
$ MAX_DIMENSIONS=1000 cargo build
I have a requirement to pass to a procedural macro either a text file or the contents of a text file, such that the procedural macro acts based on the contents of that text file at compile time. That is, the text file configures the output of the macro. The use case for this is the file defining a register map which the macro builds into a library.
The second requirement is that the text file is properly handled by Cargo, such that changes to the text file trigger a recompile in the same way as changes to the source file trigger a recompile.
My initial thought was to create a static string using the include_str! macro. This solves the second requirement but I can't see how to pass that to the macro - at that point I only have the identifier of the string to pass in:
use my_macro_lib::my_macro;
static MYSTRING: &'static str = include_str!("myfile");
my_macro!(MYSTRING); // Not the string itself!
I can pass a string to the macro with the name of the file in a string literal, and open the file inside the macro:
my_macro!("myfile");
At which point I have two problems:
It's not obvious how to get the path of the calling function in order to get the path of the file. I initially thought this would be exposed through the token Span, but it seems in general not (perhaps I'm missing something?).
It's not obvious how to make the file make Cargo trigger a recompile on changes. One idea I had to force this was to add an include_str!("myfile") to the output of the macro, which would hopefully result in the compile being made aware of "myfile", but this is a bit mucky.
Is there some way to do what I'm trying to do? Perhaps either by somehow getting the contents of the string inside the macro that was created outside, or reliably getting the path of the calling rust file (then making Cargo treat changes properly).
As an aside, I've read various places that tell me I can't get access to the contents of variables inside the macro, but it seems to me that this is exactly what the quote macro is doing with #variables. How is this working?
So it turns out this is possible in essentially the way I was hoping with the stable compiler.
If we accept that we need to work relative to the crate root, we can define our paths as such.
Helpfully, inside the macro code, std::env::current_dir() will return the current working directory as the root of the crate containing the call site. This means, even if the macro invocation is inside some crate hierarchy, it will still return a path that is meaningful at the location of the macro invocation.
The following example macro does essentially what I need. For brevity, it's not designed to handle errors properly:
extern crate proc_macro;
use quote::quote;
use proc_macro::TokenStream;
use syn::parse::{Parse, ParseStream, Result};
use syn;
use std;
use std::fs::File;
use std::io::Read;
#[derive(Debug)]
struct FileName {
filename: String,
}
impl Parse for FileName {
fn parse(input: ParseStream) -> Result<Self> {
let lit_file: syn::LitStr = input.parse()?;
Ok(Self { filename: lit_file.value() })
}
}
#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
let input = syn::parse_macro_input!(input as FileName);
let cwd = std::env::current_dir().unwrap();
let file_path = cwd.join(&input.filename);
let file_path_str = format!("{}", file_path.display());
println!("path: {}", file_path.display());
let mut file = File::open(file_path).unwrap();
let mut contents = String::new();
file.read_to_string(&mut contents).unwrap();
println!("contents: {:?}", contents);
let result = quote!(
const FILE_STR: &'static str = include_str!(#file_path_str);
pub fn foo() -> bool {
println!("Hello");
true
}
);
TokenStream::from(result)
}
Which can be invoked with
my_macro!("mydir/myfile");
where mydir is a directory in the root of the invoking crate.
This uses the hack of using an include_str!() in the macro output to cause rebuilds on changes to myfile. This is necessary and does what is expected. I would expect this to be optimised out if it's never actually used.
I'd be interested to know if this approach falls over in any situation.
Relevant to my original question, current nightly implements the source_file() method on Span. This might be a better way to implement the above, but I'd rather stick with stable. The tracking issue for this is here.
Edit:
The above implementation fails when the package is in a workspace, at which point the current working directory is the workspace root, not the crate root. This is easy to work around with something like as follows (inserted between cwd and file_path declarations).
let mut cwd = std::env::current_dir().unwrap();
let cargo_path = cwd.join("Cargo.toml");
let mut cargo_file = File::open(cargo_path).unwrap();
let mut cargo_contents = String::new();
cargo_file.read_to_string(&mut cargo_contents).unwrap();
// Use a simple regex to detect the suitable tag in the toml file. Much
// simpler than using the toml crate and probably good enough according to
// the workspace RFC.
let cargo_re = regex::Regex::new(r"(?m)^\[workspace\][ \t]*$").unwrap();
let workspace_path = match cargo_re.find(&cargo_contents) {
Some(val) => std::env::var("CARGO_PKG_NAME"),
None => "".to_string()
};
let file_path = cwd.join(workspace_path).join(input.filename);
let file_path_str = format!("{}", file_path.display());
Is it possible to define a constant in source code that can be overridden by a compiler flag? That is, something like setting a #define value in the C preprocessor with the -D key=val option to the compiler.
I have read about conditional compilation via the #[cfg(...)] attribute, but that only seems to support booleans. I want to allow the user to set the value of a constant during compilation.
Something like this:
#[from_cfg("max_dimensions")]
const MAX_DIMENSIONS: usize = 100_000;
Building on Lukas Kalbertodt's answer, you can get the environment variable as a constant number with some extra indirection, namely by using a build script.
build.rs
use std::{env, fs::File, io::Write, path::Path};
fn main() {
let out_dir = env::var("OUT_DIR").expect("No out dir");
let dest_path = Path::new(&out_dir).join("constants.rs");
let mut f = File::create(&dest_path).expect("Could not create file");
let max_dimensions = option_env!("MAX_DIMENSIONS");
let max_dimensions = max_dimensions
.map_or(Ok(10_000), str::parse)
.expect("Could not parse MAX_DIMENSIONS");
write!(&mut f, "const MAX_DIMENSIONS: usize = {};", max_dimensions)
.expect("Could not write file");
println!("cargo:rerun-if-env-changed=MAX_DIMENSIONS");
}
main.rs
include!(concat!(env!("OUT_DIR"), "/constants.rs"));
fn main() {
println!("The value is {} ({})", MAX_DIMENSIONS, MAX_DIMENSIONS + 1);
}
$ cargo run
The value is 10000 (10001)
$ MAX_DIMENSIONS=17 cargo run
The value is 17 (18)
$ MAX_DIMENSIONS=1 cargo run
The value is 1 (2)
No, you can't define constants (read: const bindings) with a compiler flag. But you can use the env! macro for something similar. It reads some environment variable at compile time.
const MAX_DIMENSIONS_RAW: &'static str = env!("MAX_DIMENSIONS");
Sadly, this returns a string and not an integer. Furthermore we can't yet call arbitrary functions (like parse) at compile time to calculate a constant. You could use lazy_static to achieve something similar:
lazy_static! {
static ref MAX_DIMENSIONS: usize = MAX_DIMENSIONS_RAW.parse().unwrap();
}
Of course you should add proper error handling. If your user doesn't need to define the environment variable, you can use option_env!.
With this approach, you can pass the setting at build time:
$ MAX_DIMENSIONS=1000 cargo build
Is there an idiomatic way to process a file one character at a time in Rust?
This seems to be roughly what I'm after:
let mut f = io::BufReader::new(try!(fs::File::open("input.txt")));
for c in f.chars() {
println!("Character: {}", c.unwrap());
}
But Read::chars is still unstable as of Rust v1.6.0.
I considered using Read::read_to_string, but the file may be large and I don't want to read it all into memory.
Let's compare 4 approaches.
1. Read::chars
You could copy Read::chars implementation, but it is marked unstable with
the semantics of a partial read/write of where errors happen is currently unclear and may change
so some care must be taken. Anyway, this seems to be the best approach.
2. flat_map
The flat_map alternative does not compile:
use std::io::{BufRead, BufReader};
use std::fs::File;
pub fn main() {
let mut f = BufReader::new(File::open("input.txt").expect("open failed"));
for c in f.lines().flat_map(|l| l.expect("lines failed").chars()) {
println!("Character: {}", c);
}
}
The problems is that chars borrows from the string, but l.expect("lines failed") lives only inside the closure, so compiler gives the error borrowed value does not live long enough.
3. Nested for
This code
use std::io::{BufRead, BufReader};
use std::fs::File;
pub fn main() {
let mut f = BufReader::new(File::open("input.txt").expect("open failed"));
for line in f.lines() {
for c in line.expect("lines failed").chars() {
println!("Character: {}", c);
}
}
}
works, but it keeps allocation a string for each line. Besides, if there is no line break on the input file, the whole file would be load to the memory.
4. BufRead::read_until
A memory efficient alternative to approach 3 is to use Read::read_until, and use a single string to read each line:
use std::io::{BufRead, BufReader};
use std::fs::File;
pub fn main() {
let mut f = BufReader::new(File::open("input.txt").expect("open failed"));
let mut buf = Vec::<u8>::new();
while f.read_until(b'\n', &mut buf).expect("read_until failed") != 0 {
// this moves the ownership of the read data to s
// there is no allocation
let s = String::from_utf8(buf).expect("from_utf8 failed");
for c in s.chars() {
println!("Character: {}", c);
}
// this returns the ownership of the read data to buf
// there is no allocation
buf = s.into_bytes();
buf.clear();
}
}
I cannot use lines() because my file could be a single line that is gigabytes in size. This an improvement on #malbarbo's recommendation of copying Read::chars from the an old version of Rust. The utf8-chars crate already adds .chars() to BufRead for you.
Inspecting their repository, it doesn't look like they load more than 4 bytes at a time.
Your code will look the same as it did before Rust removed Read::chars:
use std::io::stdin;
use utf8_chars::BufReadCharsExt;
fn main() {
for c in stdin().lock().chars().map(|x| x.unwrap()) {
println!("{}", c);
}
}
Add the following to your Cargo.toml:
[dependencies]
utf8-chars = "1.0.0"
There are two solutions that make sense here.
First, you could copy the implementation of Read::chars() and use it; that would make it completely trivial to move your code over to the standard library implementation if/when it stabilizes.
On the other hand, you could simply iterate line by line (using f.lines()) and then use line.chars() on each line to get the chars. This is a little more hacky, but it will definitely work.
If you only wanted one loop, you could use flat_map() with a lambda like |line| line.chars().