I created a "hello world" Rust app using cargo new. When I executed git status it showed a bunch of files:
A rust/welcomec/Cargo.lock
A rust/welcomec/Cargo.toml
A rust/welcomec/src/main.rs
A rust/welcomec/target/debug/.cargo-lock
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/bin-welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/bin-welcome-2d68725c8fae6fd1.json
A rust/welcomec/target/debug/.fingerprint/welcomec-2d68725c8fae6fd1/dep-bin-welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/deps/welcome-2d68725c8fae6fd1
A rust/welcomec/target/debug/welcome
A rust/welcomec/target/debug/welcome.d
Can I safely ignore any of these files and/or directories?
Summary
.gitignore for library crates
# Generated files
/target/
# The library shouldn't decide about the exact versions of
# its dependencies, but let the downstream crate decide.
Cargo.lock
.gitignore for executable crates
# Generated files
/target/
Details
You need one or two entries in your .gitignore, depending on what kind of crate you're building. The target/ folder can be ignore completely regardless of crate type; it only contains generated files (e.g. compile artifacts).
The Cargo.lock file should be included in the repository if you're writing an executable, and should be ignored if you're writing a library. You can read more about this in the FAQ. To quote the most important part:
The purpose of a Cargo.lock is to describe the state of the world at the time of a successful build. [...]
This property is most desirable from applications and projects which are at the very end of the dependency chain (binaries). As a result, it is recommended that all binaries check in their Cargo.lock.
For libraries the situation is somewhat different. [...] If a library ends up being used transitively by several dependencies, it’s likely that just a single copy of the library is desired (based on semver compatibility). If all libraries were to check in their Cargo.lock, then multiple copies of the library would be used, and perhaps even a version conflict.
Also, please note that cargo new and cargo init automatically generates a .gitignore file in the project, unless the parameter --vcs none is passed.
You can take some inspiration from GitHub's gitignore for Rust. At the time of writing, the file goes as follows:
# Generated by Cargo
# will have compiled files and executables
debug/
target/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Cargo.lock
# These are backup files generated by rustfmt
**/*.rs.bk
# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb
Related
I am trying to create a library in rust to be used with rust executables. In C you can just create your .a or .so (or .lib or .dll on windows) and use tools like CMake to link everything, however rust does not seem to have this kind of infrastructure?
It is possible to make an executable with cargo (cargo new ) and create a library by adding the --lib flag (cargo new --lib), but then how would you use the resulting .rlib file (from the library cargo project)? I managed to link the .rlib file as follows:
rustc main.rs --extern foo=libfoo.rlib
and that works beautifully, though, I am not interested in writing a thousand rustc commands to build the final executable (which depends on the .rlib) if there is cargo that can do that for you. I tried working with a build script (which works perfectly for any C library, static or dynamic), but if I try it with the .rlib file, cargo says that it cannot find "foo" (-lfoo), the build script:
fn main() {
println!("cargo:rustc-link-search=.");
println!("cargo:rustc-link-lib=foo");
}
I tried replacing the path (search) to different directories (whilst also moving the .rlib file to the correct directory), also tried different combinations of libfoo, libfoo.rlib, ... (note that for the C libaries, foo is sufficient).
So my question really is: How can you create a rust library for private use, and how do you use it with a rust executable in a proper way, avoiding manual rustc commands? Are there tools that do this? Am I missing something in the build script? Perhaps there exists something like CMake for rust?
I suppose it is possible to just create a C interface over the rust code and compile another C project as that does work with cargo.
I do NOT want to publish the code to crates.io as I want this library strictly for private use.
Cargo does not support using pre-compiled .rlibs. Cargo is designed to compile programs fully from source (not counting native libraries).
How can you create a rust library for private use … I do NOT want to publish the code to crates.io as I want this library strictly for private use.
To use a private library, you write a dependency using a path or git dependency (or use a private package registry, but that's more work to set up).
[dependencies]
my-lib-a = { path = "../my-lib-a/" }
my-lib-b = { git = "https://my-git-host.example/my-lib-b", branch = "stable" }
Your private library is now compiled exactly like a “public” one.
I was reading this at the guessing game tutorial inside the rust book
Ensuring Reproducible Builds with the Cargo.lock File
Cargo has a mechanism that ensures you can rebuild the same artifact every time you or anyone else builds your code: Cargo will use only the versions of the dependencies you specified until you indicate otherwise. For example, say that next week version 0.8.4 of the rand crate comes out, and that version contains an important bug fix, but it also contains a regression that will break your code. To handle this, Rust creates the Cargo.lock file the first time you run cargo build, so we now have this in the guessing_game directory.
When you build a project for the first time, Cargo figures out all the versions of the dependencies that fit the criteria and then writes them to the Cargo.lock file. When you build your project in the future, Cargo will see that the Cargo.lock file exists and use the versions specified there rather than doing all the work of figuring out versions again. This lets you have a reproducible build automatically. In other words, your project will remain at 0.8.3 until you explicitly upgrade, thanks to the Cargo.lock file.
but didn't quite understand how cargo garantees that we have/need the correct version of a dependency. For instance, if we put in the [dependecies] rand="0.8.3" it will not exactly download the crate at this version, but the crate the match the needs of our program but it's compatible with this version(?)
Please clarify this logic!
When specifying a crate in Cargo.toml, you can give an exact version (e.g. =0.8.3) or more general indications (e.g. 0.8.*).
If you specify rand = "=0.8.3" in Cargo.toml, then cargo will take version 0.8.3 (note the extra = inside the version requirement). But if you specify rand = "*", then the first time you build your crate it will take the latest version and write this version to Cargo.lock. That way if you rebuild your crate later cargo will reuse the same version even if a new version has been published on crates.io in the meantime.
Note btw that specifying rand = "0.8.3" does not mean "exactly version 0.8.3", but instead means "any version >=0.8.3 and <0.9": link.
I have a Haskell library that I am developing using Stack. As I am developing the library, I like to write small test/experimentation programs that use the library. I keep a collection of these test programs for myself in a directory locally. These test modules are very quick and informal, and not appropriate to include as unit tests in the committed library code. Typically, most of them aren't even maintained and won't compile against the latest version of the library, but I keep them around in case I want to update them later. When I'm working on a test program, I want it to build against my working copy of the library, with any changes that I've made to the library locally.
How should I set up my Stack build environment for this situation? Here are some options I've tried, and the problems with each options.
Two Cabal packages, one Stack configuration. The stack.yaml file lists both packages and defines the build environment for both at once.
Problem: The stack.yaml file needs to be included as part of the committed library source code, so that other developers can build the library from source reproducibly. I don't want the public stack.yaml file for my library to include build information for my local test projects.
Problem: As far as I know, to make this work I need to have a .cabal file that lists all the executables and modules for my test programs. This is annoying to update whenever I want to throw together a quick experimental script, and will fail to build any of the test programs if I have even a single module that doesn't compile. I can't have a .cabal file with no sections, because Cabal gives "No executables, libraries,tests, or benchmarks found. Nothing to do.", and because this offers nowhere to list build-depends.
Create a Cabal sandbox for the test programs. Use cabal sandbox add-source to add the local library as a package. See also this answer.
Problem: Using Cabal sandboxes instead of Stack reintroduces a lot of the dependency problems that Stack is supposed to fix, such as using the system-global GHC instead of the GHC defined by the resolver.
Have a separate stack.yaml for the test programs. Add the library under packages as location: 'C:\Path\To\Local\Library' and set extra-dep: true for that dependency. (See here for more info on this feature.) Don't put any other Cabal packages under packages in the stack.yaml for the test programs. Use stack runghc to invoke test programs within the scope of their stack.yaml.
Problem: I just can't get this one to work. Running stack build inside the test program directory gives "Error parsing targets: The project contains no local packages (packages not marked with 'extra-dep')". Running stack runghc acts as if no dependencies are present at all. I don't want to add a Cabal package for the test programs because this has the same problem as option 1 with needing to construct an explicit .cabal file describing the modules to build.
Problem: Stack build configuration info that I want to be identical between the library and the test programs has to be copied manually. For example, if I change the resolver in my library's stack.yaml, I also need to change it in the stack.yaml for my test programs separately.
Have a directory inside my working copy of the library that contains all of my test programs. Use stack runghc to invoke test programs in the context of the library.
Problem: I'd like the directory with my test programs to be outside of the directory containing my library source code and build configuration, so that I don't have to tell the version control for my library to ignore my test code, and can have my own local version control just for the test programs.
Problem: Only works with a single local library dependency. If my test programs need to depend on local working copies of two different libraries with their own stack.yaml files, I'm out of luck.
Add a symbolic link inside my working copy of the library to a separate directory that contains all of my test programs. Navigate through the symlink and use stack runghc to invoke test programs in the context of the library.
Problem: Super awkward to use, especially since I'm on Windows and Windows has terrible symlink support.
Problem: Still need to tell my version control system to ignore the symlink.
Problem: Still only works with a single library dependency.
If only one local library is involved, I use option 4. You can put your tests outside the directory of your library, and either invoke stack from the directory of your library, or using --stack-yaml path/to/library/stack.yaml.
Otherwise, I use option 3, creating a separate stack project without setting extra-dep.
...
packages:
- 'path/to/package1'
- 'path/to/package2'
...
I can't think of a good workaround for the issue of configuration duplication. There would otherwise be conflicts if multiple packages specified different resolvers/package versions.
Edit: Actually a stub library works better, so edited to add.
I think the way to get #3 to work is -- under your scratch program directory -- (1) add . under packages in stack.yaml alongside the location/extra-dep: true package:
packages:
- '.'
- location: ../mylib
extra-dep: true
(2) create an executable clause in scratch.cabal that points to a stub main program (i.e., a "Hello World" program that compiles but need not do anything) which depends on your library:
executable main
hs-source-dirs: src
main-is: Stub.hs
build-depends: base
, mylib
default-language: Haskell2010
or a library clause with no exposed modules, again that depends on your mylib library:
library
hs-source-dirs: src
build-depends: base >= 4.7 && < 5
, mylib
default-language: Haskell2010
and (3) run stack build in the scratch directory. This should build and register mylib, and now stack runghc Prog1.hs should work fine for running programs that depend on mylib modules.
If you use the executable approach, the stub program is compiled as a side effect but otherwise ignored. If you use the library approach, it looks like the stub library isn't even built; and you then have the option of actually building a scratch library by adding some exposed modules of shared code for your test programs to use, if it's convenient, so the stub library might be best.
None of this solves the problem of keeping stack.yaml info like the resolver version in sync, but it seems to address all the problems you list in 1, 2, 4, and 5. In particular, it should work fine for test programs that depend on multiple local libraries you're developing.
It seems that the default behavior of Cargo when searching for its configuration directory is to look in the current user's home directory (~/.cargo on my system). How can this behavior be modified to make Cargo look in a user-supplied directory instead?
Environment variables Cargo reads
You can override these environment variables to change Cargo's
behavior on your system:
CARGO_HOME — Cargo maintains a local cache of the registry index and of git checkouts of crates. By default these are stored under
$HOME/.cargo, but this variable overrides the location of this
directory. Once a crate is cached it is not removed by the clean
command.
CARGO_TARGET_DIR — Location of where to place all generated artifacts, relative to the current working directory.
RUSTC — Instead of running rustc, Cargo will execute this specified compiler instead.
RUSTC_WRAPPER — Instead of simply running rustc, Cargo will execute this specified wrapper instead, passing as its commandline
arguments the rustc invocation, with the first argument being rustc.
RUSTDOC — Instead of running rustdoc, Cargo will execute this specified rustdoc instance instead.
RUSTDOCFLAGS — A space-separated list of custom flags to pass to all rustdoc invocations that Cargo performs. In contrast with
cargo rustdoc, this is useful for passing a flag to all
rustdoc instances.
RUSTFLAGS — A space-separated list of custom flags to pass to all compiler invocations that Cargo performs. In contrast with cargo rustc,
this is useful for passing a flag to all compiler
instances.
CARGO_INCREMENTAL — If this is set to 1 then Cargo will force incremental compilation to be enabled for the current compilation,
and when set to 0 it will force disabling it. If this env var isn't
present then cargo's defaults will otherwise be used.
CARGO_CACHE_RUSTC_INFO — If this is set to 0 then Cargo will not try to cache compiler version information.
The Cargo documentation
Hierarchical structure
Cargo allows local configuration for a particular package as well as
global configuration, like git. Cargo extends this to a hierarchical
strategy. If, for example, Cargo were invoked in
/projects/foo/bar/baz, then the following configuration files would
be probed for and unified in this order:
/projects/foo/bar/baz/.cargo/config
/projects/foo/bar/.cargo/config
/projects/foo/.cargo/config
/projects/.cargo/config
/.cargo/config
$HOME/.cargo/config
With this structure, you can specify configuration per-package, and
even possibly check it into version control. You can also specify
personal defaults with a configuration file in your home directory.
The Cargo documentation
The default location can be changed by means of the environment variable $CARGO_HOME, it overrides the default location which is $HOME/.cargo.
Suppose I execute cargo new one --bin and cargo new two --bin then add the same dependency to each project's Cargo.toml and build them.
Now there are two absolutely identical sets of libraries:
/one/target/debug/deps/ *.rlib
/two/target/debug/deps/ *.rlib
They are same files and waste storage space, but really the problem is that I have to compile these libraries again for every project. It takes a very much time. There is the same problem with cargo install.
Can I specify a place to store compiled libraries to avoid recompilation?
Several Cargo projects might share the libraries by using the same target dir.
.cargo/config
Place a ".cargo" folder in a project and create a "config" file there containing:
[build]
target-dir = "/path/to/your/shared/target/dir"
On Unix this might look like:
mkdir ~/shared_rust_target
mkdir .cargo
echo "[build]" > .cargo/config
echo "target-dir = \"$HOME/shared_rust_target\"" >> .cargo/config
CARGO_TARGET_DIR
Set the CARGO_TARGET_DIR environment variable.
On Unix this might look like:
export CARGO_TARGET_DIR = "$HOME/shared_rust_target"
See this commit for some extra target-dir documentation.
In particular, prior to Cargo 1.9 you shouldn't build several projects into the same target dir concurrently. (Here's more on how Cargo 1.9 supports concurrent builds).
target-dir is also mentioned in the Cargo docs.
Should work, according to this issue.
P.S. It is now also possible to achieve crate reuse with workspaces.
P.S. https://docs.rs/cargo-hakari/latest/cargo_hakari/ helps with keeping some of dependencies compatible between projects.
Even if there is a way to do it, you probably don't want to. Just because you happen to be using the same libraries doesn't mean that they were compiled the same. For example, Cargo supports the concept of features, compilation time configuration that changes how the crate was compiled.
Likewise, you may need to support multiple versions of Rust, such as nightly and stable. Or perhaps you need to cross-compile for a different architecture. Each of those will produce different code.
Cargo will cache the build products of a single project, so I've found the overhead not really noticeable, and I compile a lot of projects from people asking questions on Stack Overflow! :-)
I managed to piece together code from a few answers, but mainly this one.
This Dockerfile should not just cache Cargo dependancy downloads, but also their compilations and the crates.io index. All the other answers I could find only cached the downloads or the index, not both.
FROM arm64v8/rust as builder
# Capture dependencies
COPY Cargo.toml Cargo.lock /app/
# We create a dummy main.rs to build deps
WORKDIR /app
RUN mkdir src && echo "fn main() {}" > src/main.rs
# RUN rustup install nightly && rustup default nightly
# This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
# Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release && rm src/main.rs
# Copy our sources
COPY ./src /app/src
# A bit of magic here!
# * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
# * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
RUN --mount=type=cache,target=/usr/local/cargo/registry \
set -e && \
# update timestamps to force a new build &&
touch /app/src/main.rs && \
cargo build --release
# Again, our final image is the same - a slim base and just our app
FROM debian:buster-slim as app
COPY --from=builder /app/target/release/app/app
CMD ["/app"]
Take note of the FROM arm64v8, if you are targeting x86, replace the builder and app FROMs with their respective x86 versions.