I got an error trying this code, which realizes a simple linked list.
use std::rc::Rc;
use std::cell::RefCell;
struct Node {
a : Option<Rc<RefCell<Node>>>,
value: i32
}
impl Node {
fn new(value: i32) -> Rc<RefCell<Node>> {
let node = Node {
a: None,
value: value
};
Rc::new(RefCell::new(node))
}
}
fn main() {
let first = Node::new(0);
let mut t = first.clone();
for i in 1 .. 10_000
{
if t.borrow().a.is_none() {
t.borrow_mut().a = Some(Node::new(i));
}
if t.borrow().a.is_some() {
t = t.borrow().a.as_ref().unwrap().clone();
}
}
println!("Done!");
}
Why does it happen? Does this mean that Rust is not as safe as positioned?
UPD:
If I add this method, the program does not crash.
impl Drop for Node {
fn drop(&mut self) {
let mut children = mem::replace(&mut self.a, None);
loop {
children = match children {
Some(mut n) => mem::replace(&mut n.borrow_mut().a, None),
None => break,
}
}
}
}
But I am not sure that this is the right solution.
Does this mean that Rust is not as safe as positioned?
Rust is only safe against certain kinds of failures; specifically memory corrupting crashes, which are documented here: http://doc.rust-lang.org/reference.html#behavior-considered-undefined
Unfortunately there is a tendency to sometimes expect rust to be more robust against certain sorts of failures that are not memory corrupting. Specifically, you should read http://doc.rust-lang.org/reference.html#behavior-considered-undefined.
tldr; In rust, many things can cause a panic. A panic will cause the current thread to halt, performing shutdown operations.
This may superficially appear similar to a memory corrupting crash from other languages, but it is important to understand although it is an application failure, it is not a memory corrupting failure.
For example, you can treat panic's like exceptions by running actions in a different thread and gracefully handling failure when the thread panics (for whatever reason).
In this specific example, you're using up too much memory on the stack.
This simple example will also fail:
fn main() {
let foo:&mut [i8] = &mut [1i8; 1024 * 1024];
}
(On most rustc; depending on the stack size on that particularly implementation)
I would have thought that moving your allocations to the stack using Box::new() would fix it in this example...
use std::rc::Rc;
use std::cell::RefCell;
#[derive(Debug)]
struct Node {
a : Option<Box<Rc<RefCell<Node>>>>,
value: i32
}
impl Node {
fn new(value: i32) -> Box<Rc<RefCell<Node>>> {
let node = Node {
a: None,
value: value
};
Box::new(Rc::new(RefCell::new(node)))
}
}
fn main() {
let first = Node::new(0);
let mut t = first.clone();
for i in 1 .. 10000
{
if t.borrow().a.is_none() {
t.borrow_mut().a = Some(Node::new(i));
}
if t.borrow().a.is_some() {
let c:Box<Rc<RefCell<Node>>>;
{ c = t.borrow().a.as_ref().unwrap().clone(); }
t = c;
println!("{:?}", t);
}
}
println!("Done!");
}
...but it doesn't. I don't really understand why, but hopefully someone else can look at this and post a more authoritative answer about what exactly is causing stack exhaustion in your code.
For those who come here and are specifically interested in the case where the large struct is a contiguous chunk of memory (instead of a tree of boxes), I found this GitHub issue with further discussion, as well as a solution that worked for me:
https://github.com/rust-lang/rust/issues/53827
Vec's method into_boxed_slice() returns a Box<[T]>, and does not overflow the stack for me.
vec![-1; 3000000].into_boxed_slice()
A note of difference with the vec! macro and array expressions from the docs:
This will use clone to duplicate an expression, so one should be careful using this with types having a nonstandard Clone implementation.
There is also the with_capacity() method on Vec, which is shown in the into_boxed_slice() examples.
Related
I would like to write a simulation algorithm in Rust that has three main parts. The first part is a struct that maintains the current state of the system and associated methods to make allowed state transitions. The second part is a simulation strategy, this tells which state transition(s) to make next (e.g. I will have a slow but accurate simulation strategy and a quick but approximate strategy). Finally, I have a way to serialise the system after a simulation step was taken (e.g. one method writes the system to csv while another one to json).
I would like to choose both the simulation strategy and the serialisation method at run time.
Coming from Python, my first try for the main simulation loop looked like this:
let mut system = System { ... };
let simulator = GillespieAlgorithm { system: &mut system, ... }; // this will be dynamic eventually
let output_formatter = CSVFormatter { system: &system, ... }; // this will be dynamic eventually
output_formatter.start();
for _ in 1..100 {
simulator.step();
output_formatter.write_current_state();
}
output_formatter.stop();
Of course, this doesn't work because I want to borrow system twice with one of them being a mutable reference.
What would be the idiomatic Rust solution to this problem? My understanding is that I should somehow attach a behaviour dynamically to System instead of passing system to other structs.
You've mentioned in the comments, that you want to keep the &mut system in your simulation. Thats fine and you can still use system, as long as you're getting it via the GillespieAlgorithm. If you're fine to pass it to the formatter by method argument rather than constructor, this might be a solution for you (playground)
struct System();
struct GillespieAlgorithm<'a> { system: &'a mut System }
struct CSVFormatter();
fn main() {
let mut system = System();
let mut simulator : Box<dyn Algorithm> = Box::new(GillespieAlgorithm { system: &mut system }); // this will be dynamic eventually
let output_formatter: Box<dyn Formatter> = Box::new(CSVFormatter());
output_formatter.start();
for _ in 1..100 {
simulator.step();
output_formatter.write_current_state(simulator.borrow_system());
}
output_formatter.stop();
}
trait Algorithm {
fn step(&mut self) {}
fn borrow_system(&self) -> &System;
}
impl<'a> Algorithm for GillespieAlgorithm<'a> {
fn step(&mut self) {}
fn borrow_system(&self) -> &System {
self.system
}
}
trait Formatter {
fn start(&self);
fn write_current_state(&self, system: &System);
fn stop(&self);
}
impl Formatter for CSVFormatter {
fn start(&self) {}
fn write_current_state(&self, _system: &System) {}
fn stop(&self) {}
}
If you don't need the entire System in the CSVFormatter, you could also return a dedicated state struct in step() and pass this to your formatter.
I'm trying to create a set of structs in Rust that use a contiguous block of memory. E.g:
<------------ Memory Pool -------------->
[ Child | Child | Child | Child ]
These structs:
may each contain slices of the pool of different sizes
should allow access to their slice of the pool without any blocking operations once initialized (I intend to access them on an audio thread).
I'm very new to Rust but I'm well versed in C++ so the main hurdle thus far has been working with the ownership semantics - I'm guessing there's a trivial way to achieve this (without using unsafe) but the solution isn't clear to me. I've written a small (broken) example of what I'm trying to do:
pub struct Child<'a> {
pub slice: &'a mut [f32],
}
impl Child<'_> {
pub fn new<'a>(s: &mut [f32]) -> Child {
Child {
slice: s,
}
}
}
pub struct Parent<'a> {
memory_pool: Vec<f32>,
children: Vec<Child<'a>>,
}
impl Parent<'_> {
pub fn new<'a>() -> Parent<'a> {
const SIZE: usize = 100;
let p = vec![0f32; SIZE];
let mut p = Parent {
memory_pool: p,
children: Vec::new(),
};
// Two children using different parts of the memory pool:
let (lower_pool, upper_pool) = p.memory_pool.split_at_mut(SIZE / 2);
p.children = vec!{ Child::new(lower_pool), Child::new(upper_pool) };
return p; // ERROR - p.memory_pool is borrowed 2 lines earlier
}
}
I would prefer a solution that doesn't involve unsafe but I'm not entirely opposed to using it. Any suggestions would be very much appreciated, as would any corrections on how I'm (mis?)using Rust in my example.
Yes, it's currently impossible (or quite hard) in Rust to contain references into sibling data, for example, as you have here, a Vec and slices into that Vec as fields in the same struct. Depending on the architecture of your program, you might solve this by storing the original Vec at some higher-level of your code (for example, it could live on the stack in main() if you're not writing a library) and the slice references at some lower level in a way that the compiler can clearly infer it won't go out of scope before the Vec (doing it in main() after the Vec has been instantiated could work, for example).
This is the perfect use case for an arena allocator. There are quite a few. The following demonstration uses bumpalo:
//# bumpalo = "2.6.0"
use bumpalo::Bump;
use std::mem::size_of;
struct Child1(u32, u32);
struct Child2(f32, f32, f32);
fn main() {
let arena = Bump::new();
let c1 = arena.alloc(Child1(1, 2));
let c2 = arena.alloc(Child2(1.0, 2.0, 3.0));
let c3 = arena.alloc(Child1(10, 11));
// let's verify that they are indeed continuous in memory
let ptr1 = c1 as *mut _ as usize;
let ptr2 = c2 as *mut _ as usize;
let ptr3 = c3 as *mut _ as usize;
assert_eq!(ptr1 + size_of::<Child1>(), ptr2);
assert_eq!(ptr1 + size_of::<Child1>() + size_of::<Child2>(), ptr3);
}
There are caveats, too. The main concern is of course the alignment; there may be some padding between two consecutive allocations. It is up to you to make sure that doesn't gonna happen if it is a deal breaker.
The other is allocator specific. The bumpalo arena allocator used here, for example, doesn't drop object when itself gets deallocated.
Other than that, I do believe a higher level abstraction like this will benefit your project. Otherwise, it'll just be pointer manipulating c/c++ disguised as rust.
I've found that mem::drop does not necessary run near where it gets called, which likely results in Mutex or RwLock guards being held during expensive computations. How can I control when drop gets called?
As a simple example, I've made the following test for a zeroing drop of cryptographic material work by using unsafe { ::std::intrinsics::drop_in_place(&mut s); } instead of simply ::std::mem::drop(s).
#[derive(Debug, Default)]
pub struct Secret<T>(pub T);
impl<T> Drop for Secret<T> {
fn drop(&mut self) {
unsafe { ::std::intrinsics::volatile_set_memory::<Secret<T>>(self, 0, 1); }
}
}
#[derive(Debug, Default)]
pub struct AnotherSecret(pub [u8; 32]);
impl Drop for AnotherSecret {
fn drop(&mut self) {
unsafe { ::std::ptr::write_volatile::<$t>(self, AnotherSecret([0u8; 32])); }
assert_eq!(self.0,[0u8; 32]);
}
}
#[cfg(test)]
mod tests {
macro_rules! zeroing_drop_test {
($n:path) => {
let p : *const $n;
{
let mut s = $n([3u8; 32]); p = &s;
unsafe { ::std::intrinsics::drop_in_place(&mut s); }
}
unsafe { assert_eq!((*p).0,[0u8; 32]); }
}
}
#[test]
fn zeroing_drops() {
zeroing_drop_test!(super::Secret<[u8; 32]>);
zeroing_drop_test!(super::AnotherSecret);
}
}
This test fails if I use ::std::mem::drop(s) or even
#[inline(never)]
pub fn drop_now<T>(_x: T) { }
It's obviously fine to use drop_in_place for a test that a buffer gets zeroed, but I'd worry that calling drop_in_place on a Mutex or RwLock guard might result in use after free.
These two guards could maybe be handled with this approach :
#[inline(never)]
pub fn drop_now<T>(t: mut T) {
unsafe { ::std::intrinsics::drop_in_place(&mut t); }
unsafe { ::std::intrinsics::volatile_set_memory::<Secret<T>>(&t, 0, 1); }
}
Answer from https://github.com/rust-lang/rfcs/issues/1850 :
In debug mode, any call to ::std::mem::drop(s) physically moves s on the stack, so p points to an old copy that does not get erased. And unsafe { ::std::intrinsics::drop_in_place(&mut s); } works because it does not move s.
In general, there is no good way to either prevent LLVM from moving values around on the stack, or else to zero after moving them, so you must never put cryptographically sensitive data on the stack. Instead you must Box any sensitive data, like say
#[derive(Debug, Default)]
pub struct AnotherSecret(Box<[u8; 32]>);
impl Drop for AnotherSecret {
fn drop(&mut self) {
*self.0 = [0u8; 32];
}
}
There should be no problem with Mutex or RwLock because they can safely leave residue on the stack when they are droped.
Yes: side effects.
Optimizers in general, and LLVM in particular, operate under the as-if rule: you build a program which has a specific observable behavior, and the optimizer is given free reign to produce whatever binary it wants as long as it has the very same observable behavior.
Note that the burden of proof is on the compiler. That is, when calling an opaque function (defined in another library, for example) the compiler has to assume it may have side effects. Furthermore, side effects cannot be re-ordered, as this could change the observable behavior.
In the case of Mutex, for example, acquiring and releasing the Mutex is generally opaque for the compiler (it requires an OS call), so it is seen as a side effect. I would expect compilers not to fiddle with those.
On the other hand, your Secret is a tricky case: most of the time there is no side-effect in dropping the secret (zeroing out to-be-released memory is a dead-write, to be optimized out), which is why you need to go out of your way to ensure it occurs... by convincing the compiler that there are side effects using a volatile write.
I'm wrapping a C API which allows the caller to set/get an arbitrary pointer via function calls. In this way, the C API allows a caller to associate arbitrary data with one of the C API objects. This data is not used in any callbacks, it's just a pointer that a user can stash away and get at later.
My wrapper struct implements the Drop trait for the C object that contains this pointer. What I'd like to be able to do, but am not sure it's possible, is have the data dropped correctly if the pointer is not null when the wrapper struct drops. I'm not sure how I would recover the correct type though from a raw c_void pointer.
Two alternatives I'm thinking of are
Implement the behavior of these two calls in the wrapper. Don't make any calls to the C API.
Don't attempt to offer any kind of safer interface to these functions. Document that the pointer must be managed by the caller of the wrapper.
Is what I want to do possible? If not, is there a generally accepted practice for these kinds of situations?
A naive + fully automatic approach is NOT possible for the following reasons:
freeing memory does not call drop/deconstructors/...: the C API can be used from languages which can have objects which should be deconstructed properly, e.g. C++ or Rust itself. So when you only store a memory pointer you do not know you to call the proper function (you neither know which function not how the calling conventions look like).
which memory allocator?: memory allocation and deallocation isn't a trivial thing. your program needs to request memory from the OS and then manage this resources in an intelligent way to be efficient and correct. This is usually done by a library. In case of Rust, jemalloc is used (but can be changed). So even when you ask the API caller to only pass Plain Old Data (which should be easier to destruct) you still don't know which library function to call to deallocate memory. Just using libc::free won't work (it can but it could horrible fail).
Solutions:
dealloc callback: you can ask the API user to set an additional pointer to, let's say a void destruct(void* ptr) function. If this one is not NULL, you call that function during your drop. You could also use int as an return type to signal when the destruction went wrong. In that case you could for example panic!.
global callback: let's assume you requested your user to only pass POD (plain old data). To know which free function of the memory allocator to call, you could request the user to register a global void (*free)(void* ptr) pointer which is called during drop. You could also make that one optional.
Although I was able to follow the advice in this thread, I wasn't entirely satisfied with my results, so I asked the question on the Rust forums and found the answer I was really looking for. (play)
use std::any::Any;
static mut foreign_ptr: *mut () = 0 as *mut ();
unsafe fn api_set_fp(ptr: *mut ()) {
foreign_ptr = ptr;
}
unsafe fn api_get_fp() -> *mut() {
foreign_ptr
}
struct ApiWrapper {}
impl ApiWrapper {
fn set_foreign<T: Any>(&mut self, value: Box<T>) {
self.free_foreign();
unsafe {
let raw = Box::into_raw(Box::new(value as Box<Any>));
api_set_fp(raw as *mut ());
}
}
fn get_foreign_ref<T: Any>(&self) -> Option<&T> {
unsafe {
let raw = api_get_fp() as *const Box<Any>;
if !raw.is_null() {
let b: &Box<Any> = &*raw;
b.downcast_ref()
} else {
None
}
}
}
fn get_foreign_mut<T: Any>(&mut self) -> Option<&mut T> {
unsafe {
let raw = api_get_fp() as *mut Box<Any>;
if !raw.is_null() {
let b: &mut Box<Any> = &mut *raw;
b.downcast_mut()
} else {
None
}
}
}
fn free_foreign(&mut self) {
unsafe {
let raw = api_get_fp() as *mut Box<Any>;
if !raw.is_null() {
Box::from_raw(raw);
}
}
}
}
impl Drop for ApiWrapper {
fn drop(&mut self) {
self.free_foreign();
}
}
struct MyData {
i: i32,
}
impl Drop for MyData {
fn drop(&mut self) {
println!("Dropping MyData with value {}", self.i);
}
}
fn main() {
let p1 = Box::new(MyData {i: 1});
let mut api = ApiWrapper{};
api.set_foreign(p1);
{
let p2 = api.get_foreign_ref::<MyData>().unwrap();
println!("i is {}", p2.i);
}
api.set_foreign(Box::new("Hello!"));
{
let p3 = api.get_foreign_ref::<&'static str>().unwrap();
println!("payload is {}", p3);
}
}
I have a struct:
struct MyData {
x: i32
}
I want to asynchronously start a long operation on this struct.
My first attempt was this:
fn foo(&self) { //should return immediately
std::thread::Thread::spawn(move || {
println!("{:?}",self.x); //consider a very long operation
});
}
Clearly the compiler cannot infer an appropriate lifetime due to conflicting requirements because self may be on the stack frame and thus cannot be guaranteed to exist by the time the operation is running on a different stack frame.
To solve this, I attempted to make a copy of self and provide that copy to the new thread:
fn foo(&self) { //should return immediately
let clone = self.clone();
std::thread::Thread::spawn(move || {
println!("{:?}",clone.x); //consider a very long operation
});
}
I think that does not compile because now clone is on the stack frame which is similar to before. I also tried to do the clone inside the thread, and that does not compile either, I think for similar reasons.
Then I decided maybe I could use a channel to push the copied data into the thread, on the theory that perhaps channel can magically move (copy?) stack-allocated data between threads, which is suggested by this example in the documentation. However the compiler cannot infer a lifetime for this either:
fn foo(&self) { //should return immediately
let (tx, rx) = std::sync::mpsc::channel();
tx.send(self.clone());
std::thread::Thread::spawn(move || {
println!("{:?}",rx.recv().unwrap().x); //consider a very long operation
});
}
Finally, I decided to just copy my struct onto the heap explicitly, and pass an Arc into the thread. But not even here can the compiler figure out a lifetime:
fn foo(&self) { //should return immediately
let arc = std::sync::Arc::new(self.clone());
std::thread::Thread::spawn(move || {
println!("{:?}",arc.clone().x); //consider a very long operation
});
}
Okay borrow checker, I give up. How do I get a copy of self onto my new thread?
I think your issue is simply because your structure does not derive the Clone trait. You can get your second example to compile and run by adding a #[derive(Clone)] before your struct's definition.
What I don't understand in the compiler behaviour here is what .clone() function it tried to use here. Your structure indeed did not implement the Clone trait so should not by default have a .clone() function.
playpen
You may also want to consider in your function taking self by value, and let your caller decide whether it should make a clone, or just a move.
As an alternative solution, you could use thread::scoped and maintain a handle to the thread. This allows the thread to hold a reference, without the need to copy it in:
#![feature(old_io,std_misc)]
use std::thread::{self,JoinGuard};
use std::old_io::timer;
use std::time::duration::Duration;
struct MyData {
x: i32,
}
// returns immediately
impl MyData {
fn foo(&self) -> JoinGuard<()> {
thread::scoped(move || {
timer::sleep(Duration::milliseconds(300));
println!("{:?}", self.x); //consider a very long operation
timer::sleep(Duration::milliseconds(300));
})
}
}
fn main() {
let d = MyData { x: 42 };
let _thread = d.foo();
println!("I'm so fast!");
}