This question already has answers here:
How to store rusqlite Connection and Statement objects in the same struct in Rust? [duplicate]
(1 answer)
Why can't I store a value and a reference to that value in the same struct?
(4 answers)
Closed 3 years ago.
I am trying to provide a thin abstraction over a database in Rust. Here's a simplified version of the underlying database API as I understand it:
// Connection to a database
struct Connection {}
impl Connection {
fn new() -> Connection {
Connection {}
}
}
// Prepared SQL statement
struct Statement<'a> {
conn: &'a Connection,
}
impl Connection {
// Create a prepared statement
fn prepare(&self) -> Statement {
return Statement { conn: self };
}
}
I want to provide a wrapper which upon construction connects to a database and stores some prepared statements. After constructing the wrapper I don't actually need the raw connection any more.
Here's a simplified version of my code:
struct Wrapper<'a> {
stmt: Statement<'a>,
}
impl<'a> Wrapper<'a> {
fn new() -> Wrapper<'a> {
let conn = Connection::new();
Wrapper {
stmt: conn.prepare(),
}
}
}
fn main() {
let _w = Wrapper::new();
}
Wrapper::new fails to compile because the returned value references conn whose lifetime ends at the end of the function ("error[E0515]: cannot return value referencing local variable conn").
The simplest solution is just to require the caller to construct a Connection; but then this is a rather leaky "wrapper" for the database.
Ideally I'd like to move conn out of the function. Since I'm moving the Wrapper out of the function, I thought I could just move conn into the Wrapper:
struct Wrapper<'a> {
conn: Connection,
stmt: Statement<'a>,
}
impl<'a> Wrapper<'a> {
fn new() -> Wrapper<'a> {
let conn = Connection {};
Wrapper {
stmt: conn.prepare(),
conn,
}
}
}
But this just adds another error: "cannot move out of conn because it is borrowed", even if I try to dodge the chicken-and-egg initialisation of the .stmt and .conn fields with Option & mutation hackery. I've read that you can't really have one struct member reference another in rust, which would explain this.
For a while I thought I could use an Rc, but for this to work I think it would need to be on the Statement's reference to the Connection; which is in library code.
Can anyone see which Rust design pattern I'm missing?
Related
I am having a hard time figuring out how to sort out this issue.
So I have a class ArcWorker holding a shared reference to Worker (as you can remark below).
I wrote a function in ArcWorker called join() in which the line self.internal.lock().unwrap().join(); fails with the following error:
cannot move out of dereference of std::sync::MutexGuard<'_, models::worker::Worker>
What I attempt through that line is to lock the mutex, unwrap and call the join() function from the Worker class.
As far as I understand, once that the lock function is called and it borrows a reference to self (&self), then I need some way to get to pass self by value to join (std::thread's join function requires passing self by value).
What can I do to make this work? Tried to find an answer to my question for hours but to no avail.
pub struct Worker {
accounts: Vec<Arc<Mutex<Account>>>,
thread_join_handle: Option<thread::JoinHandle<()>>
}
pub struct ArcWorker {
internal: Arc<Mutex<Worker>>
}
impl ArcWorker {
pub fn new(accounts: Vec<Arc<Mutex<Account>>>) -> ArcWorker {
return ArcWorker {
internal: Arc::new(Mutex::new(Worker {
accounts: accounts,
thread_join_handle: None
}))
}
}
pub fn spawn(&self) {
let local_self_1 = self.internal.clone();
self.internal.lock().unwrap().thread_join_handle = Some(thread::spawn(move || {
println!("Spawn worker");
local_self_1.lock().unwrap().perform_random_transactions();
}));
}
pub fn join(&self) {
self.internal.lock().unwrap().join();
}
}
impl Worker {
fn join(self) {
if let Some(thread_join_handle) = self.thread_join_handle {
thread_join_handle.join().expect("Couldn't join the associated threads.")
}
}
fn perform_random_transactions(&self) {
}
}
Since you already hold JoinHandle in an option, you can make Worker::join() take &mut self instead of self and change the if let condition to:
// note added `.take()`
if let Some(thread_join_handle) = self.thread_join_handle.take() {
Option::take() will move the handle out of the option and give you ownership over it, while leaving None in self.thread_join_handle. With this change ArcWorker::join() should compile as-is.
I have an object that I know that is inside an Arc because all the instances are always Arced. I would like to be able to pass a cloned Arc of myself in a function call. The thing I am calling will call me back later on other threads.
In C++, there is a standard mixin called enable_shared_from_this. It enables me to do exactly this
class Bus : public std::enable_shared_from_this<Bus>
{
....
void SetupDevice(Device device,...)
{
device->Attach(shared_from_this());
}
}
If this object is not under shared_ptr management (the closest C++ has to Arc) then this will fail at run time.
I cannot find an equivalent.
EDIT:
Here is an example of why its needed. I have a timerqueue library. It allows a client to request an arbitrary closure to be run at some point in the future. The code is run on a dedicated thread. To use it you must pass a closure of the function you want to be executed later.
use std::time::{Duration, Instant};
use timerqueue::*;
use parking_lot::Mutex;
use std::sync::{Arc,Weak};
use std::ops::{DerefMut};
// inline me keeper cos not on github
pub struct MeKeeper<T> {
them: Mutex<Weak<T>>,
}
impl<T> MeKeeper<T> {
pub fn new() -> Self {
Self {
them: Mutex::new(Weak::new()),
}
}
pub fn save(&self, arc: &Arc<T>) {
*self.them.lock().deref_mut() = Arc::downgrade(arc);
}
pub fn get(&self) -> Arc<T> {
match self.them.lock().upgrade() {
Some(arc) => return arc,
None => unreachable!(),
}
}
}
// -----------------------------------
struct Test {
data:String,
me: MeKeeper<Self>,
}
impl Test {
pub fn new() -> Arc<Test>{
let arc = Arc::new(Self {
me: MeKeeper::new(),
data: "Yo".to_string()
});
arc.me.save(&arc);
arc
}
fn task(&self) {
println!("{}", self.data);
}
// in real use case the TQ and a ton of other status data is passed in the new call for Test
// to keep things simple here the 'container' passes tq as an arg
pub fn do_stuff(&self, tq: &TimerQueue) {
// stuff includes a async task that must be done in 1 second
//.....
let me = self.me.get().clone();
tq.queue(
Box::new(move || me.task()),
"x".to_string(),
Instant::now() + Duration::from_millis(1000),
);
}
}
fn main() {
// in real case (PDP11 emulator) there is a Bus class owning tons of objects thats
// alive for the whole duration
let tq = Arc::new(TimerQueue::new());
let test = Test::new();
test.do_stuff(&*tq);
// just to keep everything alive while we wait
let mut input = String::new();
std::io::stdin().read_line(&mut input).unwrap();
}
cargo toml
[package]
name = "tqclient"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
timerqueue = { git = "https://github.com/pm100/timerqueue.git" }
parking_lot = "0.11"
There is no way to go from a &self to the Arc that self is stored in. This is because:
Rust references have additional assumptions compared to C++ references that would make such a conversion undefined behavior.
Rust's implementation of Arc does not even expose the information necessary to determine whether self is stored in an Arc or not.
Luckily, there is an alternative approach. Instead of creating a &self to the value inside the Arc, and passing that to the method, pass the Arc directly to the method that needs to access it. You can do that like this:
use std::sync::Arc;
struct Shared {
field: String,
}
impl Shared {
fn print_field(self: Arc<Self>) {
let clone: Arc<Shared> = self.clone();
println!("{}", clone.field);
}
}
Then the print_field function can only be called on an Shared encapsulated in an Arc.
having found that I needed this three times in recent days I decided to stop trying to come up with other designs. Maybe poor data design as far as rust is concerned but I needed it.
Works by changing the new function of the types using it to return an Arc rather than a raw self. All my objects are arced anyway, before they were arced by the caller, now its forced.
mini util library called mekeeper
use parking_lot::Mutex;
use std::sync::{Arc,Weak};
use std::ops::{DerefMut};
pub struct MeKeeper<T> {
them: Mutex<Weak<T>>,
}
impl<T> MeKeeper<T> {
pub fn new() -> Self {
Self {
them: Mutex::new(Weak::new()),
}
}
pub fn save(&self, arc: &Arc<T>) {
*self.them.lock().deref_mut() = Arc::downgrade(arc);
}
pub fn get(&self) -> Arc<T> {
match self.them.lock().upgrade() {
Some(arc) => return arc,
None => unreachable!(),
}
}
}
to use it
pub struct Test {
me: MeKeeper<Self>,
foo:i8,
}
impl Test {
pub fn new() -> Arc<Self> {
let arc = Arc::new(Test {
me: MeKeeper::new(),
foo:42
});
arc.me.save(&arc);
arc
}
}
now when an instance of Test wants to call a function that requires it to pass in an Arc it does:
fn nargle(){
let me = me.get();
Ooddle::fertang(me,42);// fertang needs an Arc<T>
}
the weak use is what the shared_from_this does so as to prevent refcount deadlocks, I stole that idea.
The unreachable path is safe because the only place that can call MeKeeper::get is the instance of T (Test here) that owns it and that call can only happen if the T instance is alive. Hence no none return from weak::upgrade
This question already has answers here:
Why can't I store a value and a reference to that value in the same struct?
(4 answers)
Closed 2 years ago.
I'm having troubles with a managed State with Rocket. This state holds a Database connection and a collection of Cursors on that Database. Each one of theses have a reference on the Database.
One of the operations on that state require to create a new cursor on the Database and keep it for later use. Unfortunatly, I am stuck with a lifetime problem. Usually, I have no problem dealing with thoses, but right now, I'm out of ideas...
I have recreated the problem bellow in a short example.
#![feature(proc_macro_hygiene, decl_macro)]
#[macro_use]
extern crate rocket;
use rocket::State;
struct Database;
impl Database {
fn create_cursor(&self) -> Cursor {
Cursor { database: self }
}
}
struct Cursor<'a> {
database: &'a Database
}
struct Controller<'a> {
database: Database,
cursors: Vec<Cursor<'a>>,
}
impl<'a> Controller<'a> {
fn do_work(&'a mut self) {
let cursor = self.database.create_cursor();
self.cursors.push(cursor)
}
}
fn main() {
let database = Database;
let controller = Controller { database, cursors: Vec::new() };
rocket::ignite()
.manage(controller)
.launch();
}
#[get("/")]
pub fn do_work_route(
mut controller: State<'static, Controller<'static>>
) -> &'static str {
controller.do_work();
"Something..."
}
error[E0621]: explicit lifetime required in the type of `__req`
--> src/main.rs:42:9
|
40 | #[get("/")]
| ----------- help: add explicit lifetime `'static` to the type of `__req`: `&'_b rocket::Request<'static>`
41 | pub fn do_work_route(
42 | mut controller: State<'static, Controller<'static>>
| ^^^^^^^^^^ lifetime `'static` required
Any lead would be appreciated. Meanwhile, I'll continue digging.
Thanks a lot!
Your Controller struct as written is self-referential, which is not possible: https://users.rust-lang.org/t/how-to-write-software-without-self-referential-structs/13819
The reason is that when a Controller gets moved, the references to the database in its cursors member would become invalid, because the memory location of its database member changed.
The best way forward is probably to step back and think about a design that is not self-referential. A possible solution is to make the database a static variable, then your cursors could store &'static Database references.
If that fails, the link above mentions the rental crate, but it doesn't seem to be easy to use.
I'm trying to design a struct to carry around a Postgres connection, transaction, and a bunch of prepared statements, and then execute the prepared statements repeatedly. But I'm running into lifetime problems. Here is what I've got:
extern crate postgres;
use postgres::{Connection, TlsMode};
use postgres::transaction::Transaction;
use postgres::stmt::Statement;
pub struct Db<'a> {
conn: Connection,
tx: Transaction<'a>,
insert_user: Statement<'a>,
}
fn make_db(url: &str) -> Db {
let conn = Connection::connect(url, TlsMode::None).unwrap();
let tx = conn.transaction().unwrap();
let insert_user = tx.prepare("INSERT INTO users VALUES ($1)").unwrap();
Db {
conn: conn,
tx: tx,
insert_user: insert_user,
}
}
pub fn main() {
let db = make_db("postgres://paul#localhost/t");
for u in &["foo", "bar"] {
db.insert_user.execute(&[&u]);
}
db.tx.commit().unwrap();
}
Here is the error I'm getting (on Rust 1.15.0 stable):
error: `conn` does not live long enough
--> src/main.rs:15:14
|
15 | let tx = conn.transaction().unwrap();
| ^^^^ does not live long enough
...
22 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the body at 13:28...
--> src/main.rs:13:29
|
13 | fn make_db(url: &str) -> Db {
| ^
I've read the Rust book (I've lost count how many times), but I'm not sure how to make progress here. Any suggestions?
EDIT: Thinking about this some more I still don't understand why in principle I can't tell Rust, "conn lives as long as Db does". The issue is with moving conn, but what if I don't move it? I understand why in C you can't return a pointer to stack-allocated memory, e.g.:
#include <stdio.h>
int *build_array() {
int ar[] = {1,2,3};
return ar;
}
int main() {
int *ar = build_array();
printf("%d\n", ar[1]);
}
And I get how that is similar to in Rust returning a &str or returning a vec slice.
But in Rust you can do this:
#[derive(Debug)]
struct S {
ar: Vec<i32>,
}
fn build_array() -> S {
let v = vec![1, 2, 3];
S { ar: v }
}
fn main() {
let s = build_array();
println!("{:?}", s);
}
And my understanding is that Rust is smart enough so that returning S doesn't actually require a move; essentially it is going straight to the caller's stack frame.
So I don't understand why it can't also put Db (including conn) in the caller's stack frame. Then no moves would be required, and tx would never hold an invalid address. I feel like Rust should be able to figure that out. I tried adding a lifetime hint, like this:
pub struct Db<'a> {
conn: Connection<'a>,
tx: Transaction<'a>,
insert_user: Statement<'a>,
}
But that gives an "unexpected lifetime parameter" error. I can accept that Rust can't follow the logic, but I'm curious if there is a reason why in principle it couldn't.
It does seem that putting conn on the heap should solve my problems, but I can't get this to work either:
pub struct Db<'a> {
conn: Box<Connection>,
tx: Transaction<'a>,
insert_user: Statement<'a>,
}
Even with a let conn = Box::new(Connection::connect(...));, Rust still tells me "conn does not live long enough". Is there some way to make this work with Box, or is that a dead end?
EDIT 2: I tried doing this with macros also, to avoid any extra stack frames:
extern crate postgres;
use postgres::{Connection, TlsMode};
use postgres::transaction::Transaction;
use postgres::stmt::Statement;
pub struct Db<'a> {
conn: Connection,
tx: Transaction<'a>,
insert_user: Statement<'a>,
}
macro_rules! make_db {
( $x:expr ) => {
{
let conn = Connection::connect($x, TlsMode::None).unwrap();
let tx = conn.transaction().unwrap();
let insert_user = tx.prepare("INSERT INTO users VALUES ($1)").unwrap();
Db {
conn: conn,
tx: tx,
insert_user: insert_user,
}
}
}
}
pub fn main() {
let db = make_db!("postgres://paul#localhost/t");
for u in &["foo", "bar"] {
db.insert_user.execute(&[&u]);
}
db.tx.commit().unwrap();
}
But that still tells me that conn does not live long enough. It seems that moving it into the struct should really not require any real RAM changes, but Rust still won't let me do it.
Starting with this function:
fn make_db(url: &str) -> Db {
unimplemented!()
}
Due to lifetime elision, this is equivalent to:
fn make_db<'a>(url: &'a str) -> Db<'a> {
unimplemented!()
}
That is, the lifetimes of all the references inside the Db struct must live as long as the string slice passed in. That only makes sense if the struct is holding on to the string slice.
To "solve" that, we can try to separate the lifetimes:
fn make_db<'a, 'b>(url: &'a str) -> Db<'b> {
unimplemented!()
}
Now this makes even less sense because now we are just making up a lifetime. Where is that 'b coming from? What happens if the caller of make_db decides that the concrete lifetime for the generic lifetime parameter 'b should be 'static? This is further explained in Why can't I store a value and a reference to that value in the same struct?, search for "something is really wrong with our creation function".
We also see the part of the question with "Sometimes, I'm not even taking a reference of the value" in the other question, which says in the answer:
the Child instance contains a reference to the Parent that created it,
If we check out the definition for Connection::transaction:
fn transaction<'a>(&'a self) -> Result<Transaction<'a>>
or the definition if you don't believe the docs:
pub struct Transaction<'conn> {
conn: &'conn Connection,
depth: u32,
savepoint_name: Option<String>,
commit: Cell<bool>,
finished: bool,
}
Yup, a Transaction keeps a reference to its parent Connection. Now that we see that Transaction has a reference to Connection we can return to the other question to see how to solve the problem: split apart the structs so that the nesting mirrors the lifetimes.
This was a very long-winded way of saying: no, you cannot create a single structure that contains a database and a transaction of that database due to the implementation of the postgres crate. Presumably the crate is implemented in this fashion for maximum performance.
I don't see why [returning Db<'b>] makes less sense. Normally when a function returns a thing, the thing lives as long as it is assigned to something. Why can't -> Db work the same way?
The entire point of references is that you don't own the referred-to value. You return Db and the caller of make_db would own that, but what owns the thing that Db is referring to? Where did it come from? You cannot return a reference to something local as that would violate all of Rust's safety rules. If you want to transfer ownership, you just do that.
See also
Is there any way to return a reference to a variable created in a function?
Return local String as a slice (&str)
Using the other answer, I put together working code that lets me bundle up the transaction and all the prepared statements, and pass them around together:
extern crate postgres;
use postgres::{Connection, TlsMode};
use postgres::transaction::Transaction;
use postgres::stmt::Statement;
pub struct Db<'a> {
tx: Transaction<'a>,
insert_user: Statement<'a>,
}
fn make_db(conn: &Connection) -> Db {
let tx = conn.transaction().unwrap();
let insert_user = tx.prepare("INSERT INTO users VALUES ($1)").unwrap();
Db {
tx: tx,
insert_user: insert_user,
}
}
pub fn main() {
let conn = Connection::connect("postgres://paul#localhost/t", TlsMode::None).unwrap();
let db = make_db(&conn);
for u in &["foo", "bar"] {
db.insert_user.execute(&[&u]);
}
db.tx.commit().unwrap();
}
As I understand it, Rust wants to guarantee that conn lives as long as db, so by keeping conn outside of the "constructor", the lexical structure ensures that it won't get removed too early.
My struct still doesn't encapsulate conn, which seems too bad to me, but at least it lets me keep everything else together.
When creating a struct in Rust it seems like it's difficult to create one without having all of the fields set. For example with the following code
struct Connection {
url: String,
stream: TcpStream
}
You aren't able to set url without giving stream as well.
// Compilation error asking for 'stream'
let m = Connection { url: "www.google.com".to_string() };
How are you able to create these references that might be Option<None> until a later time?
The best I have found is using the Default trait, but I'd rather not have to create the TcpStream until a later time than when the struct is initialised. Am I able to do this with something like a Box?
One thing you can do is to wrap the TcpStream in an Option, i.e. Option<TcpStream>. When you first construct the struct, it'll be None, and when you initialize it you make it self.stream = Some(<initialize tcp stream>). Wherever you use the TCPStream, you'll have to check if it's Some, i.e. if it has already been initialized. If you can guarantee your behavior then you can just unwrap(), but it's probably better to make a check anyways.
struct Connection {
url: String,
stream: Option<TcpStream>
}
impl Connection {
pub fn new() -> Connection {
Connection {
url: "www.google.com".to_string(),
stream: None,
}
}
pub fn initialize_stream(&mut self) {
self.stream = Some(TcpStream::connect("127.0.0.1:34254").unwrap());
}
pub fn method_that_uses_stream(&self) {
if let Some(ref stream) = self.stream {
// can use the stream here
} else {
println!("the stream hasn't been initialized yet");
}
}
}
This is similar to what is done in Swift, in case you're familiar with that language.
All fields indeed have to be initialized when creating the struct instance (there is no null in Rust) so all the memory is allocated.
There is often a dedicated method (like new) that sets default values for fields which are supposed to be modified at a later stage.
I'd use the Box when you don't know the size of the field (like Vec does).
As an extension to Jorge Israel Peña's answer, you can use a builder. The builder has all the optional fields and produces the final value without Options:
use std::net::TcpStream;
struct ConnectionBuilder {
url: String,
stream: Option<TcpStream>,
}
impl ConnectionBuilder {
fn new(url: impl Into<String>) -> Self {
Self {
url: url.into(),
stream: None,
}
}
fn stream(mut self, stream: TcpStream) -> Self {
self.stream = Some(stream);
self
}
fn build(self) -> Connection {
let url = self.url;
let stream = self
.stream
.expect("Perform actual error handling or default value");
Connection { url, stream }
}
}
struct Connection {
url: String,
stream: TcpStream,
}
impl Connection {
fn method_that_uses_stream(&self) {
// can use self.stream here
}
}
This means that you don't have to litter your code with checks to see if the stream has been set yet.
See also:
How to initialize a struct with a series of arguments
Do Rust builder patterns have to use redundant struct code?
Is it possible to create a macro to implement builder pattern methods?
How to write an idiomatic build pattern with chained method calls in Rust?