I'm working on a project around embedded systems that needs to use filesystem. I want to use liitlefs crate in rust but it uses mem::uninitialized that deprecated since 1.39.0 .
https://github.com/brandonedens/rust-littlefs
pub struct LittleFs<T: Storage> {
storage: T,
lfs_config: lfs::lfs_config,
lfs: lfs::lfs_t,
read_buffer: [u8; READ_SIZE],
prog_buffer: [u8; PROG_SIZE],
lookahead_buffer: [u8; LOOKAHEAD / 8],
}
impl<T: Storage> LittleFs<T> {
/// Create a new instance of the LittleFS.
pub fn new(storage: T) -> Self {
LittleFs {
storage: storage,
lfs: unsafe { mem::uninitialized::<lfs::lfs>() },
lfs_config: unsafe { mem::uninitialized::<lfs::lfs_config>() },
read_buffer: [0u8; READ_SIZE],
prog_buffer: [0u8; PROG_SIZE],
lookahead_buffer: [0u8; LOOKAHEAD / 8],
}
}
I tried to replace it in the following way but it caused unspecified behaviors by the system:
pub struct LittleFs<T: Storage> {
storage: T,
lfs_config: lfs::lfs_config,
lfs: lfs::lfs_t,
read_buffer: [u8; READ_SIZE],
prog_buffer: [u8; PROG_SIZE],
lookahead_buffer: [u8; LOOKAHEAD / 8],
}
impl<T: Storage> LittleFs<T> {
pub fn new(storage: T) -> Self {
LittleFs {
storage,
lfs: unsafe { MaybeUninit::uninit().assume_init() },
lfs_config: unsafe { MaybeUninit::uninit().assume_init() },
read_buffer: [0u8; READ_SIZE],
prog_buffer: [0u8; PROG_SIZE],
lookahead_buffer: [0u8; LOOKAHEAD / 8],
}
}
Does anyone have a solution to this problem? Thank you in advance.
You can't do uninit().assume_init() — it's definitely not initialized. assume_init() is only to be called once the data has been initialized with real values. The idea behind calling it is that you're promising, "Yes, I know this value is marked as maybe initialized. I promise that it is initialized now. It's not just a maybe."
You'll need to change the types of lfs and lfs_config:
lfs_config: MaybeUninit<lfs::lfs_config>,
lfs: MaybeUninit<lfs::lfs_t>,
That will have a ripple effect downstream, requiring you to then modify any code that accesses the fields.
Initialize them with write.
Read them with assume_init_ref or assume_init_mut.
Related
I try to get the following code to compile.
Due to the buffer it wanted more explicit lifetimes inserted. So I tried that but still
cannot figure out yet how to make it also work in for_each.
use std::{sync::mpsc::Sender, ffi::OsStr};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher<'a> {
buffer: [u8; 1024],
sender: Sender<Event<&'a OsStr>>,
}
impl <'a> Watcher<'a> {
pub fn new(sender: Sender<Event<&OsStr>>) -> Watcher{
Watcher{
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
events.filter(|e| -> bool { !e.mask.intersects(EventMask::ISDIR) }).for_each(|e| sender.send(e).unwrap());
}
}
EDIT:
I ended up giving up on no-copy of strings:
use std::{ffi::OsStr, sync::mpsc::Sender};
use inotify::{Event, EventMask, Inotify, WatchMask};
pub struct Watcher {
buffer: [u8; 1024],
sender: Sender<Event<String>>,
}
impl Watcher {
pub fn new(sender: Sender<Event<String>>) -> Watcher {
Watcher {
buffer: [0; 1024],
sender: sender,
}
}
pub fn watch_for_led_directories(&mut self, dir: String) {
let sender = self.sender.clone();
let mut inotify = Inotify::init().expect("Error while initializing inotify instance");
// Watch for modify and close events.
inotify
.add_watch(dir, WatchMask::CREATE | WatchMask::DELETE)
.expect("Failed to add file watch");
// Read events that were added with `add_watch` above.
let events = inotify
.read_events_blocking(&mut self.buffer)
.expect("Error while reading events");
let filter = |e: &Event<&OsStr>| -> bool {
!e.mask.intersects(EventMask::ISDIR) || e.name.is_none()
};
events.filter(filter).for_each(|e| {
sender
.send(Event {
cookie: e.cookie,
mask: e.mask,
name: Some(e.name.unwrap().to_str().expect("Expected valid unicode string").into()),
wd: e.wd,
})
.unwrap()
});
}
}
The issue is that while self has the lifetime of 'a, &mut self in watch_for_led_directories() has a different lifetime.
inotify.read_events_blocking() returns events that has the same lifetime as &mut self.buffer, which, as a mutable borrow, has the lifetime of &mut self. But sender is bound to life time 'a. Therefore compiler cannot reconcile these two lifetimes.
One way to solve it is to provide an external buffer, instead of having it owned by Watcher, e.g.:
pub fn watch_for_led_directories(&self, buffer: &'a mut [u8], dir: String) {
...
let events: Events = inotify
.read_events_blocking(buffer)
.expect("Error while reading events");
...
Compiler derives the lifetime of events from buffer and therefore it will be 'a.
To use this method the externally allocated buffer must live at least as long as the Watcher object, which should not be too difficult to arrange.
A Rust newbie here, attempting to write a webservice by combining
https://github.com/seanmonstar/warp/blob/master/examples/todos.rs and https://github.com/launchbadge/sqlx/blob/master/examples/postgres/todos/src/main.rs
The following code is in running state. My question is, do I need to clone dbpool for every handler? What's the idiomatic way in Rust (I am coming from Java/Kotlin->Go background, FWIW)
#![deny(warnings)]
use sqlx::postgres::{PgPoolOptions};
use std::env;
use warp::Filter;
#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
let pool = PgPoolOptions::new()
.max_connections(5)
.connect("postgres://:#localhost/todo_db").await?;
if env::var_os("RUST_LOG").is_none() {
env::set_var("RUST_LOG", "todos=info");
}
pretty_env_logger::init();
let api = filters::todos(pool);
let routes = api.with(warp::log("todos"));
// Start up the server...
warp::serve(routes).run(([127, 0, 0, 1], 3030)).await;
Ok(())
}
mod filters {
use sqlx::{Pool, Postgres};
use super::handlers;
use super::models::{ListOptions, Todo};
use warp::Filter;
pub fn todos(
db: Pool<Postgres>,
) -> impl Filter<Extract=impl warp::Reply, Error=warp::Rejection> + Clone {
todos_list(db)
}
/// GET /todos?offset=3&limit=5
pub fn todos_list(
db: Pool<Postgres>,
) -> impl Filter<Extract=impl warp::Reply, Error=warp::Rejection> + Clone {
warp::path!("todos")
.and(warp::get())
.and(warp::query::<ListOptions>())
.and(with_db(db))
.and_then(handlers::list_todos)
}
fn with_db(db: Pool<Postgres>) -> impl Filter<Extract=(Pool<Postgres>, ), Error=std::convert::Infallible> + Clone {
warp::any().map(move || db.clone())
}
fn _json_body() -> impl Filter<Extract=(Todo, ), Error=warp::Rejection> + Clone {
warp::body::content_length_limit(1024 * 16).and(warp::body::json())
}
}
mod handlers {
use super::models::{ListOptions};
use std::convert::Infallible;
use sqlx::{Pool, Postgres};
use crate::models::Todo;
pub async fn list_todos(_opts: ListOptions, db: Pool<Postgres>) -> Result<impl warp::Reply, Infallible> {
let recs = sqlx::query!(
r#"
SELECT id, description, done
FROM todos
ORDER BY id
"#
)
.fetch_all(&db).await.expect("Some error message");
let x: Vec<Todo> = recs.iter().map(|rec| {
Todo { id: rec.id, text: rec.description.clone(), completed: rec.done }
}).collect();
Ok(warp::reply::json(&x))
}
}
mod models {
use serde_derive::{Deserialize, Serialize};
#[derive(Debug, Deserialize, Serialize)]
pub struct Todo {
pub id: i64,
pub text: String,
pub completed: bool,
}
// The query parameters for list_todos.
#[derive(Debug, Deserialize)]
pub struct ListOptions {
pub offset: Option<usize>,
pub limit: Option<usize>,
}
}
While copying a pool only increase the reference counter in an Arc and is relatively cheap, as #cdhowie points out, you can avoid it if you fancy doing so: .fetch_all(db) only needs an immutable reference. You could thus pass in a &'static Pool<…>. The one tricky thing is: You can't directly declare a
static POOL: Pool<Postgres> = …;
because there's nothing you could put for the …. You can only use const fn when initializing statics, and you can't use .await.
Instead, you can use a OnceCell. Multiple variants exist, the one included in tokio is probably most convenient here:
static POOL: OnceCell<Pool<Postgres>> = OnceCell::const_new();
#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
POOL.get_or_try_init(|| async {
PgPoolOptions::new()
.max_connections(5)
.connect("postgres://:#localhost/todo_db")
.await
})
.await?;
// Later, just access your pool with POOL.get().unwrap()
// You don't need the with_db filter anymore
Personally though, I prefer creating connections or pools that live as long as the application itself with Box::leak(Box::new(PgPoolOptions()….await?)). If you think this is bad because it (obviously…) leaks memory, consider this: the OnceCell will never be dropped or free'd either. This also means that neither OnceCell nor Box::leak will allow for a clean shutdown of the connection pool, which your code with the internal Arcs could in theory do.
I am currently experimenting with the FFI functionality of Rust and implemented a simble HTTP request using libcurl as an exercise. Consider the following self-contained example:
use std::ffi::c_void;
#[repr(C)]
struct CURL {
_private: [u8; 0],
}
// Global CURL codes
const CURL_GLOBAL_DEFAULT: i64 = 3;
const CURLOPT_WRITEDATA: i64 = 10001;
const CURLOPT_URL: i64 = 10002;
const CURLOPT_WRITEFUNCTION: i64 = 20011;
// Curl types
type CURLcode = i64;
type CURLoption = i64;
// Curl function bindings
#[link(name = "curl")]
extern "C" {
fn curl_easy_init() -> *mut CURL;
fn curl_easy_setopt(handle: *mut CURL, option: CURLoption, value: *mut c_void) -> CURLcode;
fn curl_easy_perform(handle: *mut CURL) -> CURLcode;
fn curl_global_init(flags: i64) -> CURLcode;
}
// Curl callback for data retrieving
extern "C" fn callback_writefunction(
data: *mut u8,
size: usize,
nmemb: usize,
user_data: *mut c_void,
) -> usize {
let slice = unsafe { std::slice::from_raw_parts(data, size * nmemb) };
let mut vec = unsafe { Box::from_raw(user_data as *mut Vec<u8>) };
vec.extend_from_slice(slice);
Box::into_raw(vec);
nmemb * size
}
type Result<T> = std::result::Result<T, CURLcode>;
// Our own curl handle
pub struct Curl {
handle: *mut CURL,
data_ptr: *mut Vec<u8>,
}
impl Curl {
pub fn new() -> std::result::Result<Curl, CURLcode> {
let ret = unsafe { curl_global_init(CURL_GLOBAL_DEFAULT) };
if ret != 0 {
return Err(ret);
}
let handle = unsafe { curl_easy_init() };
if handle.is_null() {
return Err(2); // CURLE_FAILED_INIT according to libcurl-errors(3)
}
// Set data callback
let ret = unsafe {
curl_easy_setopt(
handle,
CURLOPT_WRITEFUNCTION,
callback_writefunction as *mut c_void,
)
};
if ret != 0 {
return Err(2);
}
// Set data pointer
let data_buf = Box::new(Vec::new());
let data_ptr = Box::into_raw(data_buf);
let ret = unsafe {
curl_easy_setopt(handle, CURLOPT_WRITEDATA, data_ptr as *mut std::ffi::c_void)
};
match ret {
0 => Ok(Curl { handle, data_ptr }),
_ => Err(2),
}
}
pub fn set_url(&self, url: &str) -> Result<()> {
let url_cstr = std::ffi::CString::new(url.as_bytes()).unwrap();
let ret = unsafe {
curl_easy_setopt(
self.handle,
CURLOPT_URL,
url_cstr.as_ptr() as *mut std::ffi::c_void,
)
};
match ret {
0 => Ok(()),
x => Err(x),
}
}
pub fn perform(&self) -> Result<String> {
let ret = unsafe { curl_easy_perform(self.handle) };
if ret == 0 {
let b = unsafe { Box::from_raw(self.data_ptr) };
let data = (*b).clone();
Box::into_raw(b);
Ok(String::from_utf8(data).unwrap())
} else {
Err(ret)
}
}
}
fn main() -> Result<()> {
let my_curl = Curl::new().unwrap();
my_curl.set_url("https://www.example.com")?;
my_curl.perform().and_then(|data| Ok(println!("{}", data)))
// No cleanup code in this example for the sake of brevity.
}
While this works, I found it surprising that my_curl does not need to be declared mut, since none of the methods use &mut self, even though they pass a mut* pointer to the FFI function
s.
Should I change the declaration of perform to use &mut self instead of &self (for safety), since the internal buffer gets modified? Rust does not enforce this, but of course Rust does not know that the buffer gets modified by libcurl.
This small example runs fine, but I am unsure if I would be facing any kind of issues in larger programs, when the compiler might optimize for non-mutable access on the Curl struct, even though the instance of the struct is getting modified - or at least the data the pointer is pointing to.
Contrary to popular belief, there is absolutely no borrowchecker-induced restriction in Rust on passing *const/*mut pointers. There doesn't need to be, because dereferencing pointers is inherently unsafe, and can only be done in such blocks, with the programmer verifying all necessary invariants manually. In your case, you need to tell the compiler that is a mutable reference, as you already suspected.
The interested reader should definitely give the ffi section of the nomicon a read, to find out about some interesting ways to shoot yourself in the foot with it.
I'm trying to create a method that can return a reference to Data that is either in a constant global array or inside an Option in an item. The lifetimes are certainly different, but it's safe to assume that the lifetime of the data is at least as long as the lifetime of the item. While doing this, I expected the compiler to warn if I did anything wrong, but it's instead generating wrong instructions and the program is crashing with SIGILL.
Concretely speaking, I have the following code failing in Rust 1.27.2:
#[derive(Debug)]
pub enum Type {
TYPE1,
TYPE2,
}
#[derive(Debug)]
pub struct Data {
pub ctype: Type,
pub int: i32,
}
#[derive(Debug)]
pub struct Entity {
pub idata: usize,
pub modifier: Option<Data>,
}
impl Entity {
pub fn data(&self) -> &Data {
if self.modifier.is_none() {
&DATA[self.idata]
} else {
self.modifier.as_ref().unwrap()
}
}
}
pub const DATA: [Data; 1] = [Data {
ctype: Type::TYPE2,
int: 1,
}];
fn main() {
let mut itemvec = vec![Entity {
idata: 0,
modifier: None,
}];
eprintln!("vec[0]: {:p} = {:?}", &itemvec[0], itemvec[0]);
eprintln!("removed item 0");
let item = itemvec.remove(0);
eprintln!("item: {:p} = {:?}", &item, item);
eprintln!("modifier: {:p} = {:?}", &item.modifier, item.modifier);
eprintln!("DATA: {:p} = {:?}", &DATA[0], DATA[0]);
let itemdata = item.data();
eprintln!("itemdata: {:p} = {:?}", itemdata, itemdata);
}
Complete code
I can't understand what I'm doing wrong. Why isn't the compiler generating a warning? Is it the removal of the (non-copy) item of the vector? Is it the ambiguous lifetimes?
How to return a reference to a global vector or an internal Option?
By using Option::unwrap_or_else:
impl Entity {
pub fn data(&self) -> &Data {
self.modifier.as_ref().unwrap_or_else(|| &DATA[self.idata])
}
}
but it's instead generating wrong instructions and the program is crashing with SIGILL
The code in your question does not have this behavior on macOS with Rust 1.27.2 or 1.28.0. On Ubuntu I see an issue when running the program in Valgrind, but the problem goes away in Rust 1.28.0.
See also:
Why should I prefer `Option::ok_or_else` instead of `Option::ok_or`?
What is this unwrap thing: sometimes it's unwrap sometimes it's unwrap_or
I'm writing some code in Rust that connects to a remote server, and depending on the messages sent by that server, computes some statistics or executes actions based on these statistics. But this is more of a learning project for me and I've run into an issue.
Here is the code that I have reduced to a bare minimum to reproduce the problem :
// Repro code for error[E0502]: cannot borrow `*self` as mutable because `self.server` is also borrowed as immutable
use std::collections::HashMap;
struct ServerReader {
server: Vec<u32>, // A vec for demo purposes, but please imagine this is a server object
counters: HashMap<u32, usize>,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6), // Filling my "server" with some messages
counters: HashMap::new(),
}
}
fn run(&mut self) {
println!("Connecting..."); // ... here there should be some code to connect to the server ...
for message in self.server.iter() { // We wait for the network messages sent by the server, and process them as they come
// ----------- immutable borrow occurs here
println!("Received {}", message);
self.process_message(*message); // HOW
// ^^^^ mutable borrow occurs here
}
// - immutable borrow ends here
println!("Disconnected");
}
fn process_message(&mut self, message: u32) {
// Please imagine that this function contains complex stuff
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
fn main() {
let mut reader = ServerReader::new();
reader.run();
println!("Done");
}
While I think I understand why the compiler is unhappy, I'm struggling to come up with a solution. I cannot manipulate my structure outside of the loop, since I have to work while connected and listening to the server. I also could put everything directly in the loop and not call any method, but I don't want to end up with a 1000 line loop (and I'd prefer to understand what an actual solution would look like).
As you've worked out, you can't call a &mut self method while you're borrowing part of self, so you need to restructure somehow.
The way I would do it is to split the state needed by process_message into a separate type (in your example that's basically the HashMap, but in the real application it's likely to contain more), and move the method to that type. This works because you can separately borrow fields from a struct.
struct SomeState {
counters: HashMap<u32, usize>,
}
impl SomeState {
pub fn new() -> SomeState {
SomeState {
counters: HashMap::new(),
}
}
fn process_message(&mut self, message: u32) {
let counter = self.counters.entry(message).or_insert(0);
*counter += 1;
}
}
struct ServerReader {
server: Vec<u32>,
state: SomeState,
}
impl ServerReader {
fn new() -> ServerReader {
ServerReader {
server: vec!(1, 2, 5, 2, 7, 9, 1, 1, 5, 6),
state: SomeState::new(),
}
}
fn run(&mut self) {
println!("Connecting...");
for message in self.server.iter() {
println!("Received {}", message);
self.state.process_message(*message);
}
println!("Disconnected");
}
}
An alternative (which may or may not be possible in your real example) would be to avoid borrowing in the loop, making it more like:
loop {
// if next_message() returns an owned message, ie not still borrowing
// self
let message = self.next_message();
// now no borrow left
self.process_message(message);
}
Given that you don't need the full ServerReader for processing a message, you could make process_message a free function and just pass &mut self.counters to it. Then you have disjoint borrows of server and counters, which is fine.
Or if your non-server part of ServerReader is larger, extract that into its own struct, and make process_message an impl method of that struct.
In order to allow mutability in an Iterator, you should use iter_mut() and work on mutable references (&mut message). Then, to avoid the additional borrow, you could just perform the addition in the body of the loop:
for &mut message in self.server.iter_mut() {
println!("Received {}", message);
*self.counters.entry(message).or_insert(0) += 1;
}