I have the following code:
use sqlx::mysql::*;
mod db;
#[tokio::main]
async fn main() -> Result<(), sqlx::Error> {
println!("Hello, world!");
let pool = MySqlPoolOptions::new()
.max_connections(5)
.connect("connection-string").await?;
#[derive(Debug, PartialEq, Eq)]
struct Room {
name: String
}
let mut rooms: Vec<Room> = vec![];
let mut stream = sqlx::query("SELECT name FROM rooms")
.map(|row: MySqlRow| {
// map the row into a user-defined domain type
//rooms.push( Room { row.name } );
println!("Tester print");
})
.fetch(&pool);
println!("{:?}", rooms);
Ok(())
}
it seems to connect, it doesnt error out, but its not getting any data, or at least the print inside the map function is not getting executed. Anyone know why?
In sqlx, map and fetch are lazy, meaning that they don't do anything unless and until explicitly consumed. This is a very common pattern in Rust and avoids wasting time if you only need part of the results. For your use-case where you want to store (transformed) values into a Vec, the best way to do this is with collect or fetch_all.
collect applies to all kinds of Streams, not just the results from an sqlx query:
let mut rooms = sqlx::query("SELECT name FROM rooms")
.map(|row: MySqlRow| { Room { row.name } })
.fetch(&pool)
.collect::<Result<Vec<_>>>()
.await?;
or fetch_all makes for slightly shorter code but is specific to sqlx (if you look at its source code, it does basically the same thing):
let mut rooms = sqlx::query("SELECT name FROM rooms")
.map(|row: MySqlRow| { Room { row.name } })
.fetch_all(&pool)
.await?;
Related
I am saving in append mode a stream of events on a YAML log file, where each event is represented by an indivual document, like this:
---
type: event
id: 1
---
type: trigger
id: 2
At some point later I want to iterate on these events, parsing each via serde_yaml. To my understanding though, serde_yaml doesn't seem to support parsing multiple documents from a single reader, as none of the available methods mention it, and trying to parse multiple documents at once results in a MoreThanOneDocument error.
use std::io::{self, BufRead};
use serde_yaml;
use serde::{self, Deserialize};
#[derive(Deserialize, Debug)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Message {
Event { id: i32 },
Trigger { id: i32},
}
fn main() -> io::Result<()> {
let yaml = "---\ntype: event\nid: 1\n---\n\ntype: trigger\nid: 2";
let v: Message = serde_yaml::from_reader(yaml.as_bytes()).unwrap();
println!("{:?}", v);
Ok(())
}
I'm totally new to Rust, so maybe I completely missed the point of serde and just did not understand how to do it.
How would you parse such YAML, please?
I cooked up something that looks like a working solution, but I think I'll try to post it among the answers instead, because I don't want to bias other answers too much towards my solution. I kindly encourage you to have a look at it as well however, any feedback is welcome.
The documentation of serde_yaml::Deserializer shows an example very similar to yours. It would work like this:
use serde::Deserialize;
#[derive(Deserialize, Debug)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Message {
Event { id: i32 },
Trigger { id: i32 },
}
fn main() {
let yaml = "---\ntype: event\nid: 1\n---\ntype: trigger\nid: 2\n";
for document in serde_yaml::Deserializer::from_str(yaml) {
let v = Message::deserialize(document).unwrap();
println!("{:?}", v);
}
}
I really hope to find a native solution by using serde and serde_yaml only, but until then the way I got it working is as follows.
trait BufReaderYamlExt {
fn read_next_yaml(&mut self) -> io::Result<Option<String>>;
}
impl<T: io::Read> BufReaderYamlExt for io::BufReader<T> {
fn read_next_yaml(&mut self) -> io::Result<Option<String>> {
const sep : &str = "\n---\n";
let mut doc = String::with_capacity(200);
while self.read_line(&mut doc)? > 0 {
if doc.len() > sep.len() && doc.ends_with(sep) {
doc.truncate(doc.len() - sep.len());
break;
}
}
if !doc.is_empty() {
doc.shrink_to_fit();
Ok(Some(doc))
} else {
Ok(None)
}
}
}
The trait extends the BufReader with an extra method that returns an optional owned String (or None at the end of the stream) containing just the portion with a single YAML document.
By iterating on it one could then apply serde_json::from_str() to parse the document into a Message struct.
fn main() -> io::Result<()> {
let yaml = "---\ntype: event\nid: 1\n\n---\n\ntype: trigger\nid: 2\n";
let mut r = io::BufReader::new(yaml.as_bytes());
while let Some(next) = r.read_next_yaml()? {
let d: Message = serde_yaml::from_str(&next).unwrap();
println!("parsed: {:?}", d);
}
Ok(())
}
I've made available the full source on the rust playground as well.
I want to send Events between the game client and server and I already got it working, but I do not know how to do it with bevy.
I am dependent to use tokios async TcpStream, because I have to be able to split the stream into a OwnedWriteHalf and OwnedReadhalf using stream.into_split().
My first idea was to just spawn a thread that handles the connection and then send the received events to a queue using mpsc::channel
Then I include this queue into a bevy resource using app.insert_resource(Queue) and pull events from it in the game loop.
the Queue:
use tokio::sync::mpsc;
pub enum Instruction {
Push(GameEvent),
Pull(mpsc::Sender<Option<GameEvent>>),
}
#[derive(Clone, Debug)]
pub struct Queue {
sender: mpsc::Sender<Instruction>,
}
impl Queue {
pub fn init() -> Self {
let (tx, rx) = mpsc::channel(1024);
init(rx);
Self{sender: tx}
}
pub async fn send(&self, event: GameEvent) {
self.sender.send(Instruction::Push(event)).await.unwrap();
}
pub async fn pull(&self) -> Option<GameEvent> {
println!("new pull");
let (tx, mut rx) = mpsc::channel(1);
self.sender.send(Instruction::Pull(tx)).await.unwrap();
rx.recv().await.unwrap()
}
}
fn init(mut rx: mpsc::Receiver<Instruction>) {
tokio::spawn(async move {
let mut queue: Vec<GameEvent> = Vec::new();
loop {
match rx.recv().await.unwrap() {
Instruction::Push(ev) => {
queue.push(ev);
}
Instruction::Pull(sender) => {
sender.send(queue.pop()).await.unwrap();
}
}
}
});
}
But because all this has to be async I have block the pull() function in the sync game loop.
I do this using the futures-lite crate:
fn event_pull(
communication: Res<Communication>
) {
let ev = future::block_on(communication.event_queue.pull());
println!("got event: {:?}", ev);
}
And this works fine, BUT after around 5 seconds the whole program just halts and does not receive any more events.
It seems like that future::block_on() does block indefinitely.
Having the main function, in which bevy::prelude::App gets built and run, to be the async tokio::main function might also be a problem here.
It would probably be best to wrap the async TcpStream initialisation and tokio::sync::mpsc::Sender and thus also Queue.pull into synchronous functions, but I do not know how to do this.
Can anyone help?
How to reproduce
The repo can be found here
Just compile both server and client and then run both in the same order.
I got it to work by just replacing every tokio::sync::mpsc with crossbeam::channel, which might be a problem, as it does block
and manually initializing the tokio runtime.
so the init code looks like this:
pub struct Communicator {
pub event_bridge: bridge::Bridge,
pub event_queue: event_queue::Queue,
_runtime: Runtime,
}
impl Communicator {
pub fn init(ip: &str) -> Self {
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_io()
.build()
.unwrap();
let (bridge, queue, game_rx) = rt.block_on(async move {
let socket = TcpStream::connect(ip).await.unwrap();
let (read, write) = socket.into_split();
let reader = TcpReader::new(read);
let writer = TcpWriter::new(write);
let (bridge, tcp_rx, game_rx) = bridge::Bridge::init();
reader::init(bridge.clone(), reader);
writer::init(tcp_rx, writer);
let event_queue = event_queue::Queue::init();
return (bridge, event_queue, game_rx);
});
// game of game_rx events to queue for game loop
let eq_clone = queue.clone();
rt.spawn(async move {
loop {
let event = game_rx.recv().unwrap();
eq_clone.send(event);
}
});
Self {
event_bridge: bridge,
event_queue: queue,
_runtime: rt,
}
}
}
And main.rs looks like this:
fn main() {
let communicator = communication::Communicator::init("0.0.0.0:8000");
communicator.event_bridge.push_tcp(TcpEvent::Register{name: String::from("luca")});
App::new()
.insert_resource(communicator)
.add_system(event_pull)
.add_plugins(DefaultPlugins)
.run();
}
fn event_pull(
communication: Res<communication::Communicator>
) {
let ev = communication.event_queue.pull();
if let Some(ev) = ev {
println!("ev");
}
}
Perhaps there might be a better solution.
I want to create a vector with all of the matching field id from the struct, process that new vector and then repeat the process. Basically grouping together the structs with matching field id.
Is there a way to do this by not using the unstable feature drain_filter?
#![feature(drain_filter)]
#[derive(Debug)]
struct Person {
id: u32,
}
fn main() {
let mut people = vec![];
for p in 0..10 {
people.push(Person { id: p });
}
while !people.is_empty() {
let first_person_id = people.first().unwrap().id;
let drained: Vec<Person> = people.drain_filter(|p| p.id == first_person_id).collect();
println!("{:#?}", drained);
}
}
Playground
If you are looking to group your vector by the person id, it's likely to be more efficient using a HashMap from id to Vec<Person>, where each id hold a vector of persons. And then you can loop through the HashMap and process each vector / group. This is potentially more efficient than draining people in each iteration, which in worst case has O(N^2) time complexity while with a HashMap the time complexity is O(N).
#![feature(drain_filter)]
use std::collections::HashMap;
#[derive(Debug)]
struct Person {
id: u32,
}
fn main() {
let mut people = vec![];
let mut groups: HashMap<u32, Vec<Person>> = HashMap::new();
for p in 0..10 {
people.push(Person { id: p });
}
people.into_iter().for_each(|person| {
let group = groups.entry(person.id).or_insert(vec![]);
group.push(person);
});
for (_id, group) in groups {
println!("{:#?}", group);
}
}
Playground
I am attempting to make a future that continuously finds new work to do and then maintains a set of futures for those work items. I would like to make sure neither my main future that finds work to be blocked for long periods of time and to have my work being done concurrently.
Here is a rough overview of what I am trying to do. Specifically isDone does not exist and also from what I can understand from the docs isn't necessarily a valid way to use futures in Rust. What is the idomatic way of doing this kind of thing?
use std::collections::HashMap;
use tokio::runtime::Runtime;
async fn find_work() -> HashMap<i64, String> {
// Go read from the DB or something...
let mut work = HashMap::new();
work.insert(1, "test".to_string());
work.insert(2, "test".to_string());
return work;
}
async fn do_work(id: i64, value: String) -> () {
// Result<(), Error> {
println!("{}: {}", id, value);
}
async fn async_main() -> () {
let mut pending_work = HashMap::new();
loop {
for (id, value) in find_work().await {
if !pending_work.contains_key(&id) {
let fut = do_work(id, value);
pending_work.insert(id, fut);
}
}
pending_work.retain(|id, fut| {
if isDone(fut) {
// do something with the result
false
} else {
true
}
});
}
}
fn main() {
let runtime = Runtime::new().unwrap();
let exec = runtime.executor();
exec.spawn(async_main());
runtime.shutdown_on_idle();
}
I have a set of jobs that I am trying to run in parallel. I want to run each task on its own thread and gather the responses on the calling thread.
Some jobs may take much longer than others, so I'd like to start using each result as it comes in, and not have to wait for all jobs to complete.
Here is an attempt:
struct Container<T> {
items : Vec<T>
}
#[derive(Debug)]
struct Item {
x: i32
}
impl Item {
fn foo (&mut self) {
self.x += 1; //consider an expensive mutating computation
}
}
fn main() {
use std;
use std::sync::{Mutex, Arc};
use std::collections::RingBuf;
//set up a container with 2 items
let mut item1 = Item { x: 0};
let mut item2 = Item { x: 1};
let container = Container { items: vec![item1, item2]};
//set a gather system for our results
let ringBuf = Arc::new(Mutex::new(RingBuf::<Item>::new()));
//farm out each job to its own thread...
for item in container.items {
std::thread::Thread::spawn(|| {
item.foo(); //job
ringBuf.lock().unwrap().push_back(item); //push item back to caller
});
}
loop {
let rb = ringBuf.lock().unwrap();
if rb.len() > 0 { //gather results as soon as they are available
println!("{:?}",rb[0]);
rb.pop_front();
}
}
}
For starters, this does not compile due to the impenetrable cannot infer an appropriate lifetime due to conflicting requirements error.
What am I doing wrong and how do I do it right?
You've got a couple compounding issues, but the first one is a misuse / misunderstanding of Arc. You need to give each thread it's own copy of the Arc. Arc itself will make sure that changes are synchronized. The main changes were the addition of .clone() and the move keyword:
for item in container.items {
let mrb = ringBuf.clone();
std::thread::Thread::spawn(move || {
item.foo(); //job
mrb.lock().unwrap().push_back(item); //push item back to caller
});
}
After changing this, you'll run into some simpler errors about forgotten mut qualifiers, and then you hit another problem - you are trying to send mutable references across threads. Your for loop will need to return &mut Item to call foo, but this doesn't match your Vec. Changing it, we can get to something that compiles:
for mut item in container.items.into_iter() {
let mrb = ringBuf.clone();
std::thread::Thread::spawn(move || {
item.foo(); //job
mrb.lock().unwrap().push_back(item); //push item back to caller
});
}
Here, we consume the input vector, moving each of the Items to the worker thread. Unfortunately, this hits the Playpen timeout, so there's probably some deeper issue.
All that being said, I'd highly recommend using channels:
#![feature(std_misc)]
use std::sync::mpsc::channel;
#[derive(Debug)]
struct Item {
x: i32
}
impl Item {
fn foo(&mut self) { self.x += 1; }
}
fn main() {
let items = vec![Item { x: 0 }, Item { x: 1 }];
let rx = {
let (tx, rx) = channel();
for item in items.into_iter() {
let my_tx = tx.clone();
std::thread::Thread::spawn(move || {
let mut item = item;
item.foo();
my_tx.send(item).unwrap();
});
}
rx
};
for item in rx.iter() {
println!("{:?}", item);
}
}
This also times-out in the playpen, but works fine when compiled and run locally.