I'm looking for the string which occurs most frequently in the second part of the tuple of Vec<(String, Vec<String>)>:
use itertools::Itertools; // 0.8.0
fn main() {
let edges: Vec<(String, Vec<String>)> = vec![];
let x = edges
.iter()
.flat_map(|x| &x.1)
.map(|x| &x[..])
.sorted()
.group_by(|x| x)
.max_by_key(|x| x.len());
}
Playground
This:
takes the iterator
flat-maps to the second part of the tuple
turns elements into a &str
sorts it (via itertools)
groups it by string (via itertools)
find the group with the highest count
This supposedly gives me the group with the most frequently occurring string, except it doesn't compile:
error[E0599]: no method named `max_by_key` found for type `itertools::groupbylazy::GroupBy<&&str, std::vec::IntoIter<&str>, [closure#src/lib.rs:9:19: 9:24]>` in the current scope
--> src/lib.rs:10:10
|
10 | .max_by_key(|x| x.len());
| ^^^^^^^^^^
|
= note: the method `max_by_key` exists but the following trait bounds were not satisfied:
`&mut itertools::groupbylazy::GroupBy<&&str, std::vec::IntoIter<&str>, [closure#src/lib.rs:9:19: 9:24]> : std::iter::Iterator`
I'm totally lost in these types.
You didn't read the documentation for a function you are using. This is not a good idea.
This type implements IntoIterator (it is not an iterator itself),
because the group iterators need to borrow from this value. It should
be stored in a local variable or temporary and iterated.
Personally, I'd just use a BTreeMap or HashMap:
let mut counts = BTreeMap::new();
for word in edges.iter().flat_map(|x| &x.1) {
*counts.entry(word).or_insert(0) += 1;
}
let max = counts.into_iter().max_by_key(|&(_, count)| count);
println!("{:?}", max);
If you really wanted to use the iterators, it could look something like this:
let groups = edges
.iter()
.flat_map(|x| &x.1)
.sorted()
.group_by(|&x| x);
let max = groups
.into_iter()
.map(|(key, group)| (key, group.count()))
.max_by_key(|&(_, count)| count);
Related
I am new to rust. I want to write a function which later can be imported into Python as a module using the pyo3 crate.
Below is the Python implementation of the function I want to implement in Rust:
def pcompare(a, b):
letters = []
for i, letter in enumerate(a):
if letter != b[i]:
letters.append(f'{letter}{i + 1}{b[i]}')
return letters
The first Rust implemention I wrote looks like this:
use pyo3::prelude::*;
#[pyfunction]
fn compare_strings_to_vec(a: &str, b: &str) -> PyResult<Vec<String>> {
if a.len() != b.len() {
panic!(
"Reads are not the same length!
First string is length {} and second string is length {}.",
a.len(), b.len());
}
let a_vec: Vec<char> = a.chars().collect();
let b_vec: Vec<char> = b.chars().collect();
let mut mismatched_chars = Vec::new();
for (mut index,(i,j)) in a_vec.iter().zip(b_vec.iter()).enumerate() {
if i != j {
index += 1;
let mutation = format!("{i}{index}{j}");
mismatched_chars.push(mutation);
}
}
Ok(mismatched_chars)
}
#[pymodule]
fn compare_strings(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(compare_strings_to_vec, m)?)?;
Ok(())
}
Which I builded in --release mode. The module could be imported to Python, but the performance was quite similar to the performance of the Python implementation.
My first question is: Why is the Python and Rust function similar in speed?
Now I am working on a parallelization implementation in Rust. When just printing the result variable, the function works:
use rayon::prelude::*;
fn main() {
let a: Vec<char> = String::from("aaaa").chars().collect();
let b: Vec<char> = String::from("aaab").chars().collect();
let length = a.len();
let index: Vec<_> = (1..=length).collect();
let mut mismatched_chars: Vec<String> = Vec::new();
(a, index, b).into_par_iter().for_each(|(x, i, y)| {
if x != y {
let mutation = format!("{}{}{}", x, i, y).to_string();
println!("{mutation}");
//mismatched_chars.push(mutation);
}
});
}
However, when I try to push the mutation variable to the mismatched_charsvector:
use rayon::prelude::*;
fn main() {
let a: Vec<char> = String::from("aaaa").chars().collect();
let b: Vec<char> = String::from("aaab").chars().collect();
let length = a.len();
let index: Vec<_> = (1..=length).collect();
let mut mismatched_chars: Vec<String> = Vec::new();
(a, index, b).into_par_iter().for_each(|(x, i, y)| {
if x != y {
let mutation = format!("{}{}{}", x, i, y).to_string();
//println!("{mutation}");
mismatched_chars.push(mutation);
}
});
}
I get the following error:
error[E0596]: cannot borrow `mismatched_chars` as mutable, as it is a captured variable in a `Fn` closure
--> src/main.rs:16:13
|
16 | mismatched_chars.push(mutation);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
For more information about this error, try `rustc --explain E0596`.
error: could not compile `testing_compare_strings` due to previous error
I tried A LOT of different things. When I do:
use rayon::prelude::*;
fn main() {
let a: Vec<char> = String::from("aaaa").chars().collect();
let b: Vec<char> = String::from("aaab").chars().collect();
let length = a.len();
let index: Vec<_> = (1..=length).collect();
let mut mismatched_chars: Vec<&str> = Vec::new();
(a, index, b).into_par_iter().for_each(|(x, i, y)| {
if x != y {
let mutation = format!("{}{}{}", x, i, y).to_string();
mismatched_chars.push(&mutation);
}
});
}
The error becomes:
error[E0596]: cannot borrow `mismatched_chars` as mutable, as it is a captured variable in a `Fn` closure
--> src/main.rs:16:13
|
16 | mismatched_chars.push(&mutation);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
error[E0597]: `mutation` does not live long enough
--> src/main.rs:16:35
|
10 | let mut mismatched_chars: Vec<&str> = Vec::new();
| -------------------- lifetime `'1` appears in the type of `mismatched_chars`
...
16 | mismatched_chars.push(&mutation);
| ----------------------^^^^^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `mutation` is borrowed for `'1`
17 | }
| - `mutation` dropped here while still borrowed
I suspect that the solution is quite simple, but I cannot see it myself.
You have the right idea with what you are doing, but you will want to try to use an iterator chain with filter and map to remove or convert iterator items into different values. Rayon also provides a collect method similar to regular iterators to convert items into a type T: FromIterator (such as Vec<T>).
fn compare_strings_to_vec(a: &str, b: &str) -> Vec<String> {
// Same as with the if statement, but just a little shorter to write
// Plus, it will print out the two values it is comparing if it errors.
assert_eq!(a.len(), b.len(), "Reads are not the same length!");
// Zip the character iterators from a and b together
a.chars().zip(b.chars())
// Iterate with the index of each item
.enumerate()
// Rayon function which turns a regular iterator into a parallel one
.par_bridge()
// Filter out values where the characters are the same
.filter(|(_, (a, b))| a != b)
// Convert the remaining values into an error string
.map(|(index, (a, b))| {
format!("{}{}{}", a, index + 1, b)
})
// Turn the items of this iterator into a Vec (Or any other FromIterator type).
.collect()
}
Rust Playground
Optimizing for speed
On the other hand, if you want speed we need to approach this problem from a different direction. You may have noticed, but the rayon version is quite slow since the cost of spawning a thread and using concurrency structures is orders of magnitude more than just simply comparing the bytes in the original thread. In my benchmarks, I found that even with better workload distribution, additional threads were only helpful on my machine (64GB RAM, 16 cores) when the strings were at least 1-2 million bytes long. Given that you have stated they are typically ~30,000 bytes long I think using rayon (or really any other threading for comparisons of this size) will only slow down your code.
Using criterion for benchmarking, I eventually came to this implementation. It generally gets about 2.8156 µs per run on strings of 30,000 characters with 10 different bytes. For comparison, the code posted in the original question usually gets around 61.156 µs on my system under the same conditions so this should give a ~20x speedup. It can vary a bit, but it consistently got the best results in the benchmark. I'm guessing this should be fast enough to have this step no-longer be the bottleneck in your code.
This key focus of this implementation is to do the comparisons in batches. We can take advantage of the 128bit registers on most CPUs to compare the input in 16 byte batches. Upon an inequality being found, the 16 byte section it covers is re-scanned for the exact position of the discrepancy. This gives a decent boost to performance. I initially thought that a usize would work better, but it seems that was not the case. I also attempted to use the portable_simd nightly feature to write a simd version of this code, but I was unable to match the speed of this code. I suspect this was either due to missed optimizations or a lack of experience to effectively use simd on my part.
I was worried about drops in speed due to alignment of chunks not being enforced for u128 values, but it seems to mostly be a non-issue. First of all, it is generally quite difficult to find allocators which are willing to allocate to an address which is not a multiple of the system word size. Of course, this is due to practicality rather than any actual requirement. When I manually gave it unaligned slices (unaligned for u128s), it is not significantly effected. This is why I do not attempt to enforce that the start index of the slice be aligned to align_of::<u128>().
fn compare_strings_to_vec(a: &str, b: &str) -> Vec<String> {
let a_bytes = a.as_bytes();
let b_bytes = b.as_bytes();
let remainder = a_bytes.len() % size_of::<u128>();
// Strongly suggest to the compiler we are iterating though u128
a_bytes
.chunks_exact(size_of::<u128>())
.zip(b_bytes.chunks_exact(size_of::<u128>()))
.enumerate()
.filter(|(_, (a, b))| {
let a_block: &[u8; 16] = (*a).try_into().unwrap();
let b_block: &[u8; 16] = (*b).try_into().unwrap();
u128::from_ne_bytes(*a_block) != u128::from_ne_bytes(*b_block)
})
.flat_map(|(word_index, (a, b))| {
fast_path(a, b).map(move |x| word_index * size_of::<u128>() + x)
})
.chain(
fast_path(
&a_bytes[a_bytes.len() - remainder..],
&b_bytes[b_bytes.len() - remainder..],
)
.map(|x| a_bytes.len() - remainder + x),
)
.map(|index| {
format!(
"{}{}{}",
char::from(a_bytes[index]),
index + 1,
char::from(b_bytes[index])
)
})
.collect()
}
/// Very similar to regular route, but with nothing fancy, just get the indices of the overlays
#[inline(always)]
fn fast_path<'a>(a: &'a [u8], b: &'a [u8]) -> impl 'a + Iterator<Item = usize> {
a.iter()
.zip(b.iter())
.enumerate()
.filter_map(|(x, (a, b))| (a != b).then_some(x))
}
You cannot directly access the field mismatched_chars in a multithreading environment.
You can use Arc<RwLock> to access the field in multithreading.
use rayon::prelude::*;
use std::sync::{Arc, RwLock};
fn main() {
let a: Vec<char> = String::from("aaaa").chars().collect();
let b: Vec<char> = String::from("aaab").chars().collect();
let length = a.len();
let index: Vec<_> = (1..=length).collect();
let mismatched_chars: Arc<RwLock<Vec<String>>> = Arc::new(RwLock::new(Vec::new()));
(a, index, b).into_par_iter().for_each(|(x, i, y)| {
if x != y {
let mutation = format!("{}{}{}", x, i, y);
mismatched_chars
.write()
.expect("could not acquire write lock")
.push(mutation);
}
});
for mismatch in mismatched_chars
.read()
.expect("could not acquire read lock")
.iter()
{
eprintln!("{}", mismatch);
}
}
I'm learning Rust and noticed the following iterator pattern in a number of places:
let some_vector: &[& str] = &["hello", "world", "zombies", "pants"];
let result: Vec<&str> = some_vector
.iter()
.filter(|&x| *x == "hello")
.map(|&x| x)
.collect();
What's the purpose of that .map(|&x| x)? Why is it necessary? Does it create a copy?
When I remove it, I get the following compiler error:
error[E0277]: a value of type `Vec<&str>` cannot be built from an iterator over elements of type `&&str`
--> src/main.rs:7:6
|
7 | .collect();
| ^^^^^^^ value of type `Vec<&str>` cannot be built from `std::iter::Iterator<Item=&&str>`
|
= help: the trait `FromIterator<&&str>` is not implemented for `Vec<&str>`
note: required by a bound in `collect`
For more information about this error, try `rustc --explain E0277`.
So the map turns an iterator over references to string slices into an iterator over string slices? Removing one level of indirection? Is that right?
In addition to #AlexW's answer, actually there is no need to write that, because there is a builtin iterator adapter that does it better (more clear, more performant): copied().
let some_vector: &[&str] = &["hello", "world", "zombies", "pants"];
let result: Vec<&str> = some_vector
.iter()
.filter(|&x| *x == "hello")
.copied()
.collect();
There is also cloned() which is equal to .map(|x| x.clone()).
Assuming you're using 2021 edition, it converts from impl Iterator< Item = &&str> to impl Iterator< Item = &str>:
let some_vector: &[& str] = &["hello", "world", "zombies", "pants"];
let result: Vec<&str> = some_vector // &[&str]
.iter() // Iter<&str>
.filter(|&x| *x == "hello") // Impl Iterator< Item = &&str>
.map(|&x| x) // Impl Iterator< Item = &str>
.collect();
And the reason it's necessary is because the FromIterator trait is already implemented for &str as it's a relatively more common use case and it's not implemented for &&str as the error message says:
the trait `FromIterator<&&str>` is not implemented for `Vec<&str>`
I have the following code:
fn main() {
let mut vec = Vec::new();
vec.push(String::from("Foo"));
let mut row = vec.get_mut(0).unwrap();
row.push('!');
println!("{}", vec[0])
}
It prints out "Foo!", but the compiler tells me:
warning: variable does not need to be mutable
--> src/main.rs:4:9
|
4 | let mut row = vec.get_mut(0).unwrap();
| ----^^^
| |
| help: remove this `mut`
Surprisingly, removing the mut works. This raises a few questions:
Why does this work?
Why doesn't this work when I use vec.get instead of vec.get_mut, regardless of whether I use let or let mut?
Why doesn't vec work in the same way, i.e. when I use let vec = Vec::new(), why can't I call vec.push()?
vec.get_mut(0) returns an Option<&mut String>, so when you unwrap that value you will have a mutable borrow of a String. Remember, that a let statement's left side is using pattern matching, so when your pattern is just a variable name you essentially say match whatever is on the right and call it name. Thus row matches against &mut String so it already is mutable.
Here's a much simpler and more straightforward example to illustrate the case (which you can try in the playground):
fn main() {
let mut x = 55i32;
dbg!(&x);
let y = &mut x; // <-- y's type is `&mut i32`
*y = 12;
dbg!(&x);
}
I'm trying to write some Rust code to decode GPS data from an SDR receiver. I'm reading samples in from a file and converting the binary data to a series of complex numbers, which is a time-consuming process. However, there are times when I want to stream samples in without keeping them in memory (e.g. one very large file processed only one way or samples directly from the receiver) and other times when I want to keep the whole data set in memory (e.g. one small file processed in multiple different ways) to avoid repeating the work of parsing the binary file.
Therefore, I want to write functions or structs with iterators to be as general as possible, but I know they aren't sized, so I need to put them in a Box. I would have expected something like this to work.
This is the simplest example I could come up with to demonstrate the same basic problem.
fn sum_squares_plus(iter: Box<Iterator<Item = usize>>, x: usize) -> usize {
let mut ans: usize = 0;
for i in iter {
ans += i * i;
}
ans + x
}
fn main() {
// Pretend this is an expensive operation that I don't want to repeat five times
let small_data: Vec<usize> = (0..10).collect();
for x in 0..5 {
// Want to iterate over immutable references to the elements of small_data
let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter());
println!("{}: {}", x, sum_squares_plus(iterbox, x));
}
// 0..100 is more than 0..10 and I'm only using it once,
// so I want to 'stream' it instead of storing it all in memory
let x = 55;
println!("{}: {}", x, sum_squares_plus(Box::new(0..100), x));
}
I've tried several different variants of this, but none seem to work. In this particular case, I'm getting
error[E0271]: type mismatch resolving `<std::slice::Iter<'_, usize> as std::iter::Iterator>::Item == usize`
--> src/main.rs:15:52
|
15 | let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found usize
|
= note: expected type `&usize`
found type `usize`
= note: required for the cast to the object type `dyn std::iter::Iterator<Item = usize>`
I'm not worried about concurrency and I'd be happy to just get it working sequentially on a single thread, but a concurrent solution would be a nice bonus.
The current error you're running into is here:
let iterbox:Box<Iterator<Item = usize>> = Box::new(small_data.iter());
You're declaring that you want an iterator that returns usize items, but small_data.iter() is an iterator that returns references to usize items (&usize). That why you get the error "expected reference, found usize". usize is a small type that's cloneable so you can simply use the .cloned() iterator adapter to provide an iterator that actually returns a usize.
let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter().cloned());
Once you're past that hurdle, the next problem is that the iterator returned over small_data contains a reference to the small_data. Since sum_squares_plus is defined to accept a Box<Iterator<Item = usize>>, it's implied in that signature that the Iterator trait object within the box has a 'static lifetime. The iterator you're providing does not because it borrows small_data. To fix that you need to adjust the sum_squares_plus definition to
fn sum_squares_plus<'a>(iter: Box<Iterator<Item = usize> + 'a>, x: usize) -> usize
Note the 'a lifetime annotations. The code should then compile, but unless there's some constraints other than what's clearly defined here, a more idiomatic and efficient approach would be to avoid using trait objects and the associated allocations. The below code should work using static dispatch without any trait objects.
fn sum_squares_plus<I: Iterator<Item = usize>>(iter: I, x: usize) -> usize {
let mut ans: usize = 0;
for i in iter {
ans += i * i;
}
ans + x
}
fn main() {
// Pretend this is an expensive operation that I don't want to repeat five times
let small_data: Vec<usize> = (0..10).collect();
for x in 0..5 {
println!("{}: {}", x, sum_squares_plus(small_data.iter().cloned(), x));
}
// 0..100 is more than 0..10 and I'm only using it once,
// so I want to 'stream' it instead of storing it all in memory
let x = 55;
println!("{}: {}", x, sum_squares_plus(Box::new(0..100), x));
}
I'm trying to chunk an vector of uneven length strings into a vector of even length strings. The laziest way I could think of doing this is to join the arguments into a string, convert the chars to a vector, and then use Vec::chunks. Unfortunately, I'm running into issues trying to collect the chunks into strings.
let args: Vec<String> = ["123", "4", "56"].iter().map(|&s| s.into()).collect();
let result: Vec<String> = args
.join(" ")
.chars()
.collect::<Vec<_>>()
.chunks(2)
.map(|c| c.collect::<String>())
.collect::<Vec<String>>();
assert_eq!(["12", "34", "56"], result);
Results in the error:
error[E0599]: no method named `collect` found for type `&[char]` in the current scope
--> src/main.rs:9:20
|
9 | .map(|c| c.collect::<String>())
| ^^^^^^^
|
= note: the method `collect` exists but the following trait bounds were not satisfied:
`&mut &[char] : std::iter::Iterator`
`&mut [char] : std::iter::Iterator`
You weren't far off:
let result: Vec<String> = args
.join("")
.chars()
.collect::<Vec<_>>()
.chunks(2)
.map(|x| x.iter().cloned().collect())
.collect();
println!("{:?}", result);
You probably don't want a space when joining them together.
You need to convert each chunk (which is a &[char]) into an iterator via .iter(). You then have to convert the iterated type from a &char to a char via .cloned().
I might write this using Itertools::chunks though:
use itertools::Itertools; // 0.8.0
fn main() {
let args = ["123", "4", "56"];
let together = args.iter().flat_map(|x| x.chars());
let result: Vec<String> = together
.chunks(2)
.into_iter()
.map(|x| x.collect())
.collect();
println!("{:?}", result);
}
flat_map avoids the need to create a String, it just chains one iterator to the next.
Itertools::chunks allows the programmer to not create an intermediate Vec. Instead, it has an internal vector that, IIRC, will only store up to n values in it before yielding a value. This way you are buffering a smaller amount of items.