Split a string keeping the separators - rust

Question

Is there a trivial way to split a string keeping the separators?
Instead of this:
let texte = "Ten. Million. Questions. Let's celebrate all we've done together.";
let v: Vec<&str> = texte.split(|c: char| !(c.is_alphanumeric() || c == '\'')).filter(|s| !s.is_empty()).collect();
which results with ["Ten", "Million", "Questions", "Let's", "celebrate", "all", "we've", "done", "together"].
I would like something that gives me :
["Ten", ".", " ", "Million", ".", " ", "Questions", ".", " ", "Let's", " ", "celebrate", " ", "all", " ", "we've", " ", "done", " ", "together", "."].
I am trying that kind of code (it assumes the string begins with a letter and ends with a 'non'-letter) :
let texte = "Ten. Million. Questions. Let's celebrate all we've done together. ";
let v1: Vec<&str> = texte.split(|c: char| !(c.is_alphanumeric() || c == '\'')).filter(|s| !s.is_empty()).collect();
let v2: Vec<&str> = texte.split(|c: char| c.is_alphanumeric() || c == '\'').filter(|s| !s.is_empty()).collect();
let mut w: Vec<&str> = Vec::new();
let mut j = 0;
for i in v2 {
w.push(v1[j]);
w.push(i);
j = j+1;
}
It gives me almost the result I wrote earlier but it's good :
["Ten", ". ", "Million", ". ", "Questions", ". ", "Let's", " ", "celebrate", " ", "all", " ", "we've", " ", "done", " ", "together", "."]
However is there a better way to code that ? Because I tried to enumerate on v2 but it didn't work, and it looks rough to use j in the for loop.

Answer 1

Using str::match_indices:
let text = "Ten. Million. Questions. Let's celebrate all we've done together.";
let mut result = Vec::new();
let mut last = 0;
for (index, matched) in text.match_indices(|c: char| !(c.is_alphanumeric() || c == '\'')) {
if last != index {
result.push(&text[last..index]);
}
result.push(matched);
last = index + matched.len();
}
if last < text.len() {
result.push(&text[last..]);
}
println!("{:?}", result);
Prints:
["Ten", ".", " ", "Million", ".", " ", "Questions", ".", " ", "Let\'s", " ", "celebrate", " ", "all", " ", "we\'ve", " ", "done", " ", "together", "."]

Answer 2

str::split_inclusive, available since Rust 1.51, returns an iterator keeping the delimiters as part of the matched strings, and may be useful in certain cases:
#[test]
fn split_with_delimiter() {
let items: Vec<_> = "alpha,beta;gamma"
.split_inclusive(&[',', ';'][..])
.collect();
assert_eq!(&items, &["alpha,", "beta;", "gamma"]);
}
#[test]
fn split_with_delimiter_allows_consecutive_delimiters() {
let items: Vec<_> = ",;".split_inclusive(&[',', ';'][..]).collect();
assert_eq!(&items, &[",", ";"]);
}

Answer 3

I was not able to find anything in the standard library, so I wrote my own:
This version uses the unstable pattern API as it's more flexible, but the link above has a fallback that I've hardcoded for my specific stable usecase.
#![feature(pattern)]
use std::str::pattern::{Pattern, Searcher};
#[derive(Copy, Clone, Debug, PartialEq)]
pub enum SplitType<'a> {
Match(&'a str),
Delimiter(&'a str),
}
pub struct SplitKeepingDelimiter<'p, P>
where
P: Pattern<'p>,
{
searcher: P::Searcher,
start: usize,
saved: Option<usize>,
}
impl<'p, P> Iterator for SplitKeepingDelimiter<'p, P>
where
P: Pattern<'p>,
{
type Item = SplitType<'p>;
fn next(&mut self) -> Option<Self::Item> {
if self.start == self.searcher.haystack().len() {
return None;
}
if let Some(end_of_match) = self.saved.take() {
let s = &self.searcher.haystack()[self.start..end_of_match];
self.start = end_of_match;
return Some(SplitType::Delimiter(s));
}
match self.searcher.next_match() {
Some((start, end)) => {
if self.start == start {
let s = &self.searcher.haystack()[start..end];
self.start = end;
Some(SplitType::Delimiter(s))
} else {
let s = &self.searcher.haystack()[self.start..start];
self.start = start;
self.saved = Some(end);
Some(SplitType::Match(s))
}
}
None => {
let s = &self.searcher.haystack()[self.start..];
self.start = self.searcher.haystack().len();
Some(SplitType::Match(s))
}
}
}
}
pub trait SplitKeepingDelimiterExt: ::std::ops::Index<::std::ops::RangeFull, Output = str> {
fn split_keeping_delimiter<P>(&self, pattern: P) -> SplitKeepingDelimiter<P>
where
P: for<'a> Pattern<'a>,
{
SplitKeepingDelimiter {
searcher: pattern.into_searcher(&self[..]),
start: 0,
saved: None,
}
}
}
impl SplitKeepingDelimiterExt for str {}
#[cfg(test)]
mod test {
use super::SplitKeepingDelimiterExt;
#[test]
fn split_with_delimiter() {
use super::SplitType::*;
let delims = &[',', ';'][..];
let items: Vec<_> = "alpha,beta;gamma".split_keeping_delimiter(delims).collect();
assert_eq!(
&items,
&[
Match("alpha"),
Delimiter(","),
Match("beta"),
Delimiter(";"),
Match("gamma")
]
);
}
#[test]
fn split_with_delimiter_allows_consecutive_delimiters() {
use super::SplitType::*;
let delims = &[',', ';'][..];
let items: Vec<_> = ",;".split_keeping_delimiter(delims).collect();
assert_eq!(&items, &[Delimiter(","), Delimiter(";")]);
}
}
You'll note that I needed to track if something was one of the delimiters or not, but that should be easy to adapt if you don't need it.