How to return different tokens during tokenization? - rust

Question

I'm writing a compiler in Rust and, given a string, part of the logic is to find out of which "kind" the characters are.
I want to return the "value" of each character. For an input of 1 + 2 each character has a "token" and should return something like:
NumberToken, 1
WhiteSpaceToken, ' '
PlusToken, '+'
WhiteSpaceToken, ' '
NumberToken, 1
My function should return something like
enum SyntaxKind {
NumberToken,
WhiteSpaceToken,
PlusToken
}
struct SyntaxToken {
kind: SyntaxKind,
value: // Some general type
}
fn next_token(line: String) -> SyntaxToken {
// Logic goes here
}
How would I implement such logic?

Answer 1

If you're writing out a tokenizer and you wonder what such logic might look like, you can couple these values in the same enum, e.g:
#[derive(Debug)]
enum Token {
Add,
Sub,
Whitespace,
Number(f64),
}
For more, see The Rust Programming Language, "Defining an Enum" on adding data to variants.
… and then you can use a match inside of an iterator to handle it accordingly:
#[derive(Debug)]
enum Token {
Add,
Sub,
Whitespace,
Number(f64),
}
use std::str::Chars;
use std::iter::Peekable;
struct Tokens<'a> {
source: Peekable<Chars<'a>>,
}
pub type TokenIterator<'a> = Peekable<Tokens<'a>>;
impl<'a> Tokens<'a> {
pub fn new(s: &'a str) -> TokenIterator {
Self {
source: s.chars().peekable(),
}
.peekable()
}
}
impl<'a> Iterator for Tokens<'a> {
type Item = Token;
fn next(&mut self) -> Option<Self::Item> {
match self.source.next() {
Some(' ') => Some(Token::Whitespace),
Some('+') => Some(Token::Add),
Some('-') => Some(Token::Sub),
n # Some('0'..='9') => {
let mut number = String::from(n.unwrap());
while let Some(n) = self.source.next_if(char::is_ascii_digit) {
number.push(n);
}
Some(Token::Number(number.parse::<f64>().unwrap()))
}
Some(_) => unimplemented!(),
None => None,
}
}
}
fn main() {
let tokens = Tokens::new("1 + 2");
for token in tokens {
println!("{:?}", token);
}
}
This should then give you:
Number(1.0)
Whitespace
Add
Whitespace
Number(2.0)
Playground