Zip iterables with Optional and Non Optional parameter in macro - rust

Question

For the testing part of my lexer, I came up with a simple macro that let met define the expected token type (enum) and the token literal (string):
macro_rules! token_test {
($($ttype:ident: $literal:literal)*) => {
{
vec!($($ttype,)*).iter().zip(vec!($($literal,)*).iter())
}
}
}
and then I can use it like this:
for (ttype, literal) in token_test! {
Let: "let" Identifier: "five" Assign: "=" Int: "5" Semicolon: ";"
} {
//...
}
However, this is a little bit verbose and we don't need to specify the literal for most of the token since I have another macro that transforms an enum variant into a string (eg: Let -> "let").
So what I hope to do is something like:
for (ttype, literal) in token_test! {
Let Identifier: "five" Assign Int: "5" Semicolon
} {
//...
}
And if I understood properly, I can use optional parameters to match either TYPE: LITERAL or TYPE. Maybe something like:
macro_rules! token_test {
($($ttype:ident$(: $literal:literal)?)*) => {
{
//...
}
}
}
So then my question is is there a way to build Vector out of this?
To be more clear:
In the case of no literal passed, it should add the string representation of my enum (eg: Let -> "let")
In the case of literal passed, it should add the literal directly

Answer 1

Made it work with the following macro (any improvement welcomed):
macro_rules! token_test {
($($ttype:ident$(: $literal:literal)?)*) => {
vec!($($ttype,)*).iter().zip(vec!(
$(
{
let mut literal = $ttype.as_str().unwrap();
$(literal = $literal;)?
literal
}
),*).iter())
}
}
This 'iterates' over the literal macro arguments and initially set the value of the as_str which transform a enum variant to a string. Then if the $literal is defined, it replaces the local literal value to that. And finally, it returns the local literal variable.
Improvement
macro_rules! some_or_none {
() => { None };
($entity:literal) => { Some($entity) }
}
macro_rules! token_test {
($($ttype:ident$(: $literal:literal)?)*) => {
vec!($($ttype,)*).iter().zip(vec!($(
some_or_none!($($literal)?).unwrap_or($ttype.as_str().unwrap())
),*))
}
}
Removed some unnecessary scopes, the second .iter(), and added some_or_none macro. With this way I don't need to do the as_str if there is a literal provided.
Further improvement
In the above example, there are two macros that are provided. One is clearly a "private" macro, because its existence is only useful for the implementation of the other one. However, there is a small catch about how macro exports work. Unlike functions, macros cannot access a macro that was defined in the same scope, but which are not accessible from the caller. See this playground example. This is not a problem if you don't intend to export that macro, which is possible since its only purpose is to be used in a test suite. However, you might still want to expose it publicly at a crate level, without exposing some_or_none!. The conventional way to do this is to integrate some_or_none! inside the token_test! macro, by prepending it with #:
macro_rules! token_test {
(#some_or_none) => {
None
};
(#some_or_none $entity:literal) => {
Some($entity)
};
($($ttype:ident $(: $literal:literal)?)*) => {
vec!($($ttype,)*)
.iter()
.zip(vec!($(
token_test!(#some_or_none $($literal)?)
.unwrap_or($ttype.as_str().unwrap())
),*))
};
}
With this version, you can safely export test_token without any fears as shown in this playground.
Little bit more
original idea from steffahn on the Rust Forum
There is another similar way to solve that and without involving unwrap_or, instead of wrapping into an Option in the some_or_none, we can actually create two branches that take either TYPE + LITERAL or TYPE, like so:
macro_rules! token_test {
(#ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
(#ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
($($ttype:ident $(: $literal:literal)?)*) => {
vec!($($ttype,)*)
.iter()
.zip(vec![$(token_test!(#ttype_or_literal $ttype$(: $literal)?)),*])
};
}
And again
As I only need an iterable than can be deconstructed as (type, iterable), an array of pair is enough:
macro_rules! token_test {
(#ttype_or_literal $ttype:ident) => { $ttype.as_str().unwrap() };
(#ttype_or_literal $ttype:ident: $literal:literal) => { $literal };
($($ttype:ident $(: $literal:literal)?)*) => {
[$(($ttype, token_test!(#ttype_or_literal $ttype$(: $literal)?))),*]
};
}
so no more vec and no more zip.
A Smart trick
A user on the Rust forum gave this potential trick involving ignoring the second argument if it exists. I made the solution a little bit more compact by not having two macros:
macro_rules! token_test {
(#ignore_second $value:expr $(, $_ignored:expr)? $(,)?) => { $value };
($($ttype:ident $(: $literal:literal)?)*) => {
[$(($ttype, token_test!(#ignore_second $($literal,)? $ttype.as_str().unwrap()))),*]
};
}