Unexpected token error when parsing token stream with syn::parse2 - rust

Question

I'm attempting to parse a simple Rust file that looks like:
use std::result::Result;
fn init() {
}
In my main function, I read the file to a string, parse it, and then create a token stream which I then pass to syn::parse2. This returns an unexpected token error and I'm not sure why.
fn main() {
if let Err(error) = try_main() {
let _ = writeln!(io::stderr(), "{}", error);
process::exit(1);
}
}
fn try_main() -> Result<(), Error> {
let filepath = PathBuf::from("./src/file-to-parse.rs");
let code = fs::read_to_string(&filepath).map_err(Error::ReadFile)?;
let syntax = syn::parse_file(&code).map_err({
|error| Error::ParseFile {
error,
filepath,
source_code: code,
}
})?;
let token_stream = syntax.to_token_stream();
let item: Result<Item, syn::Error> = syn::parse2(token_stream);
match item {
Ok(mut v) => {
println!("success!");
}
Err(error) => {
println!("{:?}", error.to_compile_error());
}
}
}
The full error it returns me is:
TokenStream [Ident { sym: compile_error, span: bytes(56..58) }, Punct { char: '!', spacing: Alone, span: bytes(56..58) }, Group { delimiter: Brace, stream: TokenStream [Literal { lit: "unexpected token", span: bytes(56..58) }], span: bytes(56..58) }]
What's even more confusing to me is that if I remove the import and keep the function, it parses fine. Also if I remove the function and keep the import it parses fine. It's only when I keep both of them that I get an error. Does this mean that it's an issue with the empty line?

Answer 1

The reason your program throws an error is because your input code has two items (use and fn) but you're attempting to parse the whole thing as a single Item with the syn::parse2 line. This also explains why it works if you remove either the use or the fn.
If you want all the top level items in the code, you can already get that from the File struct that syn::parse_file returns:
fn main() {
let code = "use std::result::Result;
fn init() {
}";
let file = syn::parse_file(&code).unwrap();
dbg!(file.items);
}
[src/main.rs:9] file.items = [
Use(
ItemUse {
attrs: [],
vis: Inherited,
use_token: Use,
leading_colon: None,
tree: Path(
UsePath {
ident: Ident {
sym: std,
span: bytes(5..8),
},
colon2_token: Colon2,
tree: Path(
UsePath {
ident: Ident {
sym: result,
span: bytes(10..16),
},
colon2_token: Colon2,
tree: Name(
UseName {
ident: Ident {
sym: Result,
span: bytes(18..24),
},
},
),
},
),
},
),
semi_token: Semi,
},
),
Fn(
ItemFn {
attrs: [],
vis: Inherited,
sig: Signature {
constness: None,
asyncness: None,
unsafety: None,
abi: None,
fn_token: Fn,
ident: Ident {
sym: init,
span: bytes(30..34),
},
generics: Generics {
lt_token: None,
params: [],
gt_token: None,
where_clause: None,
},
paren_token: Paren,
inputs: [],
variadic: None,
output: Default,
},
block: Block {
brace_token: Brace,
stmts: [],
},
},
),
]
Playground