I've got a grammar like this to generate a parser:
grammar MyLang;
init: TERMINAL;
TERMINAL: '"' ~('"')* '"';
Using ANTLR4 and the grammar above to parse a simple input like "data", I've got an error:
mismatched input "data" expecting TERMINAL
I've read many questions against this error. But I can't figure out how to solve it. Would you pls let me know if there's anything wrong?
Related
To demonstrate the problem, I'm going to create a simple grammar to merely detect Python-like variables.
I create a virtual environment and install antlr4-python3-runtime in it, as mentioned in "Where can I get the runtime?":
Then, I create a PyVar.g4 file with the following content:
grammar PyVar;
program: IDENTIFIER+;
IDENTIFIER: [a-zA-Z_][a-zA-Z0-9_]*;
NEWLINE: '\n' | '\r\n';
WHITESPACE: [ ]+ -> skip;
Now if I test the grammar with grun, I can see that the grammar detects the variables just fine:
Now I'm trying to write a parser in Python to do just that. I generate the Lexer and Parser, using this command:
antlr4 -Dlanguage=Python3 PyVar.g4
And they're generated with no errors:
But when I use the example provided in "How do I run the generated lexer and/or parser?", I get no output:
What am I not doing right?
There are two problems here.
1. The grammar:
In the line where I had,
program: IDENTIFIER+;
the parser will only detect one or more variables, and it will not detect any newline. The output you see when running grun is the output created by the lexer, that's why newlines are present in the tokens. So I had to replace it with something like this, for the parser to detect newlines.
program: (IDENTIFIER | NEWLINE)+;
2. Printing the output of parser
In PyVar.py file, I created a tree with this line:
tree = parser.program()
But it didn't print its output, nor did I know how to, but the OP's comment on this accepted answer suggests using tree.toStringTree().
Now if we fix those, we can see that it works:
While using XText, I have come across a problem and I am not sure if Antlr4 or XText is at fault or if I'm just missing something. I understand that Antlr4 is not supported by Xtext, but it seems like this particular case should not cause a problem.
Here is a simple Xtext file:
grammar com.github.jsculley.antlr4.Test with org.eclipse.xtext.common.Terminals
generate test "http://www.github.com/jsculley/antlr4/test"
aRule:
name=STRING
;
STRING is defined in the XText rule from org.eclipse.xtext.common.Terminals:
terminal STRING :
'"' ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'|'"') )* '"' |
"'" ( '\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\'|"'") )* "'"
;
The generated Antlr grammar has the following rule:
RULE_STRING : ('"' ('\\' .|~(('\\'|'"')))* '"'|'\'' ('\\' .|~(('\\'|'\'')))* '\'');
The Antlr 3.5.2 tool has no problem with this rule, but the Antlr4 tool spits out the following errors:
error(50): InternalTest.g:102:29: syntax error: '(' came as a complete surprise to me while looking for lexer rule element
error(50): InternalTest.g:102:62: syntax error: '(' came as a complete surprise to me while looking for lexer rule element
error(50): InternalTest.g:102:74: syntax error: mismatched input ')' expecting SEMI while matching a lexer rule
error(50): InternalTest.g:106:25: syntax error: '(' came as a complete surprise to me while looking for lexer rule element
error(50): InternalTest.g:106:36: syntax error: mismatched input ')' expecting SEMI while matching a lexer rule
Antlr4 doesn't like the extra (and seemingly uneccessary) sets of parentheses around the group after each '~' operator. So the question is, is Xtext generating a bad grammar, or is Antlr4 not handling a valid construct?
Xtext generates an Antlr 3.x grammar and Antlr 4 grammars are incompatible.
It seems that ANTLR 4 does not handle parenthesis correctly: Parser issues mutual left recursion error when the left-recursive part of a rule is in parenthesis.
So, just remove useless parenthesis and ANTLR 4 should generate a fully ANLTR 3 compatible parser. I ported PL/SQL grammar from ANTLR 3 -> ANTLR 4. Moreover, ANLTR 4 have a more powerfull parsing algorithm compare to the previous version.
I'm trying to write an antlr4 parser rule that can match the content between some arbitrary string values that are same. So far I couldn't find a method to do it.
For example, in the below input, I need a rule to extract Hello and Bye. I'm not interested in extracting xyz though.
TEXT Hello TEXT
TEXT1 Bye TEXT1
TEXT5 xyz TEXT8
As it is very much similar to an XML element grammar, I tried an example for XML Parser given in ANTLR4 XML Grammar, but it parses an input like <ABC> ... </XYZ> without error which is not what I wanted.
I also tried using semantic predicates without much success.
Could anyone please help with a hint on how to match content that is embedded between same strings?
Thank you!
Satheesh
Not sure how this works out performance wise, because of many many checks the parser has to do, but you could try something like:
token:
start = IDENTIFIER WORD* end = IDENTIFIER { start == end }?
;
The part between the curly braces is a validating semantic predicate. The lexer tokens are self-explanatory, I believe.
The more I think about it, it might be better you just tokenize the input and write an owner parser that processes the input and acts accordingly. Depends of course on the complexity of the syntax.
I have this grammar:
grammar MkSh;
script
: (statement
| targetRule
)*
;
statement
: assignment
;
assignment
: ID '=' STRING
;
targetRule
: TARGET ':' TARGET*
;
ID
: ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_')*
;
WS
: ( ' '
| '\t'
| '\r'
| '\n'
) -> channel(HIDDEN)
;
STRING
: '\"' CHR* '\"'
;
fragment
CHR
: ('a'..'z'|'A'..'Z'|' ')
;
TARGET
: ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'-'|'/'|'.')+
;
and this input file:
hello="world"
target: CLASSES
When running my parser I'm getting this error:
line 3:6 mismatched input ':' expecting '='
line 3:15 mismatched input ';' expecting '='
Which is because of the parser is taking "target" as an ID instead of a TARGET. I want the parser to choose the rule based on the separator character (':' vs '=').
How can I get that to happen?
(This is my first Antlr project so I'm open to anything.)
First, you need to know that the word target is matched as a ID token and not as a TARGET token, and since you have written the rule ID before TARGET, it will always be recognized as ID by the lexer. Notice that the word target completely complies to both ID and TARGET lexer rule, (I'm going to suppose that you are writing a laguage), meaning that the target which is a keyword can also be used as an id. In the book - "The definitive ANTLR reference" there is a subtitle "Treating Keywords As Identifiers" that deals with exactely these kinds of issues. I suggest you take a look at that. Or if you prefer the quick answer the solution is to use lexer modes. Also would be better to split grammar into parser and lexer grammar.
As #cantSleepNow alludes to, you've defined a token (TARGET) that is a lexical superset of another token (ID), and then told the lexer to only tokenize a string as TARGET if it cannot be tokenized as ID. All made more obscure by the fact that ANTLR lexing rules look like ANTLR parsing rules, though they are really quite different beasts.
(Warning: writing off the top of my head without testing :-)
Your real project might be more complex, but in the possibly simplified example you posted, you could defer distinguishing the two to the parsing phase, instead of distinguishing them in the lexer:
id : TARGET
{ complain if not legal identifier (e.g., contains slashes, etc.) }
;
assignment
: id '=' STRING
;
Seems like that would solve the lexing issue, and allow you to give a more intelligent error message than "syntax error" when a user gets the syntax for ID wrong. The grammar remains ambiguous, but maybe ANTLR roulette will happen to make the choice you prefer in the ambiguous case. Of course, unambiguous grammers tend to make for languages that humans find more readable, and now you can see why the classic makefile syntax requires a newline after an assignment or target rule.
In my grammar I'd like variables to be comprised of latin, cyrillic and mandarin characters.
For this purposes I define lexer rule, like this:
CYRILLIC_RANGE: [\u0400–\u04FF];
this is what I see in my ANTLRWorks 2.1 output when I try to run expression against my query:
line 1:4 token recognition error at: 'н'
What am I missing?
I'm not sure what you are missing, as this seems to be working for me here. Have you tried the other range syntax? Both of these should be equivalent.
CYRILLIC_RANGE : [\u0400-\u04FF] ;
CYRILLIC_RANGE : '\u0400'..'\u04FF' ;