I've wrote following grammar:
S->S ( S ) S
S->e
e stands for "empty string"
So the language this grammar recognizes includes all strings with matching left and right parenthesis, like (), (()), (()()), etc.
And this grammar is not SLR, here is how I construct SLR parse table:
Augment this grammar:
S1->S
S->S(S)S
S->e
Then construct LR(0) automaton for it:
I0:
S1->.S
S->.S(S)S
S->.e
I1:
S1->S.
S->S.(S)S
...
Please note that for I0, there is no shift or reduce action for input symbol '(', which is the first token of any string this grammar generates.
So SLR parse table will generate error since on state I0, it doesn't know what to do when parsing string: (()).
My question is:
What is the culprit that makes this grammar NOT SLR? Is it the empty string production? That is:
S->e. ?
And in a general sense, can SLR grammar have empty productions? like, S->e in this example.
Thanks.
The answer is OK, if no shift/reduce action is available for current input and there is a shift on empty product, we choose to shift on this empty terminal.
Related
I'm learning ANTLR4 to write a parser for a simple language specific to the app developed by the company. So far I've managed to have working arithmetic operations, logic operations, and conditional branchments. When tackling variables though, I ran into a problem. The language defines multiple mathematical constants, such as 'e'. When parsing variables, the parser would recognize the letter e as the constant and not part of the variable.
Below is a small test grammar I wrote to test this specific case, the euler and letter parser rules are there for visual clarity in the trees below
grammar Test; r: str '\r\n' EOF;
str: euler | (letter)* ;
euler: EULER;
letter: LETTER;
EULER: 'e';
LETTER: [a-zA-Z];
Recognition of different strings with this grammar:
"e"
"test"
"qsdf"
"eee"
I thought maybe parser rule precedence had something to do with it, but whatever order the parser rules are in, the output is the same. Swapping the lexer rules allows for correct recognition of "test", but recognizes "e" using the letter rule and not the euler rule. I also thought about defining EULER as:
EULER: ~[a-zA-Z] 'e' ~[a-zA-Z]
but this wouldn't recognize var a=e correctly. Another rule i have in my lexer is the ELSE: 'else' rule, which recognizes the 'else' keyword, which works and doesn't conflict with rule EULER. This is because antlr recognizes the longest input possible, but then why doesn't it recognize "test" as (r (str (letter t) (letter e) (letter s) (letter t)) \r\n <EOF>) as it would for "qsdf"?
You should not have a lexer rule like LETTER that matches a single letter and then "glue" these letters together in a parser rule. Instead, match a variable (consisting of multiple letters) as a single lexer rule:
EULER: 'e';
VARIABLE: [a-zA-Z]+;
I suggest changing your grammar to this:
grammar Test;
r: str '\n' EOF;
str: euler | WORD ;
euler: EULER;
EULER: 'e';
WORD: [a-zA-Z]+;
It appears you wanted a stand-alone "e" to be an euler element, and any other word to be a letter element, but that's not what you coded. Your grammar is doing exactly what you told it to do: Match every "e" as an EULER token (and therefore an euler element), and any other letter as a LETTER token (and therefore a letter element), and build strs out of those two tokens.
An ANTLR4 lexer tokenizes the input stream, trying to build the longest tokens possible, and processing the tokenization rules in the order you code them. Thus EULER will capture every "e", and LETTER will capture "a"-"d", "f"-"z", and "A"-"Z". An ANTLR4 parser maps the stream of tokens (from the lexer) into elements based on the order of tokens and the rules you code. Since the parser will never get a LETTER token for "e", your str elements will always get chopped apart at the "e"s.
The fix for this is to code a lexer rule that collects sequences of letters that aren't stand-alone "e"s into a LETTER token (or, as #pavel-ganelin says, a WORD), and to present that to the parser instead of the individual letters. It's a little more complicated than that, though, becuase you probably want "easy" to be the WORD "easy", not an EULER ("e") followed by the WORD "asy". So, you need to ensure that the "e" starting a string of letters isn't captured as an EULER token. You do that by ensuring that the WORD lexer rule comes before the EULER rule, and that it ignores stand-alone "e"s:
grammar Test;
r: str '\r\n' EOF;
str: euler | word ;
euler: EULER;
word: WORD;
WORD: ('e' [a-zA-Z]+) | [a-zA-Z]+;
EULER: 'e';
I'm currently relearning ANTLR and I'm having a bit of an issue with my grammar and parsing is. I'm editing it in IntelliJ IDEA with the ANTLR plugin and I'm using ANTLR version 4.9.2.
My grammar is as follows
grammar Pattern;
pattern:
patternName
patternMeaning
patternMoves;
patternName : 'Name:' NAME ;
patternMeaning : 'Meaning:' NAME ;
patternMoves : 'Moves:' (patternStep)+ ;
patternStep : 'Turn' angle stance;
stance : 'Walking Stance';
angle : ('90'|'180'|'270'|'360') '°' 'anti-'? 'clockwise';
NAME : WORD (' ' WORD)*;
fragment WORD : [a-zA-Z]+;
WS: [ \t\r\n]+ -> skip;
now when I try and parse the following text, I get the following error line 2:9 mismatched input 'clockwise Walking Stance' expecting {'anti-', 'clockwise'}
Name: Il Jang
Meaning: Heaven and light
Moves:
Turn 90° clockwise Walking Stance
However, if I change the text to the below it works without any issues. How can I tweak my grammar to allow me to parse it on one line?
Name: Il Jang
Meaning: Heaven and light
Moves:
Turn 90° clockwise
Walking Stance
Your problem is that clockwise Walking Stance is a valid NAME, so it's interpreted as such rather than as an instance of the clockwise keyword followed by the NAME Walking Stance. Adding a line break fixes this because line breaks can't appear in names.
To fix this, you should turn WORD into a lexer rule and NAME into a parser rule. That way the name rule will only be tried in places where the parser actually expects a name, so it won't try to interpret clockwise as part of a name. And the WORD rule won't eat keywords because the match produced by the WORD rule won't be longer than the keyword, so the keyword wins.
If this is your entire grammar, then there are no lexer rules defining the handling of whaitespace. In fact, the are no explicit lexer rules. (ANTLR will create implicit lexer rules for any literal strings in your parser rules (unless the match an already define grammar rule.))
Your grammar is essentially (in ANTLR’s perception)
grammar Pattern;
patternMoves : T_1 (patternStep)+ ;
patternStep : T_2 angle stance;
stance : T_3;
angle : (T_4|T_5|T_6|T_7) T_8 T_9? T_10;
T_1: ‘Moves:’;
T_2: ‘Turn’;
T_3: 'Walking Stance';
T_4: '90';
T_5: '180';
T_6: '270';
T_7: '360';
T_8: '°';
T_9: 'anti-';
T_10: 'clockwise';
ANTLR’s processing takes a stream of characters, passes them to a lexer, which must decide what to do with all characters (even whitespace). The lexer produces a stream of tokens that the parser rules process.
You need some lexer rule that prescribes how to handle whatespace:
WS: [ \t\r\n]+ -> skip;
Is a common way of handling this. It tokenized all whitespace as a WS token, but then skips handing that token to the parser. (This is very handy as you won’t have to sprinkle WS or WS? items all through your grammar where whitespace is expected.
That your plugin accepts you input would imply to me that it may be treating each line of input as a new parse.
The rule I am trying to match is: hello followed by a sequence of characters. If that sequence contains an alphabet in it, that should match the str rule, else it should match the num rule.
For e.g.
hello123 - 123 should be matched by num rule
hello1a3 - 1a3 should be matched by the str rule
The grammar I wrote is below:
grammar Hello;
r: 'hello'seq;
// seq: str | integ;
seq: num | str;
num : DIGITS;
str : CHARS;
DIGITS: [0-9]+;
CHARS : [0-9a-zA-Z]+;
WS : [ \t\n\r]+ -> skip;
While trying to visualize the parse tree (using grun) (against the first input example above) I got the below parse tree:
However if the input had space in between there was no problem. Please explain why the error.
Lexing in ANTLR (as well as most lexer generators) works according to the maximum munch rule, which says that it always applies the lexer rule that could match the longest prefix of the current input. For the input hello123, the rule 'hello' would match hello, whereas the rule CHARS would match the entire input hello123. Therefore CHARS produces the longer match and is chosen over 'hello'.
If your CHARS and DIGITS tokens can only appear after a 'hello' token, you can use lexer modes to make it so that these rules are only available after a 'hello' has been matched.
Otherwise, to get the behaviour you want, your best bet would probably be to create a single lexer rule that matches 'hello' [0-9a-zA-Z]* and then take apart the tokens generated by that in a separate step. Though it all depends on why you need this.
Is there a common/defined design pattern which will help program an evaluator for boolean expressions.
I am writing a string matching algorithm for such expressions and looking for a design pattern which will help structure the algorithm.
Sample Expected Strings -
"nike AND (tshirt OR jerseys OR jersey OR tshirts OR (t AND shirt)) AND black"
Your expression is in the infix notation. To evaluate it, convert it to the postfix notation.
Infix expression looks like:
<operand><operator><operand>
Postfix expression looks like:
<operand><operand><operator>
You can convert your expression using Shunting Yard Algorithm.
As the expression is converted, evaluate it using this approach (pseudocode):
Begin
for each character ch in the postfix expression, do
if ch is an operator ⨀ , then
a := pop first element from stack
b := pop second element from the stack
res := b ⨀ a
push res into the stack
else if ch is an operand, then
add ch into the stack
done
return element of stack top
End
I don't know of a design pattern per se which would fit your problem, but if your programming language has regex support, we can easily enough write a pattern such as this:
(?=.*\bnike\b)(?=.*\b(?:tshirts?|jerseys?|t\b.*\bshirt|shirt\b.*\bt))(?=.*\bblack\b).*
The pattern can be explained as:
(?=.*\bnike\b) match "nike" AND
(?=.*\b(?:tshirts?|jerseys?|t\b.*\bshirt|shirt\b.*\bt))
match tshirt(s), jersey(s) or "t" and "shirt" AND
(?=.*\bblack\b) match "black"
.* then consume the entire line
Demo
I am in the process of converting an antlr3 to antlr4 grammar. I have stripped out all the syntactic predicates. I am struggling to make a correct conversion of this
relaxed_date_month_first
: relaxed_day_of_week? relaxed_month COMMA? WHITE_SPACE relaxed_day_of_month (relaxed_year_prefix relaxed_year)?
-> ^(EXPLICIT_DATE relaxed_day_of_month relaxed_month relaxed_day_of_week? relaxed_year?)
to antlr4 grammar.Everytime the antlr4 tool runs into "->" character it says "extraneous input '->' expecting {TOKEN_REF, RULE_REF...ACTION}".
How do I fix this?
ANTLR4 has no tree-rewriting. So, remove -> ... entirely:
relaxed_date_month_first
: relaxed_day_of_week? relaxed_month COMMA? WHITE_SPACE relaxed_day_of_month (relaxed_year_prefix relaxed_year)?
;