Antlr4 Adjacent Token Precedence - antlr4

Question

I'm running into a problem while building a complex grammar. A pet grammar to illustrate it is below:
grammar test;
start: (r1 | r2 | .)*
r1: A B
r2: B C
// A B C are tokens
When the following input occurs:
ABC
The parse tree looks like this:
start
| \
r1 C
| \
A B
But what I actually want is for it to look like this:
start
| \
A r2
| \
B C
I've tried reordering the rules & adding <assoc=right>, but nothing seems to work except removing rule r1, which is incorrect because I expect AB and BC to be valid inputs. What am I missing?
EDIT
It seems the above problem description oversimplifies the actual issue, so I'll give more details:
r3: rA r4 // prefers rA(classB classC) over (rA classB)classC
r4: classB? classC // also used elsewhere other than r3
rA: // rules to build A subtree, ends with classB? in 'some' cases
classB: B1 | B2 | ... | Bm
classC: C1 | C2 | ... | Cn
I've found that the following 'kind of' works:
r3: rA Bx classC | ...
But the following doesn't:
r3: <assoc=right> rA r4 | ... // still builds (rA classB)classC
I'm wondering if there's a way I can build the tree correctly while being able to utilize r4 and its associated code (and avoid having to put another m lines for all instances of B)?
PS. rA is expensive, so expanding B tokens in r3 like above throws performance to the dogs.

Answer 1

The problem I see here is that you tell the parser to produce the parse tree you don't want. If you don't want it then don't specify the grammar that is supposed to produce it.
Similar to what Mike Cargal came up with I think the real solution is to more explicitly specify what you want to see at the end. Here's something that works pretty well (using your initial problem description and MikeC's test input):
parser grammar testparser;
options {
tokenVocab = testlexer;
}
start: (A r2 | r1 | .)*? EOF;
r1: A B;
r2: B C;
lexer grammar testlexer;
A: 'A';
B: 'B';
C: 'C';
WHITE_SPACE: [ \u000B\t\r\n] -> skip ;
OTHER: .;
With the input AB!C2 I get this parse three:
Leaving out C this changes to:
The main change is that you specialise the rule to make BC match in their own sub parse tree, by adding A for the r2 alt and put that first.
Note
Moving that single A down to the r2 rule will break this, because then you tell the parser to create a sub tree with ABC in it (what you don't want).

Answer 2

I suspect that you'll find that you're really fighting the way a recursive descent parser works.
When the parser tries to match rA it will try to match as many input tokens as possible, and is not remotely aware of the parents subsequent rule (i.e. the current rule's "next sibling"). (assoc = right isn't going to make the rule look up to it's parent and next sibling, it's designed for building the correct parse trees for things like exponentiation.)
As such, you'd need to use something like a semantic predicate to "block" the wrong alternative from matching, by looking far enough ahead to determine whether it should actually match r4 (for example).
For your example, the following grammar parses the input "AB!C2" as
grammar test
;
start: r3 EOF;
r3
: rA r4 // prefers rA(classB classC) over (rA classB)classC
;
r4
: classB? classC // also used elsewhere other than r3
;
rA // rules to build A subtree, ends with classB? in 'some' cases
: A C1
| A
| {!( //
_input.get(_input.index()).getType() == C1 || //
_input.get(_input.index()).getType() == C2) //
}? A classB?
;
classB: B1 | B2;
classC: C1 | C2;
A: 'A';
B1: 'B1';
B2: 'B2';
C1: 'C1';
C2: 'C2';
WS: [ \n\r] -> skip;
OTHER: .;
However, that's making an assumption that You can determine an r4 rule match by just looking ahead at the second token that rule would possibly examine. I would suspect that this is completely untenable from a maintenance and understanding standpoint.
You could slightly improve maintainability by including functions using the #parser::members ability. This allows you to have more complex logic for your predicate, while not littering the actual grammar too badly.
grammar test
;
#parser::members {
private int[] laValues = new int[] { C1, C2 };
private boolean r4Follows() {
int la = _input.get(_input.index()).getType();
for (int i = 0; i < laValues.length; i++) {
if (la == laValues[i])
return true;
}
return false;
}
}
start: r3 EOF;
r3
: rA r4 // prefers rA(classB classC) over (rA classB)classC
;
r4
: classB? classC // also used elsewhere other than r3
;
rA // rules to build A subtree, ends with classB? in 'some' cases
: A C1
| A
| A classB {!r4Follows()}?
;
classB: B1 | B2;
classC: C1 | C2;
A: 'A';
B1: 'B1';
B2: 'B2';
C1: 'C1';
C2: 'C2';
WS: [ \n\r] -> skip;
OTHER: .;
However, this is still likely to be a mess to maintain. I think the bottom line is that what you're trying to do, creates an ambiguity as to which rule classB belongs to.
A recursive descent parser, doesn't really see an ambiguity, it just matches the rule it's working on at that time (rA) the best it can and that includes pulling in the classB match. You'll have to introduce a semantic predicate to prevent that, and that's going to be, pretty much, unmanageable.
Addendum:
Against all that is holy in the ANTLR realm, I did work out a way to use the actual r4() rule to test for the match of a rule at the current position of the token stream. I HIGHLY recommend you file this under "stupid ANTLR tricks".
Since this actually attempts the parse of r4, depending on the complexity involved, it could substantially impact performance.
Also, I can make no guarantee that this doesn't still leave some state violated, though it worked on simple examples.
NOTE: The location of the semantic predicate is important as the current index of the token stream is advancing as you progress through the rule.
grammar test
;
#parser::header {import java.util.function.Supplier;}
#parser::members {
private BailErrorStrategy bailStrategy = new BailErrorStrategy();
private boolean ruleMatches(Supplier<ParserRuleContext> rule) {
boolean result = false;
// save state
int idx = _input.index();
int savedState = this.getState();
List<ParseTree> savedChildren = _ctx.children;
_ctx.children = new ArrayList<>();
ANTLRErrorStrategy savedErrStrategy = this.getErrorHandler();
this.setErrorHandler(bailStrategy);
try {
ParserRuleContext attempt = rule.get();
result = true;
} catch (ParseCancellationException pce) {
result = false;
} finally {
// restore state
this.setErrorHandler(savedErrStrategy);
_ctx.children = savedChildren;
this.setState(savedState);
_input.seek(idx);
}
return result;
}
}
start: r3 EOF;
r3
: rA r4? // prefers rA(classB classC) over (rA classB)classC
;
r4
: classB? classC // also used elsewhere other than r3
;
rA // rules to build A subtree, ends with classB? in 'some' cases
: A {!ruleMatches(this::r4)}? classB?
| A D
;
classB: B1 | B2;
classC: C1 | C2;
A: 'A';
B1: 'B1';
B2: 'B2';
C1: 'C1';
C2: 'C2';
D: 'D';
WS: [ \n\r] -> skip;
OTHER: .;
input "AB1C1"
input "ADC1"
input "AC1"
input "AB1"
(I believe it is now required that I go sacrifice an innocent kitten to the ANTLR gods to save my soul for posting that)