Precedence of alternation vs sequencing in ANTLR4 - antlr4

Question

I believed that sequencing (implicitly given by order of subrules) had a higher priority in ANTLR4 parser than alternation (explictly given by | character), meaning that
a : x | y z ;
was semantically identical to
a : x | ( y z) ;
Looking in the ANTLR4 book and searching generally I can't find a clear statement of this but it seems reasonable, however given a rule
expression :
pmqident
|
constant
|
[snip]
|
'(' scalar_subquery ')'
|
unary_operator expression // this is unbracketed
|
expression binary_operator expression
[snip]
;
and I feed it this select - 2 / 3 I get this parse tree
whereas if I just add brackets around unary_operator expression and change absolutely nothing else, to get this
expression :
[snip]
'(' scalar_subquery ')'
|
( unary_operator expression ) // brackets added here
|
expression binary_operator expression
[snip]
;
and give it the same SQL, I get this
What am I misunderstanding?
(BTW and separately, the freaky parse of "- 2 / 3" into "(- ( 2 / 3))" is actually the one I want. That's how MSSQL does it. Mad world)
------
Ok, to reproduce (works for me), not utterly minimal but heavily stripped code. File is named MSSQL.g4:
grammar MSSQL;
expression :
constant
|
unary_operator expression // bracket/unbracket this
|
expression binary_operator expression
;
constant : INTEGER_CONST ;
INTEGER_CONST : [0-9]+ ;
binary_operator :
arithmetic_operator
;
arithmetic_operator :
subtract
|
divide
;
add_symbol : PLUS_SIGN ;
subtract : MINUS_SIGN ;
divide : DIVIDE_SIGN ;
unary_operator :
SIGN
;
SIGN : PLUS_SIGN | MINUS_SIGN ;
DIVIDE_SIGN : '/' ;
PLUS_SIGN : '+' ;
MINUS_SIGN : '-' ;
SKIPWS : [ \t\r\n]+ -> skip ; // skip spaces, tabs, newlines
The DOS crud to compile it (relevant parts given):
set CurrDir=%~dp0
set CurrDir=%CurrDir:~0,-1%
cd %CurrDir%
java org.antlr.v4.Tool -Werror -o %CurrDir%\MSSQL MSSQL.g4
IF %ERRORLEVEL% NEQ 0 goto problem
javac %CurrDir%\MSSQL\MSSQL*.java
IF %ERRORLEVEL% NEQ 0 goto problem
cd ./MSSQL
echo enter sql...
java org.antlr.v4.gui.TestRig MSSQL expression -gui -trace -tokens
input is - 2 / 3
Running on win2k8R2, versions of bits are as follows
C:\Users\jan>java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
C:\Users\jan>java org.antlr.v4.Tool
ANTLR Parser Generator Version 4.5.1
Anything else needed? Can anyone reproduce?
Frankly I'm struggling to believe this is a bug. It's just too elemental.
FYI I found this originally not by bracketing/unbracketing but by hoisting the body of a subrule into rule, and noticed behaviour changed.

Answer 1

This answer is being written in the context of antlr/antlr4#564 not being fixed.
During the code generation process, ANTLR looks for a few specific patterns when rewriting left-recursive rules to work in a recursive-descent parser.
Consider the following rule:
expression
: INT
| '++' expression
| expression '++'
| expression '+' expression
;
Suffix: Top-level alternatives which start with a recursive invocation. In the example, the alternative expression '++' falls into this category.
Prefix: Top-level alternatives which end with a recursive invocation. In the example, the alternative '++' expression falls into this category.
Binary: Top-level alternatives which start and end with a recursive invocation. In the example, the alternative expression '+' expression falls into this category.
Other: Everything else. In the example, the alternative INT falls into this category.
When matching these patterns, no simplifications are performed. This includes removing otherwise-unnecessary parentheses, which is the basis of issue antlr/antlr4#564.
By including parentheses around a top-level alternative in a left-recursive rule, you force the alternative to be treated as Other. For alternatives that would normally be Suffix or Binary, this results in a compilation error due to left recursion that was not eliminated. For Prefix alternatives (which you have), the grammar still compiles but changes behavior because the alternative is treated as a primary expression instead of an operator which overrides its original position in the operator precedence sequence.
Note that including parentheses around a top-level alternative which was already in the Other category will not change behavior at all. Likewise, including parentheses around an alternative in a rule which is not left-recursive will not change behavior.