Non-greedy matching of tokens from lists in ANTLR4 - antlr4

Question

In a previous question with a simple grammar, I learned to handle IDs that can include keywords from a keyword list. My actual grammar is a little more complex: there are several lists of keywords that are expected in different types of sentences. Here's my attempt at a simple grammar that tells the story:
grammar Hello;
file : ( fixedSentence )* EOF ;
sentence : KEYWORD1 ID+ KEYWORD2 ID+ PERIOD
| KEYWORD3 ID+ KEYWORD4 ID+ PERIOD;
KEYWORD1 : 'hello' | 'howdy' | 'hi' ;
KEYWORD2 : 'bye' | 'goodbye' | 'adios' ;
KEYWORD3 : 'dear' | 'dearest' ;
KEYWORD4 : 'love' | 'yours' ;
PERIOD : '.' ;
ID : [a-z]+ ;
WS : [ \t\r\n]+ -> skip ;
So the sentences I want to match are, for example:
hello snickers bar goodbye mars bar.
dear peter this is fun yours james.
and that works great. But I also want to match sentences that contain keywords that would not be expected to terminate the ID+ block. For example
hello hello kitty goodbye my dearest raggedy ann and andy.
hello fist appears as KEYWORD1 and then just following as part of that first ID+. Following the example of the above linked question, I can fix it like this:
// ugly solution:
fixedSentence : KEYWORD1 a=(ID|KEYWORD1|KEYWORD3|KEYWORD4)+ KEYWORD2 b=(ID|KEYWORD1|KEYWORD2|KEYWORD3|KEYWORD4)+ PERIOD
| KEYWORD3 a=(ID|KEYWORD1|KEYWORD2|KEYWORD3)+ KEYWORD4 b=(ID|KEYWORD1|KEYWORD2|KEYWORD3|KEYWORD4)+ PERIOD;
which works and does exactly what I'd like. In my real language, I've got hundreds of keyword lists, to be used in different types of sentences, so if I try for this approach, I'll certainly make a mistake doing this, and when I create new structures in my language, I have to go back and edit all the others.
What would be nice is to do non-greedy matching from a list, following the ANTLR4 book's examples for comments. So I tried this
// non-greedy matching concept:
KEYWORD : KEYWORD1 | KEYWORD2 | KEYWORD3 | KEYWORD4 ;
niceID : ( ID | KEYWORD ) ;
niceSentence : KEYWORD1 niceID+? KEYWORD2 niceID+? PERIOD
| KEYWORD2 niceID+? KEYWORD3 niceID+? PERIOD;
which I think follows the model for comments (e.g. given on p.81 of the book):
COMMENT : '/*' .*? '*/' -> skip ;
by using the ? to suggest non-greediness. (Though the example is a lexer rule, does that change the meaning here?) fixedSentence works but niceSentence is a failure. Where do I go from here?
To be specific, the errors reported in parsing the hello kitty test sentence above are,
Testing rule sentence:
line 1:6 extraneous input 'hello' expecting ID
line 1:29 extraneous input 'dearest' expecting {'.', ID}
Testing rule fixedSentence: no errors.
Testing rule niceSentence:
line 1:6 extraneous input 'hello' expecting {ID, KEYWORD}
line 1:29 extraneous input 'dearest' expecting {KEYWORD2, ID, KEYWORD}
line 1:57 extraneous input '.' expecting {KEYWORD2, ID, KEYWORD}
And if it helps to see the parse trees, here they are.

Answer 1

Recognize that the parser is ideally suited to handling syntax, i.e., structure, and not semantic distinctions. Whether a keyword is an ID terminator in one context and not in another, both being syntactically equivalent, is inherently semantic.
The typical ANTLR approach to handling semantic ambiguities is to create a parse tree recognizing as many structural distinctions as reasonably possible, and then walk the tree analyzing each node in relation to the surrounding nodes (in this case) to resolve ambiguities.
If this resolves to your parser being
sentences : ( ID+ PERIOD )* EOF ;
then your sentences are essentially free form. The more appropriate tool might be an NLP library - Stanford has a nice one.
Additional
If you define your lexer rules as
KEYWORD1 : 'hello' | 'howdy' | 'hi' ;
KEYWORD2 : 'bye' | 'goodbye' | 'adios' ;
KEYWORD3 : 'dear' | 'dearest' ;
KEYWORD4 : 'love' | 'yours' ;
. . . .
KEYWORD : KEYWORD1 | KEYWORD2 | KEYWORD3 | KEYWORD4 ;
the lexer will never emit a KEYWORD token - 'hello' is consumed and emitted as a KEYWORD1 and the KEYWORD rule is never evaluated. Since the parse tree fails to identify the type of the tokens (apparently) it is not very illuminating. Dump the token stream to see what the lexer is actually doing
hello hello kitty goodbye my dearest ...
KEYWORD1 KEYWORD1 ID KEYWORD2 ID KEYWORD3 ...
If you place the KEYWORD rule before the others, then the lexer is going to only emit KEYWORD tokens.
Changing to parser rules
niceID : ( ID | keyword ) ;
keyword : KEYWORD1 | KEYWORD2 | KEYWORD3 | KEYWORD4 ;
will allow this very limited example to work.