Parsing key value IPV6 pair - antlr4

Question

I have following input format IP:FE80:CD00::211E:729C to parse.
After parsing, I want Key as IP: and value as FE80:CD00::211E:729C
I have defined following grammar
grammar IPV6;
keyValue : KEY ip_v6_address;
ip_v6_address
: h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| '::' h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| h16? '::' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| ((h16 ':')? h16)? '::' h16 ':' h16 ':' h16 ':' ls32
| (((h16 ':')? h16 ':')? h16)? '::' h16 ':' h16 ':' ls32
| ((((h16 ':')? h16 ':')? h16 ':')? h16)? '::' h16 ':' ls32
| (((((h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16)? '::' ls32
| ((((((h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16)? '::' h16
;
h16
: hexdig hexdig hexdig hexdig
| hexdig hexdig hexdig
| hexdig hexdig
| hexdig
;
hexdig
: digit
| (A | B | C | D | E | F)
;
ls32
: h16 ':' h16
| ip_v4_address
;
ip_v4_address
: dec_octet '.' dec_octet '.' dec_octet '.' dec_octet
;
dec_octet
: digit
| non_zero_digit digit
| D1 digit digit
| D2 (D0 | D1 | D2 | D3 | D4) digit
| D2 D5 (D0 | D1 | D2 | D3 | D4 | D5)
;
digit
: D0
| non_zero_digit
;
non_zero_digit
: D1
| D2
| D3
| D4
| D5
| D6
| D7
| D8
| D9
;
D0 : '0';
D1 : '1';
D2 : '2';
D3 : '3';
D4 : '4';
D5 : '5';
D6 : '6';
D7 : '7';
D8 : '8';
D9 : '9';
A : 'a'|'A';
B : 'b'|'B';
C : 'c'|'C';
D : 'd'|'D';
E : 'e'|'E';
F : 'f'|'F';
KEY: '['? STRING SPACE* STRING']'?':';
fragment SPACE : ' ';
fragment STRING: [a-zA-Z0-9/._-]+;
WS : [ \t\r\n] + -> skip;
The above grammar gives me following tokens after running against the above example
[TOKENS]
KEY 'IP:'
KEY 'FE80:'
KEY 'CD00:'
':' ':'
KEY '211E:'
D7 '7'
D2 '2'
D9 '9'
C 'C'
EOF '<EOF>'
[PARSE-TREE]
line 1:3 mismatched input 'FE80:' expecting {'::', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', A, B, C, D, E, F}
(keyValue IP:
(ip_v6_address FE80: CD00: : 211E: 7 2 9 C))
I want to have key value pairs as output and not sure if I am writing the correct grammar. I problem that I am facing is that separator ':' can exit in value as well.
Any pointers how to fix the grammar ?

Answer 1

It doesn't work because of overlapping lexer rules (multiple rules matching the same input).
The F char from FE80: is not being tokenised as a hex digit (the F lexer rule). However, the entire chunk FE80: is being tokenised as a KEY token.
You must realise that the lexer operates independently from the parser. The parser might be trying to match a certain token, the lexer does not "listen" to this. The lexer follows 2 very simple rules:
try to match as much characters as possible for a single token
when two or more tokens match the same characters, the rule defined first "wins"
Because of these rules, the input F is tokenised as an F token, but input like FE is tokenised as a KEY token.
The solution is to move the construction of a KEY from the lexer to a key parser rule as shown below:
grammar IPV6;
key_value
: key ':' ip_v6_address
;
key
: '[' string ']'
| string
;
ip_v6_address
: h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| '::' h16 ':' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| h16? '::' h16 ':' h16 ':' h16 ':' h16 ':' ls32
| ((h16 ':')? h16)? '::' h16 ':' h16 ':' h16 ':' ls32
| (((h16 ':')? h16 ':')? h16)? '::' h16 ':' h16 ':' ls32
| ((((h16 ':')? h16 ':')? h16 ':')? h16)? '::' h16 ':' ls32
| (((((h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16)? '::' ls32
| ((((((h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16 ':')? h16)? '::' h16
;
h16
: hexdig hexdig hexdig hexdig
| hexdig hexdig hexdig
| hexdig hexdig
| hexdig
;
hexdig
: digit
| (A | B | C | D | E | F)
;
ls32
: h16 ':' h16
| ip_v4_address
;
ip_v4_address
: dec_octet '.' dec_octet '.' dec_octet '.' dec_octet
;
dec_octet
: digit
| non_zero_digit digit
| D1 digit digit
| D2 (D0 | D1 | D2 | D3 | D4) digit
| D2 D5 (D0 | D1 | D2 | D3 | D4 | D5)
;
digit
: D0
| non_zero_digit
;
non_zero_digit
: D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9
;
string
: (STRING_ATOM | hexdig)+
;
D0 : '0';
D1 : '1';
D2 : '2';
D3 : '3';
D4 : '4';
D5 : '5';
D6 : '6';
D7 : '7';
D8 : '8';
D9 : '9';
A : [aA];
B : [bB];
C : [cC];
D : [dD];
E : [eE];
F : [fF];
STRING_ATOM : [g-zG-Z/._-];
WS : [ \t\r\n] + -> skip;
resulting in the following parse tree: