lexing multiline define statements with antlr4 - antlr4

Question

I am trying to write a lexer to do preprocessing, which can handle multi-line #define statements. For example, the following input, where the multi-line definition is broken by an subsequent empty line (may contain white spaces though):
aa(bb);
#define XX pqr
#define YY pqr \
+abc
class p(XX,YY,zz);
endclass
The first step is to tokenize the input stream, where for any definition the value will be obtained as one string token. eg, for YY, I am trying to get "pqr+abc" as its string value. I have written the following lexer for tokenizing:
DEF: '#define' -> pushMode(def_mode);
ID: Letter (Letter | DecDigit)* ;
COMMENT : '/*' .*? '*/' -> channel(HIDDEN) ;
LINE_COMMENT : '//' ~('\n'|'\r')* NL -> channel(HIDDEN);
WS: ( ' ' |'\t' | NL )+ -> channel(HIDDEN) ;
SEMICOLN: ';' ;
COMMA: ',' ;
OB: '(' ;
CB: ')' ;
PLUS: '+' ;
fragment NL : '\r'? '\n' ;
fragment DecDigit: '0'..'9' ;
fragment Letter: 'A'..'Z' | 'a'..'z' | '_' ;
mode def_mode;
STR2: '\r'? '\n' -> popMode;
STR1: ~('\n'|'\r')* '\r'? '\n' ;
The above lexer gives the following tokens for the #define lines:
[#10,26:32='#define',<2>,6:0]
[#11,33:42=' YY pqr \\n',<13>,6:7]
[#12,43:51=' +abc\n',<13>,7:0]
[#13,52:52='\n',<12>,8:0]
[#14,53:57='class',<3>,9:0]
The above tokens are obtained only if there is an "empty" line after the #define lines. If there are some whitespace in that line, ie it is not really empty, then the mode is not exited. Here are the tokens when the line has whitespace:
[#10,26:32='#define',<2>,6:0]
[#11,33:42=' YY pqr \\n',<13>,6:7]
[#12,43:51=' +abc\n',<13>,7:0]
[#13,52:55=' \n',<13>,8:0]
[#14,56:74='class p(XX,YY,zz);\n',<13>,9:0]
[#15,75:83='endclass\n',<13>,10:0]
[#16,84:84='\n',<12>,11:0]
[#17,85:84='<EOF>',<-1>,12:0]
Also, the lexer is not joining the two lines. How to fix these errors?

Answer 1

I am not an expert (one grammar per year) and not a fan of mode, and don't like to make too much processing in the lexer. The parser has much more capabilities. Look at this :
grammar Question;
/* Parsing preprocessor #define */
program
: statement+
;
statement
: aClass
| function
| preprocessor
;
aClass
: 'class'
classDef // classBody
'endclass'
;
classDef
: ID '(' list ')' ';'
;
function
: ID '(' list ')' ';'
;
preprocessor
: DEFINE ID replacement
{System.out.println($DEFINE.text + " value=" + $ID.text + " -> replaced by " + $replacement.text);}
;
replacement
: expr+
;
expr
: ID
| ID '+' ID
;
list
: ID ( ',' ID )*
;
ID : LETTER ALPHAMERIC* ;
DEFINE
: '#' 'define' ;
COMMENT
: '/*' .*? '*/' -> channel(HIDDEN) ;
LINE_COMMENT
: '//' ~('\r' | '\n')* -> channel(HIDDEN) ;
WS : [ \t\r\n]+ -> channel(HIDDEN) ; // keep spaces in $<rule>.text
//WS : [ \t\r\n]+ -> skip ;
CONTINUATION
// if you want to keep the exact value including NL :
// : '\\' '\r'? '\n' -> channel(HIDDEN) ;
// to discard the continuation character :
: '\\' '\r'? '\n' -> skip ; // ignored as in Ruby, concatenates two lines
fragment LETTER : [a-zA-Z_] ;
fragment DIGIT : [0-9] ;
fragment ALPHAMERIC : LETTER | DIGIT ;
With the input data.txt :
/* function
call */
aa(bb);
#define XX pqr
#define YY long replacement value
// multiline :
#define ZZ stu \
+abc
class p(XX,YY,zz);
endclass
#define WW vwx \
+def
// preceding line contains 10 spaces
$ hexdump -C data.txt
...
000000b0 2b 64 65 66 0a 20 20 20 20 20 20 20 20 20 20 0a |+def. .|
000000c0 2f 2f 20 70 72 65 63 65 64 69 6e 67 20 6c 69 6e |// preceding lin|
000000d0 65 20 63 6f 6e 74 61 69 6e 73 20 31 30 20 73 70 |e contains 10 sp|
000000e0 61 63 65 73 |aces|
000000e4
the output is :
$ grun Question program -tokens data.txt
[#0,0:26='/* function\n call */',<COMMENT>,channel=1,1:0]
[#1,27:27='\n',<WS>,channel=1,2:15]
[#2,28:29='aa',<ID>,3:0]
...
[#8,36:42='#define',<DEFINE>,4:0]
[#9,43:43=' ',<WS>,channel=1,4:7]
[#10,44:45='XX',<ID>,4:8]
[#11,46:46=' ',<WS>,channel=1,4:10]
[#12,47:49='pqr',<ID>,4:11]
[#13,50:50='\n',<WS>,channel=1,4:14]
...
#define value=XX -> replaced by pqr
#define value=YY -> replaced by long replacement value
#define value=ZZ -> replaced by stu +abc
#define value=WW -> replaced by vwx +def
If you use the skip version of WS, you'll have pqr+abc but also longreplacementvalue. I leave this point to your astuteness.