Why is ending parenthesis not recognizes as valid in my antlr4 grammar? - antlr4

I am writing a DSL with ANTLR4, now I have a problem with ending right parenthesis. Why is this command not valid
This is the command:
set(buffer,variableX|"foo");
the parse tree with the error
Here is my grammar
grammar Expr;
prog: expr+ EOF;
expr:
statement #StatementExpr
|NOT expr #NotExpr
| expr AND expr #AndExpr
| expr (OR | XOR) expr #OrExpr
| function #FunctionExpr
| LPAREN expr RPAREN #ParenExpr
| writeCommand #WriteExpr
;
writeCommand: setCommand | setIfCommand;
statement: ID '=' getCommand NEWLINE #Assign;
setCommand: 'set' LPAREN variableType '|' parameter RPAREN SEMI;
setIfCommand: 'setIf' LPAREN variableType '|' expr '?' parameter ':' parameter RPAREN SEMI;
getCommand: getFieldValue | getInstanceAttribValue|getInstanceAttribValue|getFormAttribValue|getMandatorAttribValue;
getFieldValue: 'getFieldValue' LPAREN instanceID=ID COMMA fieldname=ID RPAREN;
getInstanceAttribValue: 'getInstanceAttrib' LPAREN instanceId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getFormAttribValue: 'getFormAttrib' LPAREN formId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getMandatorAttribValue: 'getMandatorAttrib' LPAREN mandator=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
twoParameterList: parameter '|' parameter;
parameter:variableType | constType;
pdixFuncton:ID;
constType:
ID
| '"' ID '"';
variableType:
valueType
|instanceType
|formType
|bufferType
|instanceAttribType
|formAttribType
|mandatorAttribType
;
valueType:'value' COMMA parameter (COMMA functionParameter)?;
instanceType: 'instance' COMMA instanceParameter;
formType: 'form' COMMA formParameter;
bufferType: 'buffer' COMMA ID '|' parameter;
instanceParameter: 'instanceId'
| 'instanceKey'
| 'firstpenId'
| 'lastpenId'
| 'lastUpdate'
| 'started'
;
formParameter: 'formId'
|'formKey'
|'lastUpdate'
;
functionParameter: 'lastPen'
| 'fieldGroup'
| ' fieldType'
| 'fieldSource'
| 'updateId'
| 'sessionId'
| 'icrConfidence'
| 'icrRecognition'
| 'lastUpdate';
instanceAttribType:('instattrib' | 'instanceattrib') COMMA attributeType;
formAttribType:'formattrib' COMMA attributeType;
mandatorAttribType: 'mandatorattrib' COMMA attributeType;
attributeType:ID '#' ID;
function:
commandIsSet #IsSet
| commandIsEmpty #IsEmpty
| commandIsEqual #IsEqual
;
commandIsSet: IS_SET LPAREN parameter RPAREN;
commandIsEmpty: IS_EMPTY LPAREN parameter RPAREN;
commandIsEqual: IS_EQUAL LPAREN twoParameterList RPAREN;
LPAREN : '(';
RPAREN : ')';
LBRACE : '{';
RBRACE : '}';
LBRACK : '[';
RBRACK : ']';
SEMI : ';';
COMMA : ',';
DOT : '.';
ASSIGN : '=';
GT : '>';
LT : '<';
BANG : '!';
TILDE : '~';
QUESTION : '?';
COLON : ':';
EQUAL : '==';
LE : '<=';
GE : '>=';
NOTEQUAL : '!=';
AND : 'and';
OR : 'or';
XOR :'xor';
NOT :'not' ;
INC : '++';
DEC : '--';
ADD : '+';
SUB : '-';
MUL : '*';
DIV : '/';
INT: [0-9]+;
NEWLINE: '\r'? '\n';
IS_SET:'isSet';
IS_EMPTY:'isEmpty';
IS_EQUAL:'isEqual';
WS: (' '|'\t' | '\n' | '\r' )+ -> skip;
ID
: JavaLetter JavaLetterOrDigit*
;
fragment
JavaLetter
: [a-zA-Z$_] // these are the "java letters" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierStart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment
JavaLetterOrDigit
: [a-zA-Z0-9$_] // these are the "java letters or digits" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierPart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment DoubleQuote: '"' ; // Hard to read otherwise.

The input set(buffer,variableX|"foo"); cannot be parsed by:
setCommand: 'set' LPAREN variableType '|' parameter RPAREN SEMI;
because buffer,variableX|"foo" is being matched by:
bufferType: 'buffer' COMMA ID '|' parameter;
which causes '|' parameter from the setCommand to not be able to match anything. If the input was something like set(buffer,variableX|"foo"|"bar");, it would parse successfully. Or remove '|' parameter from either the setCommand rule or from the bufferType rule.

Related

How to allow all chars in antlr lexxer inside double quotes to recognize a regex value?

I want to extend my grammar, so that it is allowed to define a regex value inside double quotes, here is an example which I want to allow
matches(value, test| ".*foobar[A-Z]");
Actually this is not recognized, because the dot and brackets are recognized before. Here is the Parse Tree
How can I resolve this, I tried it with a new rule ANY: . but with that I could not solve it. Any ideas?
This is my grammar
grammar FEL;
prog: expr+ SEMI? EOF;
expr:
statement #StatementExpr
|NOT expr #NotExpr
| expr AND expr #AndExpr
| expr (OR | XOR) expr #OrExpr
| function #FunctionExpr
| LPAREN expr RPAREN #ParenExpr
| writeCommand #WriteExpr
;
writeCommand: setCommand | setIfCommand;
statement: ID '=' getCommand NEWLINE #Assign;
setCommand: 'set' LPAREN variable = variableType '|' value = parameter RPAREN;
setIfCommand: 'setIf' LPAREN variableType '|' expr '?' p1 = parameter ':' p2 = parameter RPAREN;
getCommand: getFieldValue #FieldValue
| getInstanceAttribValue #InstanceAttribValue
| getFormAttribValue #FormAttributeValue
| getMandatorAttribValue #MandatorAttributeValue
;
getFieldValue: 'getFieldValue' LPAREN instanceID=ID COMMA fieldname=ID RPAREN;
getInstanceAttribValue: 'getInstanceAttrib' LPAREN instanceId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getFormAttribValue: 'getFormAttrib' LPAREN formId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getMandatorAttribValue: 'getMandatorAttrib' LPAREN mandator=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
parameter:
variableType
| constType
;
anyType: DoubleQuote ANY DoubleQuote;
pdixFuncton:ID;
constType:
ID #ID_Without
| '"' ID '"' #ID_WITH
| INT #INT_VALUE
| DIGIT_DOT #DIGIT_DOT_VALUE
;
variableType:
valueType #Variable_Value
|instanceType #Variable_Instance
|formType #Variable_Form
|bufferType #Variable_Buffer
|instanceAttribType #Variable_Instance_Attrib
|formAttribType #Variable_Form_Attrib
|mandatorAttribType #Variable_Mandator_Attrib
|instanceAttachmentType #Variable_Instance_Attachment
|formAttachmentType #Variable_Form_Attachment
|mandatorAttachmentType #Variable_Mandator_Attachment
;
valueType:'value' COMMA par=parameter (COMMA functionParameter)?;
instanceType: 'instance' COMMA instanceParameter;
formType: 'form' COMMA formParameter;
bufferType: 'buffer' COMMA id=ID;
instanceParameter: 'instanceId'
| 'instanceKey'
| 'firstpenId'
| 'lastpenId'
| 'lastUpdate'
| 'started'
;
formParameter: 'formId'
|'formKey'
|'lastUpdate'
;
functionParameter: 'lastPen'
| 'fieldGroup'
| ' fieldType'
| 'fieldSource'
| 'updateId'
| 'sessionId'
| 'icrConfidence'
| 'icrRecognition'
| 'lastUpdate';
instanceAttribType: p = ('instattrib' | 'instanceattrib') COMMA attributeType;
formAttribType:'formattrib' COMMA attributeType;
mandatorAttribType: 'mandatorattrib' COMMA attributeType;
instanceAttachmentType:('instattachment' | 'instanceatt') COMMA attachmentType;
formAttachmentType:'formAtt' COMMA attachmentType;
mandatorAttachmentType: 'mandatoratt' COMMA attachmentType;
attributeType: module = ID '#' attribName = ID;
attachmentType: name = ID '#' page = INT '#' index = INT;
function:
commandIsSet
|commandIsEmpty
|commandIsEqual
|commandIsNumLessEqual
|commandIsNumLess
|commandIsNumGreaterEqual
|commandIsNumGreater
|commandIsNumEqual
|commandIsLess
|commandIsLessEqual
|commandIsGreater
|commandIsGreaterEqual
|commandMatches
|commandContains
|commandEndsWith
|commandStartsWith
;
commandIsSet: IS_SET LPAREN parameter RPAREN;
commandIsEmpty: IS_EMPTY LPAREN parameter RPAREN;
commandIsEqual: IS_EQUAL LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandStartsWith: 'startsWith' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandEndsWith: 'endsWith' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandContains: 'contains' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandMatches: 'matches' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsLess: 'isLess' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsLessEqual: 'isLessEqual' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsGreater: 'isGreater' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsGreaterEqual: 'isGreaterEqual' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsNumEqual: 'isNumEqual' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsNumGreater: 'isNumGreater' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsNumGreaterEqual: 'isNumGreaterEqual' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsNumLess: 'isNumLess' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
commandIsNumLessEqual: 'isNumLessEqual' LPAREN p1 = parameter '|' p2 = parameter RPAREN;
/*
stringFunctionType:
a=substringStrFunction
| a=cutStrFunction
| a=replaceStrFunction
| a=reformatDateStrFunction
| a=translateStrFunction
| a=fillStrFunction
| a=concatStrFunction
| a=justifyStrFunction
| a=ifElseStrFunction
| a=tokenStrFunction
| a=toLowerFunction
| a=toUpperFunction
| a=trimFunction
;
*/
LPAREN : '(';
RPAREN : ')';
LBRACE : '{';
RBRACE : '}';
LBRACK : '[';
RBRACK : ']';
SEMI : ';';
COMMA : ',';
DOT : '.';
ASSIGN : '=';
GT : '>';
LT : '<';
BANG : '!';
TILDE : '~';
QUESTION : '?';
COLON : ':';
EQUAL : '==';
LE : '<=';
GE : '>=';
NOTEQUAL : '!=';
AND : 'and';
OR : 'or';
XOR :'xor';
NOT :'not' ;
INC : '++';
DEC : '--';
ADD : '+';
SUB : '-';
MUL : '*';
DIV : '/';
INT: [0-9]+;
DIGIT_DOT: FloatNumber;
IS_SET:'isSet';
IS_EMPTY:'isEmpty';
IS_EQUAL:'isEqual';
WS: (' '|'\t' | NEWLINE | '\r' )+ -> skip;
NEWLINE: '\n';
ID: JavaLetter JavaLetterOrDigit* | ANY;
ANY: . ;
fragment FloatNumber: ('0'..'9')+ ('.' ('0'..'9')+)?;
fragment
JavaLetter
: [a-zA-Z$_] // these are the "java letters" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierStart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment
JavaLetterOrDigit
: [a-zA-Z0-9$_] // these are the "java letters or digits" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierPart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment DoubleQuote: '"' ; // Hard to read otherwise.
Remove the ANY lexer rule from your ID rule, it makes no sense to let input like ^ become an ID.
And creating strings is usually done in the lexer. Something like this should do it:
anyType : STRING;
constType
: ID #ID_Without
| STRING #ID_WITH
| INT #INT_VALUE
| DIGIT_DOT #DIGIT_DOT_VALUE
;
STRING : '"' ( ~[\\"\r\n] | '\\' ~[\r\n] )* '"';
Also, in your functionParameter rule there's a ' fieldType' token. That should probably be 'fieldType' I guess.

antlr4 literal string handling

I have the following antlr4 grammar:
grammar squirrel;
program: globalstatement+;
globalstatement: globalvardef | classdef | functiondef;
globalvardef: IDENT '=' constantexpr ';';
classdef: CLASS IDENT '{' classstatement+ '}';
functiondef: FUNCTION IDENT '(' parameterlist ')' functionbody;
constructordef: CONSTRUCTOR '(' parameterlist ')' functionbody;
parameterlist: IDENT (',' IDENT)* | ;
functionbody: '{' statement* '}';
classstatement: globalvardef | functiondef | constructordef;
statement: expression ';';
expression:
IDENT # ident |
IDENT '=' expression # assignment |
IDENT ('.' IDENT)+ # lookupchain |
constantexpr # constant |
IDENT '(' expressionlist ')' # functioncall |
expression '+' expression # addition;
constantexpr: INTEGER | STRING;
expressionlist: expression (',' expression)* | ;
CONSTRUCTOR: 'constructor';
CLASS: 'class';
FUNCTION: 'function';
COMMENT: '//'.*[\n];
STRING: '"' CHAR* '"';
CHAR: [ a-zA-Z0-9];
INTEGER: [0-9]+;
IDENT: [a-zA-Z]+;
WS: [ \t\r\n]+ -> skip;
Now if I parse this file:
z = "global variable";
class Base
{
z = 10;
}
everything is fine:
#0,0:0='z',<16>,1:0
#1,2:2='=',<1>,1:2
#2,4:20='"global variable"',<14>,1:4
#3,21:21=';',<2>,1:21
#4,26:30='class',<11>,3:0
#5,32:35='Base',<16>,3:6
#6,38:38='{',<3>,4:0
#7,42:42='z',<16>,5:1
#8,44:44='=',<1>,5:3
#9,46:47='10',<15>,5:5
#10,48:48=';',<2>,5:7
#11,51:51='}',<4>,6:0
#12,56:55='<EOF>',<-1>,8:0
But with this file:
z = "global variable";
class Base
{
z = "10";
}
I get this:
#0,0:0='z',<16>,1:0
#1,2:2='=',<1>,1:2
#2,4:49='"global variable";\r\n\r\nclass Base\r\n{\r\n\tz = "10"',<14>,1:4
#3,50:50=';',<2>,5:9
#4,53:53='}',<4>,6:0
#5,58:57='<EOF>',<-1>,8:0
So it seems like everything between the first " and last " in a file gets matched to one string literal.
How do I prevent this ?
Note the string is matching from the first quote to the last possible quote.
By default, a Kleene operator (*) in ANTLR is greedy. So, change
STRING: '"' CHAR* '"';
to
STRING: '"' CHAR*? '"';
to make it non-greedy.

how can I refactor this ANTLR4 grammar so that it isn't mutually left recursive?

I can't seem to figure out why this grammar won't compile. It compiled fine until I modified line 145 from
(Identifier '.')* functionCall
to
(primary '.')? functionCall
I've been trying to figure out how to solve this issue for a while but I can't seem to be able to. Here's the error:
The following sets of rules are mutually left-recursive [primary]
grammar Tadpole;
#header
{package net.tadpole.compiler.parser;}
file
: fileContents*
;
fileContents
: structDec
| functionDec
| statement
| importDec
;
importDec
: 'import' Identifier ';'
;
literal
: IntegerLiteral
| FloatingPointLiteral
| BooleanLiteral
| CharacterLiteral
| StringLiteral
| NoneLiteral
| arrayLiteral
;
arrayLiteral
: '[' expressionList? ']'
;
expressionList
: expression (',' expression)*
;
expression
: primary
| unaryExpression
| <assoc=right> expression binaryOpPrec0 expression
| <assoc=left> expression binaryOpPrec1 expression
| <assoc=left> expression binaryOpPrec2 expression
| <assoc=left> expression binaryOpPrec3 expression
| <assoc=left> expression binaryOpPrec4 expression
| <assoc=left> expression binaryOpPrec5 expression
| <assoc=left> expression binaryOpPrec6 expression
| <assoc=left> expression binaryOpPrec7 expression
| <assoc=left> expression binaryOpPrec8 expression
| <assoc=left> expression binaryOpPrec9 expression
| <assoc=left> expression binaryOpPrec10 expression
| <assoc=right> expression binaryOpPrec11 expression
;
unaryExpression
: unaryOp expression
| prefixPostfixOp primary
| primary prefixPostfixOp
;
unaryOp
: '+'
| '-'
| '!'
| '~'
;
prefixPostfixOp
: '++'
| '--'
;
binaryOpPrec0
: '**'
;
binaryOpPrec1
: '*'
| '/'
| '%'
;
binaryOpPrec2
: '+'
| '-'
;
binaryOpPrec3
: '>>'
| '>>>'
| '<<'
;
binaryOpPrec4
: '<'
| '>'
| '<='
| '>='
| 'is'
;
binaryOpPrec5
: '=='
| '!='
;
binaryOpPrec6
: '&'
;
binaryOpPrec7
: '^'
;
binaryOpPrec8
: '|'
;
binaryOpPrec9
: '&&'
;
binaryOpPrec10
: '||'
;
binaryOpPrec11
: '='
| '**='
| '*='
| '/='
| '%='
| '+='
| '-='
| '&='
| '|='
| '^='
| '>>='
| '>>>='
| '<<='
| '<-'
;
primary
: literal
| fieldName
| '(' expression ')'
| '(' type ')' (primary | unaryExpression)
| 'new' objType '(' expressionList? ')'
| primary '.' fieldName
| primary dimension
| (primary '.')? functionCall
;
functionCall
: functionName '(' expressionList? ')'
;
functionName
: Identifier
;
dimension
: '[' expression ']'
;
statement
: '{' statement* '}'
| expression ';'
| 'recall' ';'
| 'return' expression? ';'
| variableDec
| 'if' '(' expression ')' statement ('else' statement)?
| 'while' '(' expression ')' statement
| 'do' expression 'while' '(' expression ')' ';'
| 'do' '{' statement* '}' 'while' '(' expression ')' ';'
;
structDec
: 'struct' structName ('(' parameterList ')')? '{' variableDec* functionDec* '}'
;
structName
: Identifier
;
fieldName
: Identifier
;
variableDec
: type fieldName ('=' expression)? ';'
;
type
: primitiveType ('[' ']')*
| objType ('[' ']')*
;
primitiveType
: 'byte'
| 'short'
| 'int'
| 'long'
| 'char'
| 'boolean'
| 'float'
| 'double'
;
objType
: (Identifier '.')? structName
;
functionDec
: 'def' functionName '(' parameterList? ')' ':' type '->' functionBody
;
functionBody
: statement
;
parameterList
: parameter (',' parameter)*
;
parameter
: type fieldName
;
IntegerLiteral
: DecimalIntegerLiteral
| HexIntegerLiteral
| OctalIntegerLiteral
| BinaryIntegerLiteral
;
fragment
DecimalIntegerLiteral
: DecimalNumeral IntegerSuffix?
;
fragment
HexIntegerLiteral
: HexNumeral IntegerSuffix?
;
fragment
OctalIntegerLiteral
: OctalNumeral IntegerSuffix?
;
fragment
BinaryIntegerLiteral
: BinaryNumeral IntegerSuffix?
;
fragment
IntegerSuffix
: [lL]
;
fragment
DecimalNumeral
: Digit (Digits? | Underscores Digits)
;
fragment
Digits
: Digit (DigitsAndUnderscores? Digit)?
;
fragment
Digit
: [0-9]
;
fragment
DigitsAndUnderscores
: DigitOrUnderscore+
;
fragment
DigitOrUnderscore
: Digit
| '_'
;
fragment
Underscores
: '_'+
;
fragment
HexNumeral
: '0' [xX] HexDigits
;
fragment
HexDigits
: HexDigit (HexDigitsAndUnderscores? HexDigit)?
;
fragment
HexDigit
: [0-9a-fA-F]
;
fragment
HexDigitsAndUnderscores
: HexDigitOrUnderscore+
;
fragment
HexDigitOrUnderscore
: HexDigit
| '_'
;
fragment
OctalNumeral
: '0' [oO] Underscores? OctalDigits
;
fragment
OctalDigits
: OctalDigit (OctalDigitsAndUnderscores? OctalDigit)?
;
fragment
OctalDigit
: [0-7]
;
fragment
OctalDigitsAndUnderscores
: OctalDigitOrUnderscore+
;
fragment
OctalDigitOrUnderscore
: OctalDigit
| '_'
;
fragment
BinaryNumeral
: '0' [bB] BinaryDigits
;
fragment
BinaryDigits
: BinaryDigit (BinaryDigitsAndUnderscores? BinaryDigit)?
;
fragment
BinaryDigit
: [01]
;
fragment
BinaryDigitsAndUnderscores
: BinaryDigitOrUnderscore+
;
fragment
BinaryDigitOrUnderscore
: BinaryDigit
| '_'
;
// §3.10.2 Floating-Point Literals
FloatingPointLiteral
: DecimalFloatingPointLiteral FloatingPointSuffix?
| HexadecimalFloatingPointLiteral FloatingPointSuffix?
;
fragment
FloatingPointSuffix
: [fFdD]
;
fragment
DecimalFloatingPointLiteral
: Digits '.' Digits? ExponentPart?
| '.' Digits ExponentPart?
| Digits ExponentPart
| Digits
;
fragment
ExponentPart
: ExponentIndicator SignedInteger
;
fragment
ExponentIndicator
: [eE]
;
fragment
SignedInteger
: Sign? Digits
;
fragment
Sign
: [+-]
;
fragment
HexadecimalFloatingPointLiteral
: HexSignificand BinaryExponent
;
fragment
HexSignificand
: HexNumeral '.'?
| '0' [xX] HexDigits? '.' HexDigits
;
fragment
BinaryExponent
: BinaryExponentIndicator SignedInteger
;
fragment
BinaryExponentIndicator
: [pP]
;
BooleanLiteral
: 'true'
| 'false'
;
CharacterLiteral
: '\'' SingleCharacter '\''
| '\'' EscapeSequence '\''
;
fragment
SingleCharacter
: ~['\\]
;
StringLiteral
: '"' StringCharacters? '"'
;
fragment
StringCharacters
: StringCharacter+
;
fragment
StringCharacter
: ~["\\]
| EscapeSequence
;
fragment
EscapeSequence
: '\\' [btnfr"'\\]
| OctalEscape
| UnicodeEscape
;
fragment
OctalEscape
: '\\' OctalDigit
| '\\' OctalDigit OctalDigit
| '\\' ZeroToThree OctalDigit OctalDigit
;
fragment
ZeroToThree
: [0-3]
;
fragment
UnicodeEscape
: '\\' 'u' HexDigit HexDigit HexDigit HexDigit
;
NoneLiteral
: 'nil'
;
Identifier
: IdentifierStartChar IdentifierChar*
;
fragment
IdentifierStartChar
: [a-zA-Z$_] // these are the "java letters" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierStart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment
IdentifierChar
: [a-zA-Z0-9$_] // these are the "java letters or digits" below 0xFF
| // covers all characters above 0xFF which are not a surrogate
~[\u0000-\u00FF\uD800-\uDBFF]
{Character.isJavaIdentifierPart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
WS : [ \t\r\n\u000C]+ -> skip
;
LINE_COMMENT
: '#' ~[\r\n]* -> skip
;
The left recursive invocation needs to be the first, so no parenthesis can be placed before it.
You can rewrite it like this:
primary
: literal
| fieldName
| '(' expression ')'
| '(' type ')' (primary | unaryExpression)
| 'new' objType '(' expressionList? ')'
| primary '.' fieldName
| primary dimension
| primary '.' functionCall
| functionCall
;
which is equivalent.

ANTLR4 Grammar picks up 'and' and 'or' in variable names

Please help me with my ANTLR4 Grammar.
Sample "formel":
(Arbejde.ArbejderIKommuneNr=860) and (Arbejde.ErIArbejde = 'J') &
(Arbejde.ArbejdsTimerPrUge = 40)
(Ansogeren.BorIKommunen = 'J') and (BeregnDato(Ansogeren.Fodselsdato;
'+62Å') < DagsDato)
(Arb.BorI=860)
My problem is that Arb.BorI=860 is not handled correct. I get this error:
Error: no viable alternative at input '(Arb.Bor' at linenr/position: 1/6 \r\nException: Der blev udløst en undtagelse af typen 'Antlr4.Runtime.NoViableAltException
Please notis that Arb.BorI contains the word 'or'.
I think my problem is that my 'booleanOps' in the grammar override 'datakildefelt'
So... My problem is how do I get my grammar correct - I am stuck, so any help will be appreciated.
My Grammar:
grammar UnikFormel;
formel : boolExpression # BooleanExpr
| expression # Expr
| '(' formel ')' # Parentes;
boolExpression : ( '(' expression ')' ) ( booleanOps '(' expression ')' )+;
expression : element compareOps element # Compare;
element : datakildefelt # DatakildeId
| function # Funktion
| int # Integer
| decimal # Real
| string # Text;
datakildefelt : datakilde '.' felt;
datakilde : identifyer;
felt : identifyer;
function : funktionsnavn ('(' funcParameters? ')')?;
funktionsnavn : identifyer;
funcParameters : funcParameter (';' funcParameter)*;
funcParameter : element;
identifyer : LETTER+;
int : DIGIT+;
decimal : DIGIT+ '.' DIGIT+ | '.' DIGIT+;
string : QUOTE .*? QUOTE;
booleanOps : (AND | OR);
compareOps : (LT | GT | EQ | GTEQ | LTEQ);
QUOTE : '\'';
OPERATOR: '+';
DIGIT: [0-9];
LETTER: [a-åA-Å];
MUL : '*';
DIV : '/';
ADD : '+';
SUB : '-';
GT : '>';
LT : '<';
EQ : '=';
GTEQ : '>=';
LTEQ : '<=';
AND : '&' | 'and';
OR : '?' | 'or';
WS : ' '+ -> skip;
Rules that come first always have precedence. In your case you need to move AND and OR before LETTER. Also there is the same problem with GTEQ and LTEQ, maybe somewhere else too.
EDIT
Additionally, you should make identifyer a lexer rule, i.e. start with capital letter (IDENTIFIER or Identifier). The same goes for int, decimal and string. Input is initially a stream of characters and is first processed into a stream of tokens, using only lexer rules. At this point parser rules (those starting with lowercase letter) do not come to play yet. So, to make "BorI" parse as single entity (token), you need to create a lexer rule that matches identifiers. Currently it would be parsed as 3 tokens: LETTER (B) OR (or) LETTER (I).
Thanks for your help. There were multiple problems. Reading the ANTLR4 book and using "TestRig -gui" got me on the right track. The working grammar is:
grammar UnikFormel;
formel : '(' formel ')' # Parentes
| expression # Expr
| boolExpression # BooleanExpr
;
boolExpression : '(' expression ')' ( booleanOps '(' expression ')' )+
| '(' formel ')' ( booleanOps '(' formel ')' )+;
expression : element compareOps element # Compare;
datakildefelt : ID '.' ID;
function : ID ('(' funcParameters? ')')?;
funcParameters : funcParameter (';' funcParameter)*;
funcParameter : element;
element : datakildefelt # DatakildeId
| function # Funktion
| INT # Integer
| DECIMAL # Real
| STRING # Text;
booleanOps : (AND | OR);
compareOps : ( GTEQ | LTEQ | LT | GT | EQ |);
AND : '&' | 'and';
OR : '?' | 'or';
GTEQ : '>=';
LTEQ : '<=';
GT : '>';
LT : '<';
EQ : '=';
ID : LETTER ( LETTER | DIGIT)*;
INT : DIGIT+;
DECIMAL : DIGIT+ '.' DIGIT+ | '.' DIGIT+;
STRING : QUOTE .*? QUOTE;
fragment QUOTE : '\'';
fragment DIGIT: [0-9];
fragment LETTER: [a-åA-Å];
WS : [ \t\r\n]+ -> skip;

Antlr4 Grammar/Rules - issue with solving BASIC print variable

The scenario is that I want to create a BASIC (high level) language using ANTRL4.
The test input below is the creation of a variable called C$ and assigning an integer value. The value assignment works. The print statement works except where concatenating the variable to it:-
************ TEST CASE ****************
$C=15;
print "dangerdanger!"; # print works
print "Number of GB left=" + $C;
Using a Parse Tree Inspector I can see assignments are working fine but when it gets to the identification of the variable in the string it seems there is a mismatched input '+' expecting STMTEND.
I wondered if anyone could help me out here and see what adjustment I need to make to my rules and grammar to solve this issue.
Many thanks in advance.
Kevin
PS. As a side issue I would rather have C$ than $C but early days...
********RULES************
VARNAME : '$'('A'..'Z')*
;
CONCAT : '+'
;
STMTEND : SEMICOLON NEWLINE* | NEWLINE+
;
STRING : SQUOTED_STRING (CONCAT SQUOTED_STRING | CONCAT VARNAME)*
| DQUOTED_STRING (CONCAT DQUOTED_STRING | CONCAT VARNAME)*
;
fragment SQUOTED_STRING : '\'' (~['])* '\''
;
fragment DQUOTED_STRING
: '"' ( ESC_SEQ| ~('\\'|'"') )* '"'
;
fragment ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment HEX_DIGIT : '0x' ('0'..'9' | 'a'..'f' | 'A'..'F')+
;
fragment UNICODE_ESC : '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
;
SEMICOLON : ';'
;
NEWLINE : '\r'?'\n'
************GRAMMAR************
print_command
: PRINT STRING STMTEND #printCommandLabel
;
assignment
: VARNAME EQUALS INTEGER STMTEND #assignInteger
| VARNAME EQUALS STRING STMTEND #assignString
;
You shouldn't try to create concat-expressions inside your lexer: that is the responsibility of the parser. Something like this should do it:
print_command
: PRINT STRING STMTEND #printCommandLabel
;
assignment
: VARNAME EQUALS expression STMTEND
;
expression
: expression CONCAT expression
| INTEGER
| STRING
| VARNAME
;
CONCAT
: '+'
;
VARNAME
: '$'('A'..'Z')*
;
STMTEND
: SEMICOLON NEWLINE*
| NEWLINE+
;
STRING
: SQUOTED_STRING
| DQUOTED_STRING
;
fragment SQUOTED_STRING
: '\'' (~['])* '\''
;
fragment DQUOTED_STRING
: '"' ( ESC_SEQ| ~('\\'|'"') )* '"'
;
fragment ESC_SEQ
: '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
| UNICODE_ESC
| OCTAL_ESC
;
fragment OCTAL_ESC
: '\\' ('0'..'3') ('0'..'7') ('0'..'7')
| '\\' ('0'..'7') ('0'..'7')
| '\\' ('0'..'7')
;
fragment HEX_DIGIT : '0x' ('0'..'9' | 'a'..'f' | 'A'..'F')+;
fragment UNICODE_ESC : '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT;
fragment SEMICOLON : ';';
fragment NEWLINE : '\r'?'\n';

Resources