Xtext refering to element from different file does not work - dsl

Hello I am having two files in my xtext editor, the first one containing all definitions and the second one containing the executed recipe. The Grammar looks like this:
ServiceAutomationProgram:
('package' name=QualifiedName ';')?
imports+=ServiceAutomationImport*
definitions+=Definition*;
ServiceAutomationImport:
'import' importedNamespace=QualifiedNameWithWildcard ';';
Definition:
'define' ( TypeDefinition | ServiceDefinition |
SubRecipeDefinition | RecipeDefinition) ';';
TypeDefinition:
'quantity' name=ID ;
SubRecipeDefinition:
'subrecipe' name=ID '('( subRecipeParameters+=ServiceParameterDefinition (','
subRecipeParameters+=ServiceParameterDefinition)*)? ')' '{'
recipeSteps+=RecipeStep*
'}';
RecipeDefinition:
'recipe' name=ID '{' recipeSteps+=RecipeStep* '}';
RecipeStep:
(ServiceInvocation | SubRecipeInvocation) ';';
SubRecipeInvocation:
name=ID 'subrecipe' calledSubrecipe=[SubRecipeDefinition] '('( parameters+=ServiceInvocationParameter (',' parameters+=ServiceInvocationParameter)* )?')'
;
ServiceInvocation:
name=ID 'service' service=[ServiceDefinition]
'(' (parameters+=ServiceInvocationParameter (',' parameters+=ServiceInvocationParameter)*)? ')'
;
ServiceInvocationParameter:
ServiceEngineeringQuantityParameter | SubRecipeParameter
;
ServiceEngineeringQuantityParameter:
parameterName=[ServiceParameterDefinition] value=Amount;
ServiceDefinition:
'service' name=ID ('inputs' serviceInputs+=ServiceParameterDefinition (','
serviceInputs+=ServiceParameterDefinition)*)?;
ServiceParameterDefinition:
name=ID ':' (parameterType=[TypeDefinition]);
;
SubRecipeParameter:
parameterName=[ServiceParameterDefinition]
;
QualifiedNameWithWildcard:
QualifiedName '.*'?;
QualifiedName:
ID ('.' ID)*;
Amount:
INT ;
....
definitionfile file.mydsl:
define quantity Temperature;
define service Heater inputs SetTemperature:Temperature;
define subrecipe sub_recursive() {
Heating1 service Heater(SetTemperature 10);
};
....
recipefile secondsfile.mydsl:
define recipe Main {
sub1 subrecipe sub_recursive();
};
.....
In my generator file which looks like this:
override void doGenerate(Resource resource, IFileSystemAccess2 fsa, IGeneratorContext context) {
for (e : resource.allContents. toIterable.filter (RecipeDefinition)){
e.class;//just for demonstration add breakpoint here and //traverse down the tree
}
}
I need as an example the information RecipeDefinition.recipesteps.subrecipeinvocation.calledsubrecipe.recipesteps.serviceinvocation.service.name which is not accessible (null) So some of the very deep buried information gets lost (maybe due to lazylinking?).
To make the project executable also add to the scopeprovider:
public IScope getScope(EObject context, EReference reference) {
if (context instanceof ServiceInvocationParameter
&& reference == MyDslPackage.Literals.SERVICE_INVOCATION_PARAMETER__PARAMETER_NAME) {
ServiceInvocationParameter invocationParameter = (ServiceInvocationParameter) context;
List<ServiceParameterDefinition> candidates = new ArrayList<>();
if(invocationParameter.eContainer() instanceof ServiceInvocation) {
ServiceInvocation serviceCall = (ServiceInvocation) invocationParameter.eContainer();
ServiceDefinition calledService = serviceCall.getService();
candidates.addAll(calledService.getServiceInputs());
if(serviceCall.eContainer() instanceof SubRecipeDefinition) {
SubRecipeDefinition subRecipeCall=(SubRecipeDefinition) serviceCall.eContainer();
candidates.addAll(subRecipeCall.getSubRecipeParameters());
}
return Scopes.scopeFor(candidates);
}
else if(invocationParameter.eContainer() instanceof SubRecipeInvocation) {
SubRecipeInvocation serviceCall = (SubRecipeInvocation) invocationParameter.eContainer();
SubRecipeDefinition calledSub = serviceCall.getCalledSubrecipe();
candidates.addAll(calledSub.getSubRecipeParameters());
return Scopes.scopeFor(candidates);
}
}return super.getScope(context, reference);
}
When I put all in the same file it works as it does the first time executed after launching runtime but afterwards(when dogenerate is triggered via editor saving) some information is missing. Any idea how to get to the missing informations? thanks a lot!

Related

ANTLR4: Parser adding duplicate entries

I have below input to be parsed:-
([LANGUAGE] IN ("Arabic", "Dutch") AND [Content Series] IN ("The Walking Dead") AND [PUBLISHER_NAME] IN ("Yahoo Search", "Yahoo! NAR") )
OR
([LANGUAGE] IN ("English") AND [PUBLISHER_NAME] IN ("Aol News", "Microsoft-Bing!") )
Basically the inputs have 2 groups separated by 'OR'.Both groups has several base exp(targetEntities) separated by AND. So each group has list of target entities.
Grammar file:
grammar Exp;
options {
language = Java;
}
start
: def EOF
;
def : (AND? base)+
| (OR? '(' def ')')*
;
base : key operator values ;
key : LSQR ID RSQR ;
values : '(' VALUE (',' VALUE)* ')' ;
operator : IN
| NIN
;
VALUE: '"' .*? '"' ;
AND : 'AND' ;
OR : 'OR' ;
NOT : 'not' ;
EQ : '=' ;
COMMA : ',' ;
SEMI : ';' ;
IN : 'IN' ;
NIN : 'NOT_IN' ;
LSQR : '[' ;
RSQR : ']' ;
INT : [0-9]+ ;
ID: [a-zA-Z_][a-zA-Z_0-9-!]* ;
WS: [\t\n\r\f ]+ -> skip ;
Below is the listener and parser-
#Component
#NoArgsConstructor
public class ANTLRTargetingExpressionParser {`
static List<Group> groupList = new ArrayList<>();
public String entityOperator;
public static class ExpMapper extends ExpBaseListener {
TargetEntity targetEntity;
Group group;
List<TargetEntity> targetEntities;
private static int inc = 1;
#Override
public void exitDef(ExpParser.DefContext ctx) {
group.setTargets(targetEntities);
groupList.add(group);
super.exitDef(ctx);
}
#Override
public void exitValues(ExpParser.ValuesContext ctx) {
targetEntity.setValues(
Arrays.asList(
Arrays.toString(ctx.VALUE().stream().collect(Collectors.toSet()).toArray())));
super.exitValues(ctx);
targetEntities.add(targetEntity);
}
#Override
public void exitOperator(ExpParser.OperatorContext ctx) {
targetEntity.setOperator(ctx.getText());
super.exitOperator(ctx);
}
#Override
public void exitKey(ExpParser.KeyContext ctx) {
targetEntity = new TargetEntity();
ctx.getParent();
targetEntity.setEntity(ctx.ID().getText());
super.exitKey(ctx);
}
#Override
public void enterDef(ExpParser.DefContext ctx) {
group = new Group();
targetEntities = new ArrayList<>();
super.enterDef(ctx);
}
}
public List<Group> parse(String expression) {`
ANTLRInputStream in = new ANTLRInputStream(expression);
ExpLexer lexer = new ExpLexer(in);
CommonTokenStream tokens = new CommonTokenStream(lexer);
ExpParser parser = new ExpParser(tokens);
parser.setBuildParseTree(true); // tell ANTLR to build a parse tree
ParseTree tree = parser.def();
/** Create standard walker. */
ParseTreeWalker walker = new ParseTreeWalker();
System.out.println(tree.toStringTree(parser));
ExpMapper mapper = new ExpMapper();
walker.walk(mapper, tree);
return groupList;
}
}
Output:-
[Group(targets=[{LANGUAGE, IN, [["Dutch", "Arabic"]]}, {Content_Series, IN, [["The Walking Dead"]]}, {PUBLISHER_NAME, IN, [["Yahoo Search", "Yahoo! NAR"]]}]),
Group(targets=[{LANGUAGE, IN, [["English"]]}, {PUBLISHER_NAME, IN, [["Aol News", "Microsoft-Bing!"]]}]),
Group(targets=[{LANGUAGE, IN, [["English"]]}, {PUBLISHER_NAME, IN, [["Aol News", "Microsoft-Bing!"]]}])]
Q1:- I am getting duplicate value in the grouplist at end. Tried checking the value in ctx to stop the walker but couldnt help.
Q2:- Also how can we catch the soft exception thrown by grammar file in case of wrong input given in java.
(NOTE: It's MUCH easier to sort questions out if you ensure that the examples you provide are valid and are compilable. I had to change a few things just to get a clean parse, and there's too much missing to attempt to compile and run your code.)
That said....
def : (AND? base)+
| (OR? '(' def ')')*
;
Would normally be represented as something akin to
def: '(' def ')'
| def AND def
| def OR def
| base
;
(Note: these are not exactly equivalent. Your rule requires parentheses around defs used in an OR, but disallows them when used with AND. Those would be "odd" constraints, so I'm not sure if you intended that.)
You'll notice here that it's clear that a def can contain other defs. This is also true in your rule for (but only as the second half of an OR type.
It can be really useful to use a plugin or the -gui option of the antler tool, to see a visual representation of your tree. (Both IntelliJ, and VS Code have good plugins available for this). With that visualization it would have been clear that there was a def in a subtree of a def. (The information would have been the in the output of the System.out.println(tree.toStringTree(parser));, but a bit harder to notice.
This is your clue. You're getting a duplicate of the second half of your OR and this is because you'll have a nested def and, as a result, you'll exitDef twice (and add it twice in the process).
Your listener does not handle nested structures like this properly (having only a targetEntity and a group). You'll need to do something like maintaining a stack of Group instances and pushing/popping as you enter/exit (and only dealing with the top of the stack).
A few other observations:
super.enterDef(ctx);
There's no need to call the super method on your listener overrides, the default methods are empty. (Of course, it does no harm, and it can be a "safe" practice to generally call the super method when overriding.
ctx.getParent();
You didn't do anything with this parent, as a result, this doesn't do anything.

Where is the InputMismatchException thrown?

When I execute my program with a certain token in the wrong spot, it throws the InputMismatchException, saying something along the lines of
line 21:0 mismatched input '#' expecting {'in', '||', '&&', '==', '!=', '>=', '<=', '^', '>', '<', '+', '-', '*', '/', '%', '[', ';', '?'}
Which is a terrible error message for the language I'm developing, so I'm looking to change it, but I can't find the source of it, I know why the error is being thrown, but I can't find the actual line of java code that throws the InputMismatchException, I don't think its anywhere in my project, so I assume it's somewhere in the antlr4 runtime, is there a way to disable these error messages, or at least change them?
Edit:
My grammar (the relevant parts) are as follows:
grammar Q;
parse
: header? ( allImport ';' )*? block EOF
;
block
: ( statement | functionDecl )* ( Return expression ';' )?
;
statement
: functionCall ';'
| ifStatement
| forStatement | forInStatement
| whileStatement
| tryCatchStatement
| mainFunctionStatement
| addWebServerTextStatement ';'
| reAssignment ';'
| classStatement
| constructorStatement ';'
| windowAddCompStatement ';'
| windowRenderStatement ';'
| fileWriteStatement ';'
| verifyFileStatement ';'
| objFunctionCall (';')?
| objCreateStatement ';'
| osExecStatement ';'
| anonymousFunction
| hereStatement ';'
;
And an example of the importStatement visit method is:
#Override
public QValue visitImportStatement(ImportStatementContext ctx) {
StringBuilder path = new StringBuilder();
StringBuilder text = new StringBuilder();
for (TerminalNode o : ctx.Identifier()) {
path.append("/").append(o.getText());
}
for (TerminalNode o : ctx.Identifier()) {
text.append(".").append(o.getText());
}
if (lang.allLibs.contains(text.toString().replace(".q.", "").toLowerCase(Locale.ROOT))) {
lang.parse(text.toString());
return QValue.VOID;
}
for (File f : lang.parsed) {
Path currentRelativePath = Paths.get("");
String currentPath = currentRelativePath.toAbsolutePath().toString();
File file = new File(currentPath + "/" + path + ".l");
if (f.getPath().equals(file.getPath())) {
return null;
}
}
QLexer lexer = null;
Path currentRelativePath = Paths.get("");
String currentPath = currentRelativePath.toAbsolutePath().toString();
File file = new File(currentPath + "/" + path + ".l");
lang.parsed.add(file);
try {
lexer = new QLexer(CharStreams.fromFileName(currentPath + "/" + path + ".l"));
} catch (IOException e) {
throw new Problem("Library or File not found: " + path, ctx);
}
QParser parser = new QParser(new CommonTokenStream(lexer));
parser.setBuildParseTree(true);
ParseTree tree = parser.parse();
Scope s = new Scope(lang.scope, false);
Visitor v = new Visitor(s, new HashMap<>());
v.visit(tree);
return QValue.VOID;
}
Because of the parse rule in my g4 file, the import statement MUST come before any other thing (aside from a header statement), so doing this would throw an error
class Main
#import src.main.QFiles.aLib;
fn main()
try
std::ln("orih");
onflaw
end
new Object as o();
o::set("val");
std::ln(o::get());
std::ln("itj");
end
end
And, as expected, it throws an InputMismatchException, but that's not in any of my code
You can remove the default error strategy and implement your own:
...
QParser parser = new QParser(new CommonTokenStream(lexer));
parser.removeErrorListeners();
parser.addErrorListener(new BaseErrorListener() {
#Override
public void syntaxError(Recognizer<?, ?> recognizer, Object offendingSymbol, int line, int charPositionInLine, String msg, RecognitionException e) {
throw new RuntimeException("Your own message here", e);
}
});
ParseTree tree = parser.parse();
...

ANTLR4 - named function arguments

My goal is to generate parser that could handle following code with named function parameters and nested function calls
fnCallY(namedArgStr = "xxx", namedArgZ=fnCallZ(namedArg="www"))
G4 lang file:
val : type_string
| function_call
;
function_call : function_name=ID arguments='('argument? (',' argument)* ')';
argument : name=ID '=' value=val ;
ID : [a-zA-Z_][a-zA-Z0-9_]*;
type_string : LITERAL;
fragment ESCAPED_QUOTE : '\\"';
LITERAL : '"' ( ESCAPED_QUOTE | ~('\n'|'\r') )*? '"'
| '\'' ( ESCAPED_QUOTE | ~('\n'|'\r') )*? '\'';
#Override
public void exitFunction_call(Test.Function_callContext ctx) {
List<Test.ArgumentContext> argument = ctx.argument();
for (Test.ArgumentContext arg : argument) {
Token name = arg.name;
Test.ValContext value = arg.value;
if (value.type_literal() == null || value.function_call() == null) {
throw new RuntimeException("Could not parse argument value");
}
}
}
arg.name holds correct data, but i cannot make the parser to parse the part after =.
The parser is recognizing the argument values.
(It's really valuable to learn the grun command line utility as it can test the grammar and tree structure without involving any of your own code)
This condition would appear to be your problem:
if (value.type_literal() == null || value.function_call() == null)
One or the other will always be null, so this will fail.
if (value.type_literal() == null && value.function_call() == null)
is probably what you want.

Newbie: 2.4# is accepted as a float. Is '#' a special character?

Wondering why the expression "setvalue(2#)' is happily accepted by the lexer (and parser) given my grammar/visitor. I am sure I am doing something wrong.
Below is a small sample that should illustrate the problem.
Any help is much appreciated.
grammar ExpressionEvaluator;
parse
: block EOF
;
block
: stat*
;
stat
: assignment
;
assignment
: SETVALUE OPAR expr CPAR
;
expr
: atom #atomExpr
;
atom
: OPAR expr CPAR #parExpr
| (INT | FLOAT) #numberAtom
| ID #idAtom
;
OPAR : '(';
CPAR : ')';
SETVALUE : 'setvalue';
ID
: [a-zA-Z_] [a-zA-Z_0-9]*
;
INT
: [0-9]+
;
FLOAT
: [0-9]+ '.' [0-9]*
| '.' [0-9]+
;
STRING
: '"' (~["\r\n] | '""')* '"'
;
SPACE
: [ \t\r\n] -> skip
;
Code snippet:
public override object VisitParse(ExpressionEvaluatorParser.ParseContext context)
{
return this.Visit(context.block());
}
public override object VisitAssignment(ExpressionEvaluatorParser.AssignmentContext context)
{
// TODO - Set ID Value
return Convert.ToDouble(this.Visit(context.expr()));
}
public override object VisitIdAtom(ExpressionEvaluatorParser.IdAtomContext context)
{
string id = context.GetText();
// TODO - Lookup ID value
return id;
}
public override object VisitNumberAtom(ExpressionEvaluatorParser.NumberAtomContext context)
{
return Convert.ToDouble(context.GetText());
}
public override object VisitParExpr(ExpressionEvaluatorParser.ParExprContext context)
{
return this.Visit(context.expr());
}
The # character actually isn't matching anything at all. When the lexer reaches that character, the following happen in order:
The lexer determines that no lexer rule can match the # character.
The lexer reports an error regarding the failure.
The lexer calls _input.consume() to skip past the bad character.
To ensure errors are reported as easily as possible, always add the following rule as the last rule in your lexer.
ErrChar
: .
;
The parser will report an error when it reaches an ErrChar, so you won't need to add an error listener to the lexer.

ANTLR4 Specific search

Basically I want to find, in a file, using ANTLR, every expression as defined :
WORD.WORD
for example : "end.beginning" matches
For the time being the file can have hundreds and hundreds of lines and a complexe structure.
Is there a way to skip every thing (character?) that does not match with the pattern described above, without making a grammar that fully represents the file ?
So far this is my grammar but i don't know what to do next.
grammar Dep;
program
:
dependencies
;
dependencies
:
(
dependency
)*
;
dependency
:
identifier
DOT
identifier
;
identifier
:
INDENTIFIER
;
DOT : '.' ;
INDENTIFIER
:
[a-zA-Z_] [a-zA-Z0-9_]*
;
OTHER
:
. -> skip
;
The way you're doing it now, the dependency rule would also match the tokens 'end', '.', 'beginning' from the input:
end
#####
.
#####
beginning
because the line breaks and '#'s are being skipped from the token stream.
If that is not what you want, i.e. you'd like to match "end.beginning" without any char in between, you should make a single lexer rule of it, and match that rule in your parser:
grammar Dep;
program
: DEPENDENCY* EOF
;
DEPENDENCY
: [a-zA-Z_] [a-zA-Z0-9_]* '.' [a-zA-Z_] [a-zA-Z0-9_]*
;
OTHER
: . -> skip
;
Then you could use a tree listener to do something useful with your DEPENDENCY's:
public class Main {
public static void main(String[] args) throws Exception {
String input = "### end.beginning ### end ### foo.bar mu foo.x";
DepLexer lexer = new DepLexer(new ANTLRInputStream(input));
DepParser parser = new DepParser(new CommonTokenStream(lexer));
ParseTreeWalker.DEFAULT.walk(new DepBaseListener(){
#Override
public void enterProgram(#NotNull DepParser.ProgramContext ctx) {
for (TerminalNode node : ctx.DEPENDENCY()) {
System.out.println("node=" + node.getText());
}
}
}, parser.program());
}
}
which would print:
node=end.beginning
node=foo.bar
node=foo.x

Resources