How can I make a rule to match all of its alternatives only once in any order, in ANTLR?
i.e.
rule: ('example\r\n' | 'example2\r\n') nextRule
I would like 'example' and 'example2' to match only once before moving onto the next rule.
Should match inputs of:
example
example2
or
example2
example
but not inputs of:
example
example
example2
How can I make a rule to match all of its alternatives only once in any order, in ANTLR?
"All of its alternatives only once" is simply rule: altA altB altC .... That's the easy part. Asking rule to accept all alternatives altA, altB, altC, etc. in any arrangement means accommodating every arrangement. That's easy enough to handle manually for small numbers (rule: (altA altB | altB altA);). But there's no automatic shortcut that I'm aware of to handling all the permutations for you automatically.
Here are some approaches in case there isn't a built-in way and assuming you can't relax your requirements. Caveats: I don't know the full scope of your problem; I don't know your grammar; I don't know why you want what you're asking for; I don't know what class of solution you prefer, other than you'd probably like it easier than any of these options.
First, you could bite the bullet and produce all the permutations of matches yourself, either by hand or by running a permutation generator. Then ANTLR would have what you want in a manner that it understands. It's crude but efficient: it's straight ANTLR syntax so there's no external code involved as there is in the options below.
For example, assume you have a rule field that processes input like "public static final x", with all three modifiers expected but in no particular order. The permutations would look like so:
field : modifiers ID EOF;
modifiers
: PUBLIC STATIC FINAL //ABC
| PUBLIC FINAL STATIC //ACB
| STATIC PUBLIC FINAL //BAC
| STATIC FINAL PUBLIC //BCA
| FINAL PUBLIC STATIC //CAB
| FINAL STATIC PUBLIC //CBA
;
That's the end of it. No external code, no predicates, no nothing.
Second, you could use semantic predicates in the grammar to ensure that all matches are provided and matched without duplicates. There are various ways of writing the predicates themselves, but it boils down to tracking what matches have been made (to prevent duplicates) and then testing whether the rule has matched all the parts that it expects. Here is a basic example, following the same requirements as the previous one:
field
locals [java.util.HashSet<String> names = new java.util.HashSet<String>();]
: modifiers ID EOF;
modifiers
//Ensure that the full number of modifiers have been provided
: {$field::names.size() < 3}? modifier modifiers
| {$field::names.size() == 3}? //match nothing once we have (any) three modifiers
;
modifier
//Ensure that no duplicates have been provided
: {!$field::names.contains("public")}? PUBLIC {$field::names.add("public");}
| {!$field::names.contains("static")}? STATIC {$field::names.add("static");}
| {!$field::names.contains("final")}? FINAL {$field::names.add("final");}
;
Here rule field tracks the modifier names in local variable names. Rule modifiers calls rule modifier until names contains three values. Rule modifier matches any name that doesn't have a corresponding key in names. Note that the values are manually added to names. They could be any arbitrary value as long as modifier's alternatives add the same value to both sides of the token it's matching.
My implementation is a little crude because the modifiers end up nesting in the produced parse tree (since modifiers contains one modifier and one modifiers that contains one modifier and one modifiers that...), but I hope you get the idea.
Third, you could leave the poor parser alone and test for completeness in the calling code. This can be done during parsing using a parser listener or done after parsing using the ParserRuleContext object produced by the parser. This breaks the problem into two parts: let the parser solve "any X, Y, Z in any order" and let the calling code solve "all and only just X, Y, Z."
Here is an example using the listener approach:
//partial grammar
field : modifier* ID EOF; //accept any modifiers in any order
modifier
: PUBLIC
| STATIC
| FINAL
;
//snippet of calling code
//initialize lexer and parser
parser.addParseListener(new MyGrammarBaseListener() {
#Override
public void exitField(FieldContext ctx) {
// The field rule has finished. Let's verify that no modifiers
// were duplicated.
//TODO check for duplicates, ensure all modifiers are specified.
//TODO log errors accordingly.
}
});
//Call the parser.
parser.field();
The grammar is kept clean. Modifiers can appear in the input arbitrarily before ID, in any number and in any order. The calling code performs the tests by whatever means it chooses, logging whatever errors it wants.
Here is an uberexample that pulls together the three options I mentioned, to give a clearer idea of what I'm talking about.
Modifiers.g
grammar Modifiers;
//Hard-coded version : all the permutations are specified //
permutationField : permutationModifiers ID EOF;
permutationModifiers
: PUBLIC STATIC FINAL //ABC
| PUBLIC FINAL STATIC //ACB
| STATIC PUBLIC FINAL //BAC
| STATIC FINAL PUBLIC //BCA
| FINAL PUBLIC STATIC //CAB
| FINAL STATIC PUBLIC //CBA
;
// Predicate version : use semantic predicates to prevent duplicates and ensure all the modifiers are provided //
predicateField
locals [java.util.HashSet<String> names = new java.util.HashSet<String>();]
: predicateModifiers ID EOF;
predicateModifiers
//Ensure that the full number of modifiers have been provided
: {$predicateField::names.size() < 3}? predicateModifier predicateModifiers
| {$predicateField::names.size() == 3}? //match nothing once we have (any) three modifiers
;
predicateModifier
//Ensure that no duplicates have been provided
: {!$predicateField::names.contains("public")}? PUBLIC {$predicateField::names.add("public");}
| {!$predicateField::names.contains("static")}? STATIC {$predicateField::names.add("static");}
| {!$predicateField::names.contains("final")}? FINAL {$predicateField::names.add("final");}
;
//Listener version : test everything when the parser calls the listener //
listenerField : listenerModifier* ID EOF;
listenerModifier
: PUBLIC
| STATIC
| FINAL
;
PUBLIC : 'public';
STATIC : 'static';
FINAL : 'final';
FOO : 'foo';
ID : [a-zA-Z]+;
WS : [ \r\n\t]+ -> skip;
ModifiersTest.java
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.BaseErrorListener;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.RecognitionException;
import org.antlr.v4.runtime.Recognizer;
import org.antlr.v4.runtime.misc.Nullable;
public class ModifiersTest {
public static void main(String[] args) {
run("public static final x", "ok");
run("final static public x", "ok");
run("static public static final x", "too many modifiers");
run("static x", "missing modifiers");
run("final final x", "missing & duplicated modifiers");
}
private static void run(String code, String title) {
System.out.printf("%n---%n**Input : %s**%n%n\t%s%n%n", title, code);
System.out.println("**Permutation Output**\n");
runPermutationTest(code);
System.out.println();
System.out.println("**Predicate Output**\n");
runPredicateTest(code);
System.out.println();
System.out.println("**Listener Output**\n");
runListenerTest(code);
System.out.println();
}
private static void runPermutationTest(String code) {
ModifiersParser parser = createParser(code);
parser.permutationField();
System.out.println("\t(done)");
}
private static void runPredicateTest(String code) {
ModifiersParser parser = createParser(code);
parser.predicateField();
System.out.println("\t(done)");
}
private static void runListenerTest(String code) {
ModifiersParser parser = createParser(code);
parser.addParseListener(new ModifiersBaseListener() {
#Override
public void exitListenerField(ModifiersParser.ListenerFieldContext ctx) {
// The field rule has finished. Let's verify that no modifiers
// were duplicated.
HashSet<String> uniqueNames = new HashSet<String>();
ArrayList<String> allNames = new ArrayList<String>();
HashSet<String> expectedNames = new HashSet<String>();
expectedNames.add("public");
expectedNames.add("static");
expectedNames.add("final");
if (ctx.listenerModifier() != null && !ctx.listenerModifier().isEmpty()) {
List<ModifiersParser.ListenerModifierContext> modifiers = ctx.listenerModifier();
// Collect all the modifier names in a set.
for (ModifiersParser.ListenerModifierContext modifier : modifiers) {
uniqueNames.add(modifier.getText());
allNames.add(modifier.getText());
}
}
// Is the number of unique modifiers less than the number of
// all given modifiers? If so, then there must be duplicates.
if (uniqueNames.size() < allNames.size()) {
ArrayList<String> names = new ArrayList<String>(allNames);
for (String name : uniqueNames){
names.remove(name);
}
System.out.println("\tDetected duplicate modifiers : " + names);
} else {
System.out.println("\t(No duplicate modifiers detected)");
}
//Are we missing any expected modifiers?
if (!uniqueNames.containsAll(expectedNames)) {
ArrayList<String> names = new ArrayList<String>(expectedNames);
names.removeAll(uniqueNames);
System.out.println("\tDetected missing modifiers : " + names);
} else {
System.out.println("\t(No missing modifiers detected)");
}
}
});
parser.listenerField();
System.out.println("\t(done)");
}
private static ModifiersParser createParser(String code) {
ANTLRInputStream input = new ANTLRInputStream(code);
ModifiersLexer lexer = new ModifiersLexer(input);
ModifiersParser parser = new ModifiersParser(new CommonTokenStream(lexer));
BaseErrorListener errorListener = createErrorListener();
lexer.addErrorListener(errorListener);
parser.addErrorListener(errorListener);
return parser;
}
private static BaseErrorListener createErrorListener() {
BaseErrorListener errorListener = new BaseErrorListener() {
#Override
public void syntaxError(Recognizer<?, ?> recognizer, #Nullable Object offendingSymbol, int line,
int charPositionInLine, String msg, #Nullable RecognitionException e) {
//Print the syntax error
System.out.printf("\t%s at (%d, %d)%n", msg, line, charPositionInLine);
}
};
return errorListener;
}
}
Test Scenarios (output from the code above)
Input : ok
public static final x
Permutation Output
(done)
Predicate Output
(done)
Listener Output
(No duplicate modifiers detected)
(No missing modifiers detected)
(done)
Input : ok
final static public x
Permutation Output
(done)
Predicate Output
(done)
Listener Output
(No duplicate modifiers detected)
(No missing modifiers detected)
(done)
Input : too many modifiers
static public static final x
Permutation Output
extraneous input 'static' expecting 'final' at (1, 14)
(done)
Predicate Output
no viable alternative at input 'static' at (1, 14)
(done)
Listener Output
Detected duplicate modifiers : [static]
(No missing modifiers detected)
(done)
Input : missing modifiers
static x
Permutation Output
no viable alternative at input 'staticx' at (1, 7)
(done)
Predicate Output
no viable alternative at input 'x' at (1, 7)
(done)
Listener Output
(No duplicate modifiers detected)
Detected missing modifiers : [final, public]
(done)
Input : missing & duplicated modifiers
final final x
Permutation Output
no viable alternative at input 'finalfinal' at (1, 6)
(done)
Predicate Output
no viable alternative at input 'final' at (1, 6)
(done)
Listener Output
Detected duplicate modifiers : [final]
Detected missing modifiers : [static, public]
(done)
With ANTLR 4, I would actually prefer to use something like the following, where the input is expected to contain exactly 1 each of a, b, and c.
items : (a | b | c)*;
Then in a listener, I would use code like the following:
#Override
public void enterItems(ItemsContext ctx) {
if (ctx.a().size() != 1) {
// report error
} else if (ctx.b().size() != 1) {
// report error
} else ...
}
Related
I have an antlr grammar in which embedded actions are used to collect data bottom up and build aggregated data structures. A short version is given below, where the aggregated data structures are only printed (ie no classes are created for them in this short sample code).
grammar Sample;
top returns [ArrayList l]
#init { $l = new ArrayList<String>(); }
: (mid { $l.add($mid.s); } )* ;
mid returns [String s]
: i1=identifier 'hello' i2=identifier
{ $s = $i1.s + " bye " + $i2.s; }
;
identifier returns [String s]
: ID { $s = $ID.getText(); } ;
ID : [a-z]+ ;
WS : [ \t\r\n]+ -> skip ;
Its corresponding Main program is:
public class Main {
public static void main( String[] args) throws Exception
{
SampleLexer lexer = new SampleLexer( new ANTLRFileStream(args[0]));
CommonTokenStream tokens = new CommonTokenStream( lexer );
SampleParser parser = new SampleParser( tokens );
ArrayList<String> top = parser.top().l;
System.out.println(top);
}
}
And a sample test is:
aaa hello bbb
xyz hello pqr
Since one of the objectives of antlr is to keep the grammar file reusable and action-independent, I am trying to delete the actions from this file and move it to a tree walker. I took a first stab at it with the following code:
public class Main {
public static void main( String[] args) throws Exception
{
SampleLexer lexer = new SampleLexer( new ANTLRFileStream(args[0]));
CommonTokenStream tokens = new CommonTokenStream( lexer );
SampleParser parser = new SampleParser( tokens );
ParseTree tree = parser.top();
ParseTreeWalker walker = new ParseTreeWalker();
walker.walk( new Walker(), tree );
}
}
public class Walker extends SampleBaseListener {
public void exitTop(SampleParser.TopContext ctx ) {
System.out.println( "Exit Top : " + ctx.mid() );
}
public String exitMid(SampleParser.MidContext ctx ) {
return ctx.identifier() + " bye "; // ignoring the 2nd instance here
}
public String exitIdentifier(SampleParser.IdentifierContext ctx ) {
return ctx.ID().getText() ;
}
}
But obviously this is wrong, because at the least, the return types of the Walker methods should be void, so they dont have a way to return aggregated values upstream. Secondly, I dont see a way how to access the "i1" and "i2" from the walker code, so I am not able to differentiate between the two instances of "identifier" in that rule.
Any suggestions on how to separate the actions from the grammar for this purpose?
Should I use a visitor instead of a listener here, since the visitor has the capability of returning values? If I use a visitor, how do I solve the problem of differentiating between "i1" and "i2" (as mentioned above)?
Does a visitor perform its action only at the exit of a rule (unlike the listeners, which exist for both entry and exit)? For example, if I have to initialize the list at the entry of rule "top", how can I do it with a visitor, which executes only at the conclusion of a rule? Do I need a enterTop listener for that purpose?
EDIT: After the initial post, I have modified the rule "top" to create and return a list, and pass this list back to the main program for printing. This is to illustrate why I need an initialization mechanism for the code.
Based on what you are trying to do I think you may benefit from using ANTLR's BaseVisitor Class rather than the BaseListener Class.
Assuming your grammar is this (I generalized it and I'll explain the changes below):
grammar Sample;
top : mid* ;
mid : i1=identifier 'hello' i2=identifier ;
identifier : ID ;
ID : [a-z]+ ;
WS : [ \t\r\n]+ -> skip ;
Then your Walker would look like this:
public class Walker extends SampleBaseVisitor<Object> {
public ArrayList<String> visitTop(SampleParser.TopContext ctx) {
ArrayList<String> arrayList = new ArrayList<>();
for (SampleParser.MidContext midCtx : ctx.mid()) {
arrayList.add(visitMid(midCtx));
}
return arrayList;
}
public String visitMid(SampleParser.MidContext ctx) {
return visitIdentifier(ctx.i1) + " bye " + visitIdentifier(ctx.i2);
}
public String visitIdentifier(SampleParser.IdentifierContext ctx) {
return ctx.getText();
}
}
This allows you to visit and get the result of any rule you want.
You are able to access i1 and i2, as you labeled them through the visitor methods. Note that you don't really need the identifier rule since it contains only one token and you can access a token's text directly in the visitMid, but really it's personal preference.
You should also note that SampleBaseVisitor is a generic class, where the generic parameter determines the return type of the visit methods. For your example I set the generic parameter Object, but you could even make your own class which contains the information you want to preserve and use that for your generic parameter.
Here are some more useful methods which BaseVisitor inherits which may help you out.
Lastly, your main method would end up looking something like this:
public static void main( String[] args) throws IOException {
FileInputStream fileInputStream = new FileInputStream(args[0]);
SampleLexer lexer = new SampleLexer(CharStreams.fromStream(fileInputStream));
CommonTokenStream tokens = new CommonTokenStream(lexer);
SampleParser parser = new SampleParser(tokens);
for (String string : new Walker().visitTop(parser.top())) {
System.out.println(string);
}
}
As a side note, the ANTLRFileStream class is deprecated in ANTLR4.
It is recommend to use CharStreams instead.
As Terence Parr points out in the Definitive Reference, one main difference between Visitor and Listener is that the Visitor can return values. And that can be convenient. But Listener has a place too! What I do for listener is exemplified in this answer. Granted, there are simpler ways of parsing a list of numbers, but I made that answer to show a complete and working example of how to aggregate return values from a listener into a public data structure that can be consumed later.
public class ValuesListener : ValuesBaseListener
{
public List<double> doubles = new List<double>(); // <<=== SEE HERE
public override void ExitNumber(ValuesParser.NumberContext context)
{
doubles.Add(Convert.ToDouble(context.GetChild(0).GetText()));
}
}
Looking closely at the Listener class, I include a public data collection -- a List<double> in this case -- to collect values parsed or calculated in the listener events. You can use any data structure you like: another custom class, a list, a queue, a stack (great for calculations and expression evaluation), whatever you like.
So while the Visitor is arguably more flexible, the Listener is a strong contender too, depending on how you want to aggregate your results.
I am using the 'visitor' pattern to generate XML from my parsed code. On typical context class looks like:
public static class On_dtmContext extends ParserRuleContext {
public List<FieldContext> field() {
return getRuleContexts(FieldContext.class);
}
public TerminalNode ON() { return getToken(SRC_REP_SCREENParser.ON, 0); }
public On_dtm_headerContext on_dtm_header() {
return getRuleContext(On_dtm_headerContext.class,0);
}
.....
}
and I access the element in my visitors call back function using RuleContext's 'getText' member function.
I would like to write a class inheriting from 'ParserRuleContext' and overload 'getText' in order to replace characters like '<' or '>' with their xml escape sequences. Is there a way I can have my code generated and having the context classes inheriting from my class, as:
public static class On_dtmContext extends XMLParserRuleContext {
public List<FieldContext> field() {
return getRuleContexts(FieldContext.class);
}
public TerminalNode ON() { return getToken(SRC_REP_SCREENParser.ON, 0); }
public On_dtm_headerContext on_dtm_header() {
return getRuleContext(On_dtm_headerContext.class,0);
}
.....
}
Thank you for your help!
Kind regards, wolf
Is there a reason why you are trying to extend the class, rather than creating a parser rule in your grammar to capture < and > so you can translate them as they occur?
The parser rules would look something like:
lessThan
: '<'
;
greaterThan
: '>'
;
At that point, you would have specific visitors for each of those terms and could translate them as you will.
It's been many years since my (one) compiler course so forgive me if this
question is ill-posed. I am also new to ANTLR and a C, not Java, coder.
What I'd like to do is describe my problem and then seek advice on the best
technique to employ.
I am trying to translate ASN.1 productions into a ML. For example,
Foo ::= ENUMERATED {
bar (0), -- some comment 0
baz (1) -- some comment 1
}
into
<Enumerated name="Foo">
<NamedValues>
<Unsigned name="bar" value="0" comment="some comment 0"/>
<Unsigned name="baz" value="1" comment="some comment 1"/>
</NamedValues>
</Enumerated>
My (simplified) ASN1 grammar is:
assignment : IDENTIFIER typeAssignment ;
typeAssignment : '::=' type ;
type : builtinType ;
builtinType : enumeratedType ;
enumeratedType : 'ENUMERATED' '{' enumerations '}' ;
...
Several of the examples in "The Definitive ANTLR4 Reference" demonstrate
overriding some enterNode or exitNode method in the BaseListener and
everything needed is in the Node's context. My problem is that I'd like
to override enterTypeAssignment and exitTypeAssignment, but some of the
info I need is in nodes higher (e.g. assignment) or lower (e.g. enumerations)
on the parse tree.
Is there enough description here to ask whether I should use a visitor
or listener pattern? Any advice on which book example(s) to focus on would
be greatly appreciated.
I'm having some luck with a brute force approach:
import org.antlr.v4.runtime.TokenStream;
import org.antlr.v4.runtime.misc.Interval;
public class MylListener extends ASN1BaseListener {
ASN1Parser parser;
String id = "";
String assignedType = "";
public MyListener(ASN1Parser parser) {this.parser = parser;}
#Override
public void enterAssignment(ASN1Parser.AssignmentContext ctx) {
id = ctx.IDENTIFIER().getText();
}
/** Listen to matches of typeAssignment **/
#Override
public void enterTypeAssignment(ASN1Parser.TypeAssignmentContext ctx) {
if ( ctx.type() != null ) {
if ( ctx.type().builtinType() != null ) {
if ( ctx.type().builtinType().enumeratedType() != null ) {
assignedType = "Enumerated";
System.out.println("");
System.out.println("<Enumerated name=\""+id+"\">");
...
}
}
}
}
#Override
public void exitTypeAssignment(ASN1Parser.TypeAssignmentContext ctx) {
if (assignedType.length() > 0) {
System.out.println("</"+assignedType+">");
assignedType = "";
}
}
}
but there's probably a more elegant solution...
UPDATE: I'm getting the result I want by saving TerminalNodes in global vars on my way down the tree and having those vars accessible to override methods in the Listener further down the tree. Is there a better way to access parent or grandparent context from a given node?
I'd first parse the thing and then use a visitor.
The advantage over using a Listener is that you have all the information available.
You can also use the C# target which may be easier to use with a C background.
I've seen some developers use the out keyword on parameter lists of void functions. I'm quite unclear on what the pros and cons are of code below:
List<string> listOfResult;
public void public void (out listOfResult)
{
//bla bla
}
versus
public List<string> c(out listOfResult)
{
List<string> list= new List<string>();
//bla bla
return list;
}
Are these two code snippets perfectly valid or is there any catch around the out keyword?
out keyword is handy when you need to return more than one value from function. Nice example is TryXXX methods, which return status of operation instead of throwing exceptions:
public bool TryParse(string str, out int value);
But I don't see any reason to use single out parameter with void methods... Simply return that value from your method. It will be much easier to use. Compare:
List<string> list;
GetList(out list); // confusing method name
With
List<string> list = GetList(); // nice name, one line of code
If getting of list could throw exceptions, then you can create method like this:
List<string> list;
if (TryGetList(out list)) // better than exception handling
{
// list was filled successfully
}
out parameters are quite handy when you need to return more than one value from a function.
e.g.
Return is a list of results, but you can use an out parameter to return an error message in the case when the list being returned is null.
It's a nice syntax to return multiple parameters. I personally think it's almost always better to model the return of the method as a "new object/class".
That would be:
class CResult
{
List<string> firstResult;
List<string> secondResult;
}
public CResult c()
{
// do something
return new CResult() {firstResult = ..., secondResult = ... };
}
You can see more things related to this approach here.
//out key word is used in function instead of return. we can use multiple parameters by using out key word
public void outKeyword(out string Firstname, out string SecondName)
{
Firstname = "Muhammad";
SecondName = "Ismail";
}
//on button click Event
protected void btnOutKeyword_Click(object sender, EventArgs e)
{
string first, second;
outKeyword(out first, out second);
lblOutKeyword.Text = first + " " + second;
}
is there a reason for the type of the second operand must be int?
...
// I would like to do this
public static StringList operator<<(StringList list, string s) {
list.Add(s);
return list;
}
// but only int is supported...
...
EDIT:
Just for sure... I can overload operator* for get (for example) List of string
class MyString {
string val;
public MyString(string s) {
val = s;
}
public static List<string> operator*(MyString s, int count) {
List<string> list = new List<string>();
while (count-- > 0) {
list.Add(s.val);
}
return list;
}
}
...
foreach (var s in new MyString("value") * 3) {
s.print(); // object extension (Console.WriteLine)
}
// output:
// value
// value
// value
...
but cannot overload left shift, well known from C++ std (overloaded for output), because it was unclear?
Of course, it's just a decision of C# designers.
Still it can be overloaded on something unexpected/unclear (with int).
Really the reason is that it was made an unclear code?
Yes. It's because the language specification requires it:
When declaring an overloaded shift operator, the type of the first operand must always be the class or struct containing the operator declaration, and the type of the second operand must always be int.
The language designers didn't have to make that decision - it would have been possible for them to remove that restriction if the wanted to - but I think this part of the specification explains their reasoning for this (and other) restrictions on operator overloading:
While it is possible for a user-defined operator to perform any computation it pleases, implementations that produce results other than those that are intuitively expected are strongly discouraged.
They probably wanted the bitshift operators to always behave like bitshift operators, and not as something completely surprising.
Because you want your code to look like c++? Why not an extension method .Append( item ) that returns the source list.
public static class ListExtensions
{
public static List<T> Append<T>( this List<T> source, T item )
{
if (source == null)
{
throw new NullArgumentException( "source" );
}
source.Add(item);
return source;
}
}
Used as:
var list = new List<string>();
list.Append( "foo" )
.Append( "bar" )
.Append( "baz" );