I am trying to parse a config, which would translate to a structured form. This new form requires that comments within the original config be preserved. The parsing tool is PLY. I am running into an issue with my current approach which I will describe in detail below, with links to code as well. The config file is going to look contain multiple config blocks, each of which is going to be of the following format
<optional comments>
start_of_line request_stmts(one or more)
indent reply_stmts (zero or more)
include_stmts (type 3)(zero or more)
An example config file looks like this.
While I am able to partially parse the config file with the grammar below, I fail to accomodate comments which would exist within the block.
For example, a block like this raises syntax errors, and any comments in a block of config fail to parse.
<optional comments>
start_of_line request_stmts(type 1)(one or more)
indent reply_stmts (type 2)(one or more)
<comments>
include_stmts (type 3)(one or more)(optional)
The parser.out mentions one shift/reduce conflict which I think arises because once the reply_stmts are parsed, a comments section which follows could mark start of a new block or comments within the subblock. Current grammar parsing result for the example file
[['# test comment ', '# more of this', '# does this make sense'], 'DEFAULT', [['x', '=',
'y']], [['y', '=', '1']], ['# Transmode', '# maybe something else', '# comment'],
'/random/location/test.user']
As you might notice, the second config block complete misses the username, request_stmt, reply_stmt sections.
What I have tried
I have tried moving the comments section around in the grammar, by specifying it before specific blocks or in the statement grammar. In the code link pasted above, the comments section has been specified in the overall statement grammar. Both of these approaches fail to parse comments within a config block.
username : comments username
| username
include_stmt : comments includes
| includes
I have two main questions:
Is there a mistake I am making in the implementation/understanding of LR parsing, solving which I could achieve what I want to ?
Is there a better way to achieve the same goal than my current approach ? (PLY-fu, different parser, different grammar)
P.S Wasn't able to include the actual code in the question, mentioned in the comments
You are correct that the problem is that when the parser sees a comment, it cannot know whether the comment belongs to the same section or whether the previous section is finished. In the former case, the parser needs to shift the comment, while in the latter case it needs to reduce the configuration section.
Since there could be any number of comments, the necessary lookahead could be arbitrarily large, in which case LR parsing wouldn't be possible. But a simple trick can reduce the lookahead to two tokens: just combine consecutive comments into a single token.
Any LR(k) grammar has an equivalent LR(1) grammar. In effect, the LR(1) grammars works by delaying all decisions for k-1 tokens, accumulating these tokens into the parser state. That's a massive increase in grammar size, but it's usually possible to achieve the same effect in other ways, and that's certainly the case here.
The basic idea is that any comment is (temporarily) accumulated into a list of comments. When a non-comment token is encountered, this temporary list is attached to that token.
This can be done either in the lexical scanner or in the parser actions, depending on your inclinations.
Before attempting all that, you should make sure that retaining comments is really useful to your application. Comments are normally not relevant to the semantics of a program (or configuration file), and it would certainly be much simpler for the lexer to just drop comments into the bit-bucket. If your application will end up reformatting the input, then it will have to retain comments. But if it only needs to extract information from the configuration, putting a lot of effort into handling comments is hard to justify.
Related
I've got a file format that looks a little like this:
blockA {
uniqueName42 -> uniqueName aWord1 anotherWord "Some text"
anotherUniqueName -> uniqueName23 aWord2
blockB {
thing -> anotherThing
}
}
Lots more blocks with arbitrary nesting levels.
The lines with the arrow in them define relationships between two things. Each relationship has some optional metadata (multi-word quoted or single word unquoted).
The challenge I'm having is that because the there can be an arbitrary number of metadata items in a relationship my parser is treating anotherUniqueName as a metadata item from the first relationship rather than the start of the second relationship.
You can see this in the image below. The parser is only recognising one relationshipDeclaration when a second should start with StringLiteral: anotherUniqueName
The parser looks a bit like this:
block
: BLOCK LBRACE relationshipDeclaration* RBRACE
;
relationshipDeclaration
: StringLiteral? ARROW StringLiteral StringLiteral*
;
I'm hoping to avoid lexical modes because the fact that these relationships can appear almost anywhere in the file will leave me up to my eyes in NL+ :-(
Would appreciate any ideas on what options I have. Is there a way to look ahead, spot the '->', for example?
Thanks a million.
Your example certainly looks like the NL is what signals the end of a relationshipDeclaration.
If that's the case, then you'll need NLs to be tokens available to your parse rules so the parser can know recognize the end.
As you've alluded to, you could potentially use -> to trigger a different Lexer Mode and generate different tokens for content between the -> and the NL and then use those tokens in your parse rule for relationshipDeclaration.
If it's as simple as your snippet indicates, then just capturing RD_StringLiteral tokens in that lexical mode, would probably be easier to deal with than handling all the places you might need to allow for NL. This would be pretty simple as Lexer modes go.
(BTW you can use x+ to get the same effect as x x*)
relationshipDeclaration
: StringLiteral? ARROW RD_StringLiteral+
;
I don't think there's a third option for dealing with this.
I'd like to merge two string variables (STRING_VAR1 and STRING_VAR2) into one string variable STRING_ALL, such that the content of STRING_VAR1 or STRING_VAR2 is copied into STRING_ALL depending on which of those two variables contain any data (see example_dataset). If both variables STRING_VAR1 and STRING_VAR2 contain missing cases, STRING_ALL should be missing as well.
I've tried CONCAT (see code below) but that doesn't work for some reason and leaves me with only empty cases for STRING_ALL.
STRING STRING_ALL(A4)
COMPUTE STRING_ALL = CONCAT(STRING_VAR1, STRING_VAR2)
Thanks in advance!
Eli's suggestion gave you the necessary information to solve this specific issue. If you want to know why, check the Command Order topic in the SPSS Statistics Command Syntax Reference. It discusses the different types of commands and the fact that some of them, such as COMPUTE, do not take effect immediately, but are stored pending execution of a command that causes a data pass.
I'm new to ANTL4 and I can't seem to figure out how to get lexer actions to perform properly.
I have a code snippet that looks for input text:
SIZE10 : [a-zA-Z]* {getText().length() <= 10}?
I would expect that it does not match any combinations of letters that are over 10 letters long, however what this does is treat a 10+ letter string as two different tokens, instead of just nullifying the whole set of 10+ letters. How can I get this action to nullify the whole set of letters?
In addition, where can I go to see all the different token functions I can use (other than getText())? The documentation about lexer actions is really poor. In general, I'm having a hard time figuring out what resources can give me a definitive list of everything in the language. Even an entry point into the source code for me to read would be good at this point. The documentation is too general/basic for me.
EDIT: I've figured out how to send a RuntimeException, but I don't know where to get the elements needed for a proper RecognitionException.
The predicate in a rule directs the parsing process in a way that allows to match only partial input (like in your case) or essentially switch off a part of the grammar depending on certain conditions. In your case the SIZE10 rule is matched until the predicate returns false. Everything up to this event is then returned as a match for SIZE10. After that lexing continues at the point it ended for the previous token and if that is again a letter it will again match SIZE10 as long as the predicate says it is correct. That's a bit different than what you would expect (e.g. using the predicate as an all or nothing switch).
However, if you instead want to match the full set of letters first and then check if the length is <= 10 you can do this in a listener. You can hook into the exitSIZE10() event and reject the match by throwing a recognition exception.
For the usable functions in your actions see the API documentation for ANTLR. For instance here is the one for Token which shows you other possibilities beside getText(). In your action, consider the context you have. In a lexer rule you deal with a Token, hence getText() etc. work on the token. In a parser rule you have a ParserContext instead, which also has a getText() function but that works differently (collecting all child contexts text into a comma separated list).
I would like to change the behavior of menhir's output in follwoing way:
I want it to look up all grammatical alternatives if it finds any, and put them in a list and get me back this ambigouus interpretation. It shall not reduce conflicts, just store them.
In the source code of menhir, it seems to me, that I have to look in "Engine.ml". The resultant syntactically determined token comes in a variant type item "Accepted v" as a state of a checkpoint of the grammatical automaton. This content is found by a function "accept env prod" before, that is part of a bundle of recursive functions, that change the states.
Do you have a tip, how I could change these functions to put all the possible results in the list here and proceed as if nothing happened? Or do you think, that this wont work anyway?
Thanks.
What you are looking for is a GLR parser generator (G is for generalized). Menhir is not such tool, and I doubt you could modify it easily to do what you want.
However, there is another tool that does exactly what you want: dypgen.
Consider the following sample codes:
1.Sample
var IsAdminUser = (User.Privileges == AdminPrivileges)
? 'yes'
: 'no';
console.log(IsAdminUser);
2.Sample
var IsAdminUser = (User.Privileges == AdminPrivileges)?'yes': 'no';
console.log(IsAdminUser);
The 2nd sample I am very comfortable with & I code in that style, but it was told that its wrong way of doing without any supportive reasons.
Why is it recommended not to use a single line ternary operator in Node.js?
Can anyone put some light on the reason why it is so?
Advance Thanks for great help.
With all coding standards, they are generally for readability and maintainability. My guess is the author finds it more readable on separate lines. The compiler / interpreter for your language will handle it all the same. As long as you / your project have a set standard and stick to it, you'll be fine. I recommend that the standards be worked on or at least reviewed by everyone on the project before casting them in stone. I think that if you're breaking it up on separate lines like that, you may as well define an if/else conditional block and use that.
Be wary of coding standards rules that do not have a justification.
Personally, I do not like the ternary operator as it feels unnatural to me and I always have to read the line a few times to understand what it's doing. I find separate if/else blocks easier for me to read. Personal preference of course.
It is in fact wrong to put the ? on a new line; even though it doesn’t hurt in practice.
The reason is a JS feature called “Automatic Semicolon Insertion”. When a var statement ends with a newline (without a trailing comma, which would indicate that more declarations are to follow), your JS interpreter should automatically insert a semicolon.
This semicolon would have the effect that IsAdminUser is assigned a boolean value (namely the result of User.Privileges == AdminPrivileges). After that, a new (invalid) expression would start with the question mark of what you think is a ternary operator.
As mentioned, most JS interpreters are smart enough to recognize that you have a newline where you shouldn’t have one, and implicitely fix your ternary operator. And, when minifying your script, the newline is removed anyway.
So, no problem in practice, but you’re relying on an implicit fix of common JS engines. It’s better to write the ternary operator like this:
var foo = bar ? "yes" : "no";
Or, for larger expressions:
var foo = bar ?
"The operation was successful" : "The operation has failed.";
Or even:
var foo = bar ?
"Congratulations, the operation was a total success!" :
"Oh, no! The operation has horribly failed!";
I completely disagree with the person who made this recommendation. The ternary operator is a standard feature of all 'C' style languages (C,C++,Java,C#,Javascript etc.), and most developers who code in these languages are completely comfortable with the single line version.
The first version just looks weird to me. If I was maintaining code and saw this, I would correct it back to a single line.
If you want verbose, use if-else. If you want neat and compact use a ternary.
My guess is the person who made this recommendation simply wasn't very familiar with the operator, so found it confusing.
Because it's easier on the eye and easier to read. It's much easier to see what your first snippet is doing at a glance - I don't even have to read to the end of a line. I can simply look at one spot and immediately know what values IsAdminUser will have for what conditions. Much the same reason as why you wouldn't write an entire if/else block on one line.
Remember that these are style conventions and are not necessarily backed up by objective (or technical) reasoning.
The reason for having ? and : on separate lines is so that it's easier to figure out what changed if your source control has a line-by-line comparison.
If you've just changed the stuff between the ? and : and everything is on a single line, the entire line can be marked as changed (based on your comparison tool).