Antlr4 param-value parsing - antlr4

Question

I'm new to antlr and trying to parse a file of mine (IBM Datastage export).
I have a working grammar but i'm not fully satisfied and I'm not sure I'm using antlr the right way :(
Here is an example of source file :
BEGIN HEADER
CharacterSet "CP1252"
ExportingTool "IBM InfoSphere DataStage Export"
ToolVersion "8"
ServerName "MIAIBV240"
ToolInstanceID "DFDMGBL2"
MDISVersion "1.0"
Date "2018-02-12"
Time "17.32.28"
ServerVersion "8.7"
END HEADER
BEGIN DSJOB
Identifier "job_FDM_WVD_NET_SALES_DSC_STG_REPL_Load"
DateModified "2018-02-12"
TimeModified "17.32.24"
BEGIN DSRECORD
Identifier "C52"
OLEType "CContainerStage"
Readonly "0"
Name "ShcFileAudit_Verif_Data"
NextID "3"
BEGIN DSSUBRECORD
Name "FDM_CMN"
Description "#FDM_CMN#"
ValueType "0"
END DSSUBRECORD
BEGIN DSSUBRECORD
Name "FILE_NAME"
FullDescription =+=+=+=#------------------------------------------------------------------------------- Description:
Some desc here ....
# Date Issue Version
=+=+=+=
JobVersion "56.0.0"
ControlAfterSubr "0"
Parameters "CParameters"
END DSSUBRECORD
END DSRECORD
END DSJOB
As you can see it is mostly Key/value pairs.
I use this grammar :
grammar dsxGrammar;
dsxFile : headerDeclaration? jobDeclaration* EOF;
headerDeclaration : BEGIN HEADER paramHeader* END HEADER;
jobDeclaration : BEGIN DSJOB paramJob* recordDeclaration* paramJob* END DSJOB;
recordDeclaration : BEGIN DSRECORD (paramRecord | subrecordDeclaration )* END DSRECORD;
subrecordDeclaration : BEGIN DSSUBRECORD paramSubRecord* END DSSUBRECORD;
paramHeader : PNAME PVALUE;
paramJob : PNAME PVALUE;
paramRecord : PNAME PVALUE;
paramSubRecord : PNAME PVALUE;
//PVALUE : '"' Text '"';
PVALUE : '"' .*? '"';
BEGIN : 'BEGIN';
END : 'END';
HEADER : 'HEADER';
DSJOB : 'DSJOB';
DSRECORD : 'DSRECORD';
DSSUBRECORD : 'DSSUBRECORD';
ORCHCODE : 'OrchestrateCode =+=+=+=' .*? '=+=+=+=' -> skip;
FULLDESC : 'FullDescription =+=+=+=' .*? '=+=+=+=' -> skip;
VALUECODE : 'Value =+=+=+=' .*? '=+=+=+=' -> skip;
EXPRCODE : 'Expression =+=+=+=' .*? '=+=+=+=' -> skip;
DERIVCODE : 'Derivation =+=+=+=' .*? '=+=+=+=' -> skip;
WS : [ \t\r\n\u000C]+ -> skip;
SPACE : ' ';
COMMENT : '/*' .*? '*/' -> skip;
LINE_COMMENT : '//' ~[\r\n]* -> skip;
//Identifier : NONDIGIT ( NONDIGIT | DIGIT)*;
fragment Text : NONDIGIT ( SPACE | NONDIGIT | DIGIT)*;
fragment NONDIGIT : [a-zA-Z_] ;
fragment DIGIT : [0-9];
PNAME : ~["\\ \t\r\n]+;
It's working but i have to parse PNAME and PVALUE afterwards in my python code to extract double quotes, etc...
Is it the right way to do or I can do something with a Text fragment ?
I would like to have access to HeadProperty.PNAME and HeadProperty.PVALUE for example.
Thanks for your help !

Answer 1

[...] Is it the right way to do or I can do something with a Text fragment ?
Not in the grammar (at least, not without using target specific code). You'll have to change the text in a listener or visitor. In your case, I'd go for a visitor. The code below is in Java, but should be easily converted to Python.
First a couple of remarks about your current grammar:
lexer rules like these 'OrchestrateCode =+=+=+=' can be fragile. If the input contains something like this: OrchestrateCode =+=+=+= (2 spaces), it won't be tokenised properly
your rule SPACE : ' '; is only used in a fragment, so it should also be a fragment
you're repeating PNAME PVALUE quite a lot in your parser rules: I'd create a keyValues rule for it (this also simplifies the visitor which I'll add below)
the same for =+=+=+= in your lexer rule: I'd create a fragment that matches this (DRY!)
It looks like a lot of feedback, but they're all minor points. Given that you're new to ANTLR, I think you did a good job.
OK, given the points above, your grammar could look like this:
grammar dsxGrammar;
dsxFile : headerDeclaration? jobDeclaration* EOF;
headerDeclaration : BEGIN HEADER paramHeaders=keyValues END HEADER;
jobDeclaration : BEGIN DSJOB paramJob+=keyValues recordDeclaration* paramJob+=keyValues END DSJOB;
recordDeclaration : BEGIN DSRECORD ( paramRecord+=keyValue | subrecordDeclaration )* END DSRECORD;
subrecordDeclaration : BEGIN DSSUBRECORD paramSubRecords=keyValues END DSSUBRECORD;
keyValues : keyValue*;
keyValue : PNAME PVALUE;
PVALUE : '"' .*? '"';
BEGIN : 'BEGIN';
END : 'END';
HEADER : 'HEADER';
DSJOB : 'DSJOB';
DSRECORD : 'DSRECORD';
DSSUBRECORD : 'DSSUBRECORD';
ORCHCODE : 'OrchestrateCode' SPACE* SEPARATOR .*? SEPARATOR -> skip;
FULLDESC : 'FullDescription' SPACE* SEPARATOR .*? SEPARATOR -> skip;
VALUECODE : 'Value' SPACE* SEPARATOR .*? SEPARATOR -> skip;
EXPRCODE : 'Expression' SPACE* SEPARATOR .*? SEPARATOR -> skip;
DERIVCODE : 'Derivation' SPACE* SEPARATOR .*? SEPARATOR -> skip;
WS : SPACE+ -> skip;
COMMENT : '/*' .*? '*/' -> skip;
LINE_COMMENT : '//' ~[\r\n]* -> skip;
PNAME : ~["\\ \t\r\n]+;
fragment NONDIGIT : [a-zA-Z_] ;
fragment DIGIT : [0-9];
fragment SPACE : [ \t\r\n\u000C];
fragment SEPARATOR : '=+=+=+=';
You might notice the labels in the parser rules, like paramSubRecords=... and paramJob+=.... These are good for 2 things:
readability of your grammar (keyValues doesn't mean much, while it's clear what keyValues really is in case paramSubRecords=keyValues is written)
in your listener or visitor, you can reference these labeled children from a parent context. Say you have the rule
foo : first=PNAME second=PNAME;
then you can do this in a visitor:
#Override
public Object visitFoo(dsxGrammarParser.FooContext ctx) {
String firstName = ctx.first.getText();
String secondName = ctx.second.getText();
...
}
And if you have a += in there like this:
foo : allJobs+=jobDeclaration+ PNAME allJobs+=jobDeclaration+;
then all jobDeclaration contexts are added in a java.util.List you can use in the parent context:
#Override
public Object visitFoo(dsxGrammarParser.FooContext ctx) {
List<dsxGrammarParser.JobDeclarationContext> jobs = ctx.allJobs;
...
}
If you now generate a visitor from the updated grammar, you can test it like this:
public class Main {
public static void main(String[] args) {
String source = "BEGIN HEADER\n" +
" CharacterSet \"CP1252\"\n" +
" ExportingTool \"IBM InfoSphere DataStage Export\"\n" +
" ToolVersion \"8\"\n" +
" ServerName \"MIAIBV240\"\n" +
" ToolInstanceID \"DFDMGBL2\"\n" +
" MDISVersion \"1.0\"\n" +
" Date \"2018-02-12\"\n" +
" Time \"17.32.28\"\n" +
" ServerVersion \"8.7\"\n" +
"END HEADER\n" +
"BEGIN DSJOB\n" +
" Identifier \"job_FDM_WVD_NET_SALES_DSC_STG_REPL_Load\"\n" +
" DateModified \"2018-02-12\"\n" +
" TimeModified \"17.32.24\"\n" +
" BEGIN DSRECORD\n" +
" Identifier \"C52\"\n" +
" OLEType \"CContainerStage\"\n" +
" Readonly \"0\"\n" +
" Name \"ShcFileAudit_Verif_Data\"\n" +
" NextID \"3\"\n" +
" BEGIN DSSUBRECORD\n" +
" Name \"FDM_CMN\"\n" +
" Description \"#FDM_CMN#\"\n" +
" ValueType \"0\"\n" +
" END DSSUBRECORD\n" +
" BEGIN DSSUBRECORD\n" +
" Name \"FILE_NAME\"\n" +
" FullDescription =+=+=+=#------------------------------------------------------------------------------- Description:\n" +
"\n" +
"Some desc here .... \n" +
"# Date Issue Version \n" +
"=+=+=+=\n" +
" JobVersion \"56.0.0\"\n" +
" ControlAfterSubr \"0\"\n" +
" Parameters \"CParameters\"\n" +
" END DSSUBRECORD\n" +
"END DSRECORD\n" +
"END DSJOB";
dsxGrammarLexer lexer = new dsxGrammarLexer(CharStreams.fromString(source));
dsxGrammarParser parser = new dsxGrammarParser(new CommonTokenStream(lexer));
DsxFile dsxFile = new MapVisitor().visitDsxFile(parser.dsxFile());
System.out.println(dsxFile);
}
}
class MapVisitor extends dsxGrammarBaseVisitor<Object> {
// dsxFile : headerDeclaration? jobDeclaration* EOF;
#Override
public DsxFile visitDsxFile(dsxGrammarParser.DsxFileContext ctx) {
Map<String, String> headerDeclaration = visitHeaderDeclaration(ctx.headerDeclaration());
List<JobDeclaration> jobDeclarations = new ArrayList<>();
if (ctx.jobDeclaration() != null) {
for (dsxGrammarParser.JobDeclarationContext child : ctx.jobDeclaration()) {
jobDeclarations.add(visitJobDeclaration(child));
}
}
return new DsxFile(headerDeclaration, jobDeclarations);
}
// headerDeclaration : BEGIN HEADER paramHeaders=keyValues END HEADER;
#Override
public Map<String, String> visitHeaderDeclaration(dsxGrammarParser.HeaderDeclarationContext ctx) {
return visitKeyValues(ctx.paramHeaders);
}
// jobDeclaration : BEGIN DSJOB paramJob+=keyValues recordDeclaration* paramJob+=keyValues END DSJOB;
#Override
public JobDeclaration visitJobDeclaration(dsxGrammarParser.JobDeclarationContext ctx) {
Map<String, String> paramJobs = new LinkedHashMap<>();
List<RecordDeclaration> recordDeclarations = new ArrayList<>();
for (dsxGrammarParser.KeyValuesContext child : ctx.paramJob) {
paramJobs.putAll(visitKeyValues(child));
}
for (dsxGrammarParser.RecordDeclarationContext child : ctx.recordDeclaration()) {
recordDeclarations.add(visitRecordDeclaration(child));
}
return new JobDeclaration(paramJobs, recordDeclarations);
}
// recordDeclaration : BEGIN DSRECORD ( paramRecord+=keyValue | subrecordDeclaration )* END DSRECORD;
#Override
public RecordDeclaration visitRecordDeclaration(dsxGrammarParser.RecordDeclarationContext ctx) {
Map<String, String> paramRecords = new LinkedHashMap<>();
List<Map<String, String>> subrecordDeclarations = new ArrayList<>();
for (dsxGrammarParser.KeyValueContext child : ctx.paramRecord) {
String[] keyValue = visitKeyValue(child);
paramRecords.put(keyValue[0], keyValue[1]);
}
for (dsxGrammarParser.SubrecordDeclarationContext child : ctx.subrecordDeclaration()) {
subrecordDeclarations.add(visitSubrecordDeclaration(child));
}
return new RecordDeclaration(paramRecords, subrecordDeclarations);
}
// subrecordDeclaration : BEGIN DSSUBRECORD paramSubRecords=keyValues END DSSUBRECORD;
#Override
public Map<String, String> visitSubrecordDeclaration(dsxGrammarParser.SubrecordDeclarationContext ctx) {
return visitKeyValues(ctx.paramSubRecords);
}
// keyValues : keyValue*;
#Override
public Map<String, String> visitKeyValues(dsxGrammarParser.KeyValuesContext ctx) {
Map<String, String> map = new LinkedHashMap<>();
if (ctx.keyValue() != null) {
for (dsxGrammarParser.KeyValueContext child : ctx.keyValue()) {
String[] keyValue = visitKeyValue(child);
map.put(keyValue[0], keyValue[1]);
}
}
return map;
}
// keyValue : PNAME PVALUE;
#Override
public String[] visitKeyValue(dsxGrammarParser.KeyValueContext ctx) {
return new String[] {
ctx.PNAME().getText(),
ctx.PVALUE().getText().substring(1, ctx.PVALUE().getText().length() - 1)
};
}
}
// headerDeclaration? jobDeclaration* EOF
class DsxFile {
final Map<String, String> headerDeclaration;
final List<JobDeclaration> jobDeclarations;
DsxFile(Map<String, String> headerDeclaration, List<JobDeclaration> jobDeclarations) {
this.headerDeclaration = headerDeclaration;
this.jobDeclarations = jobDeclarations;
}
#Override
public String toString() {
return "DsxFile{" +
"\n headerDeclaration=" + headerDeclaration +
"\n jobDeclarations=" + jobDeclarations +
"\n}";
}
}
// BEGIN DSJOB paramJob+=keyValues recordDeclaration* paramJob+=keyValues END DSJOB
class JobDeclaration {
final Map<String, String> paramJobs;
final List<RecordDeclaration> recordDeclarations;
JobDeclaration(Map<String, String> paramJobs, List<RecordDeclaration> recordDeclarations) {
this.paramJobs = paramJobs;
this.recordDeclarations = recordDeclarations;
}
#Override
public String toString() {
return "\n JobDeclaration{" +
"\n paramJobs=" + paramJobs +
"\n recordDeclarations=" + recordDeclarations +
"\n }";
}
}
// BEGIN DSRECORD ( paramRecord+=keyValue | subrecordDeclaration )* END DSRECORD
class RecordDeclaration {
final Map<String, String> paramRecords;
final List<Map<String, String>> subrecordDeclarations;
RecordDeclaration(Map<String, String> paramRecords, List<Map<String, String>> subrecordDeclarations) {
this.paramRecords = paramRecords;
this.subrecordDeclarations = subrecordDeclarations;
}
#Override
public String toString() {
return "\n RecordDeclaration{" +
"\n paramRecords=" + paramRecords +
"\n subrecordDeclarations=" + subrecordDeclarations +
"\n }";
}
}
If you run the main class, you'll see the following being printed to your console:
DsxFile{
headerDeclaration={CharacterSet=CP1252, ExportingTool=IBM InfoSphere DataStage Export, ToolVersion=8, ServerName=MIAIBV240, ToolInstanceID=DFDMGBL2, MDISVersion=1.0, Date=2018-02-12, Time=17.32.28, ServerVersion=8.7}
jobDeclarations=[
JobDeclaration{
paramJobs={Identifier=job_FDM_WVD_NET_SALES_DSC_STG_REPL_Load, DateModified=2018-02-12, TimeModified=17.32.24}
recordDeclarations=[
RecordDeclaration{
paramRecords={Identifier=C52, OLEType=CContainerStage, Readonly=0, Name=ShcFileAudit_Verif_Data, NextID=3}
subrecordDeclarations=[{Name=FDM_CMN, Description=#FDM_CMN#, ValueType=0}, {Name=FILE_NAME, JobVersion=56.0.0, ControlAfterSubr=0, Parameters=CParameters}]
}]
}]
}