how to deal with multiple tokens files? - antlr4

Question

I have a complicated grammar, with multiple pre-processors. Each pre-processor lives in its own package. Each one generates a tokens file. When i use a makefile to cd to the package directory and invoke antlr on the grammar it creates the tokens file in the current working directory and then carries on and process the grammar all is good.
When I use the gradle antlr4 plugin the tokens files gets put into generated-src/...packagedir... All good but antlr then cannot find the generated tokens file. I tried to use the -lib parameter but that only points to a single directory.
This is my project structure
src/..package../preproc1
preproc1.g4
preproc1Tokens.g4
src/..package../preproc2
preproc2.g4
preproc2Tokens.g4
When i run antlr on the g4 files I get
build/generated-src/antlr/main/..package../preproc1/preproc1.tokens
but the antlr process working directory is still
src/..package../preproc2 and so antlr cannot find this generated tokens file.
I tried to tell the antlr plugin to generate the output files in the src directory, as the makefile does. Not the way I would choose to do it but just wanted to keep moving forward. Unfortunately the antlr plugin does a delete on the outputdirectory using GFileUtils.cleanDirectory(outputDirectory); in the execute task. I can alter this to not do the delete if the outputDirectory is the same as the source directory, which would be ok I guess, though clean would never work properly.
I have resorted to doing a build with a makefile to get the tokens files, then copying these tokens files to a single directory and pointing -lib to that directory.
Does anyone have a better solution?

Answer 1

I use Maven instead of Gradle. Whether this is a "better" solution I can't say,
but the ANTLR 4 API Page currently favors Maven over Gradle, as there is not a Gradle plugin link that I can see. Having said that, I would hope that the two are compatible enough that the ANTLR build directories should match.
ANTLR Instructions
When I follow the ANTLR source layout as suggested by their ANTLR 4 Maven plugin instructions, then it worked for me pretty much out of the box.
The ANTLR 4 Maven plugin instructions by default will build .g4 files contained in the project folder:
<project>/src/main/antlr4/
whereas you would copy any shared .g4 files which are to be used as an imports into the specific import sub-directory:
<project>/src/main/antlr4/imports/
My pom.xml file
The following ANTLR plugin configuration <plugin\> text is set in the <project><build><plugins\> section of my pom.xml file.
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>4.5.3</version>
<executions>
<execution>
<configuration>
<outputDirectory>target/generated-sources/java</outputDirectory>
<arguments>
<argument>-visitor</argument>
</arguments>
<libDirectory>src/main/antlr4/imports</libDirectory>
</configuration>
<id>antlr</id>
<goals>
<goal>antlr4</goal>
</goals>
</execution>
</executions>
</plugin>
I found that I needed to be explicit with the <libDirectory\> entry, even though I thought that this would be the default setting.
My Project Structure
I'm working with three grammars in my ComojoProj project: Countables, Dater, and Sched. Sched imports the other two.
Each of the grammar files are in a separate sub-folder hanging off of src/main/antlr/, more or less aligning with the packages of the associated java implementations:
--- /ComojoProj/
--- pom.xml
--- .project
--- ./src/main/antlr/
--- ./com/wapitia/common/Countables.g4
--- ./com/wapitia/dates/Dater.g4
--- ./com/wapitia/sched/Sched.g4
--- ./src/main/antlr/imports/
--- Countables.g4
--- Dater.g4
--- ./src/main/java/
--- ./com/wapitia/
--- ./common/parse/antlr/CountablesProducer.java
--- ./dates/parse/antlr/DaterProducer.java
--- ./dates/parse/nodes/
--- DayOfWeekNode.java ...
--- ./sched/parse/antlr/SchedProducer.java
--- ./sched/parse/nodes/
--- MonthlyNode.java, EndDateNode.java, ...
The Sched.g4 grammar file starts like this:
grammar Sched;
import Countables, Dater;
schedule : dailySched | weeklySched ...
The two Countables.g4, Dater.g4 grammar files are found during the Sched.g4 build, because we copied them into the /src/main/antlr/imports/ directory.
When the ANTLR plugin runs it generates the following code
--- /ComojoProj/target/generated-sources/java/
--- com.wapitia.common/
--- CountablesBaseListener.java
--- CountablesBaseVisitor.java
...
--- com.wapitia.dates/ ...
--- com.wapitia.sched/ ...
--- Countables.tokens
--- CountablesLexer.tokens
...
--- SchedLexer.tokens
So the .tokens files are generated into the "top" package of some java
source directory, and so wind up in the top of the generated jar file.
I hope this helps!