JOOQ not ignoring unknown function despite configuration - jooq

I have a jOOQ codegen configuration failing because of a function not known by the framework. The configuration is using the org.jooq.meta.extensions.ddl.DDLDatabase, aiming to generate the code from the Flyway migration scripts. The error is as below:
Error running jOOQ code generation tool: Error while exporting schema: SQL [create table CUSTOMER (ID uuid not null default UUID_GENERATE_V4(), NAME varchar(255) not null, primary key (ID))]; Function "UUID_GENERATE_V4" not found;
By checking the documentation, I see there is a parseUnknownFunctions parser property that, as far as I understood, is supposed to disable this behaviour when set to IGNORE. This doesn't seem to have any effect.
I also understand there is a workaround to make jOOQ ignore parts of the SQL file by adding comments on it. This is not possible in my case, as I am not the owner of the SQL files.
Is there any other option I can try?
Below are the script causing the error and the jOOQ config in the pom.xml:
create table customer (
id uuid not null default uuid_generate_v4(),
name varchar(255) not null,
primary key (id)
);
<plugin>
<groupId>org.jooq</groupId>
<artifactId>jooq-codegen-maven</artifactId>
<executions>
<execution>
<id>generate-jooq-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<generator>
<database>
<name>org.jooq.meta.extensions.ddl.DDLDatabase</name>
<inputSchema>PUBLIC</inputSchema>
<outputSchemaToDefault>true</outputSchemaToDefault>
<outputCatalogToDefault>true</outputCatalogToDefault>
<properties>
<property>
<key>sort</key>
<value>semantic</value>
</property>
<property>
<key>scripts</key>
<value>src/main/resources/db/migration/*</value>
</property>
<property>
<key>parseUnknownFunctions</key>
<value>IGNORE</value>
</property>
</properties>
</database>
<target>
<clean>true</clean>
<packageName>com.product</packageName>
<directory>${project.generated-sources}/jooq/src/main/java</directory>
</target>
</generator>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.jooq</groupId>
<artifactId>jooq-meta-extensions</artifactId>
<version>${jooq.version}</version>
</dependency>
</dependencies>
</plugin>

If you look at the full stack trace, you'll see that the error isn't thrown by jOOQ but by H2:
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Function "UUID_GENERATE_V4" not found; SQL statement:
create table CUSTOMER (ID uuid not null default UUID_GENERATE_V4(), NAME varchar(255) not null, primary key (ID)) [90022-200]
at org.h2.message.DbException.getJdbcSQLException (DbException.java:576)
at org.h2.message.DbException.getJdbcSQLException (DbException.java:429)
at org.h2.message.DbException.get (DbException.java:205)
at org.h2.message.DbException.get (DbException.java:181)
at org.h2.command.Parser.readJavaFunction (Parser.java:3565)
at org.h2.command.Parser.readFunction (Parser.java:3770)
at org.h2.command.Parser.readTerm (Parser.java:4305)
at org.h2.command.Parser.readFactor (Parser.java:3343)
at org.h2.command.Parser.readSum (Parser.java:3330)
at org.h2.command.Parser.readConcat (Parser.java:3305)
at org.h2.command.Parser.readCondition (Parser.java:3108)
at org.h2.command.Parser.readExpression (Parser.java:3059)
at org.h2.command.Parser.parseColumnForTable (Parser.java:5727)
at org.h2.command.Parser.parseTableColumnDefinition (Parser.java:8442)
at org.h2.command.Parser.parseCreateTable (Parser.java:8379)
at org.h2.command.Parser.parseCreate (Parser.java:6276)
at org.h2.command.Parser.parsePrepared (Parser.java:903)
at org.h2.command.Parser.parse (Parser.java:843)
at org.h2.command.Parser.parse (Parser.java:815)
at org.h2.command.Parser.prepareCommand (Parser.java:738)
at org.h2.engine.Session.prepareLocal (Session.java:657)
at org.h2.engine.Session.prepareCommand (Session.java:595)
at org.h2.jdbc.JdbcConnection.prepareCommand (JdbcConnection.java:1235)
at org.h2.jdbc.JdbcStatement.executeInternal (JdbcStatement.java:212)
at org.h2.jdbc.JdbcStatement.execute (JdbcStatement.java:201)
at org.jooq.tools.jdbc.DefaultStatement.execute (DefaultStatement.java:102)
at org.jooq.impl.SettingsEnabledPreparedStatement.execute (SettingsEnabledPreparedStatement.java:227)
at org.jooq.impl.AbstractQuery.execute (AbstractQuery.java:411)
at org.jooq.impl.AbstractQuery.execute (AbstractQuery.java:332)
at org.jooq.meta.extensions.ddl.DDLDatabase.load (DDLDatabase.java:183)
at org.jooq.meta.extensions.ddl.DDLDatabase.lambda$0 (DDLDatabase.java:156)
The current (jOOQ 3.15) implementation of the DDLDatabase translates your DDL into the H2 dialect and runs it in an H2 in memory database to simulate your DDL. H2 doesn't support this function. Future jOOQ versions will interpret the DDL without using H2 (see https://github.com/jOOQ/jOOQ/issues/7034), but this is currently not the case.
As a workaround, you can use the jOOQ parser ignore comment syntax that is documented here:
https://www.jooq.org/doc/latest/manual/code-generation/codegen-ddl/
Specifically:
create table customer (
id uuid not null
/* [jooq ignore start] */ default uuid_generate_v4() /* [jooq ignore stop] */,
name varchar(255) not null,
primary key (id)
);
Alternatively (much better!) use one of the testcontainers based approaches documented here: https://github.com/jOOQ/jOOQ/issues/6551 or in this example: https://github.com/jOOQ/jOOQ/tree/main/jOOQ-examples/jOOQ-testcontainers-example

Related

Unable to use MySQL as Hive Metastore for Spark

I want to set up my local Spark to enable multiple connections. (i.e. notebook, BI tool, application, and etc) So I have to get away from Derby.
My hive-site.xml is as follows
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive_metastore?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>spark#localhost</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>spark</value>
</property>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
I set "datanucleus.schema.autoCreateTables" to true as suggested by Spark. "createDatabaseIfNotExist=true" does not seem to do anything.
But that still fails with
21/12/26 04:34:20 WARN Datastore: SQL Warning : 'BINARY as attribute of a type' is deprecated and will be removed in a future release. Please use a CHARACTER SET clause with _bin collation instead
21/12/26 04:34:20 ERROR Datastore: Error thrown executing CREATE TABLE `TBLS`
(
`TBL_ID` BIGINT NOT NULL,
`CREATE_TIME` INTEGER NOT NULL,
`DB_ID` BIGINT NULL,
`LAST_ACCESS_TIME` INTEGER NOT NULL,
`OWNER` VARCHAR(767) BINARY NULL,
`RETENTION` INTEGER NOT NULL,
`IS_REWRITE_ENABLED` BIT NOT NULL,
`SD_ID` BIGINT NULL,
`TBL_NAME` VARCHAR(256) BINARY NULL,
`TBL_TYPE` VARCHAR(128) BINARY NULL,
`VIEW_EXPANDED_TEXT` TEXT [CHARACTER SET charset_name] [COLLATE collation_name] NULL,
`VIEW_ORIGINAL_TEXT` TEXT [CHARACTER SET charset_name] [COLLATE collation_name] NULL,
CONSTRAINT `TBLS_PK` PRIMARY KEY (`TBL_ID`)
) ENGINE=INNODB : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[CHARACTER SET charset_name] [COLLATE collation_name] NULL,
and such.
Please advice.
Ok I did it.
So basically I can't rely on Spark to do this automatically, even though it was able to initialize the Derby version.
So I had to download both Hadoop and Hive, and use the schemaTool bundled within Hive to set up the metastore.
Then Spark is able to use that directly.
Alternatively, you could run Hive provided scripts directly on the DB. Different backends are at GitHub/Apache Hive: Metastore Scripts.

Touchpoint completed state got automatically deleted in HCL Connections 6.5.1

After enabling HCL Touchpoint on HCL Connections 6.5.1. According to the default configuration, we have setTimeDuration set in /mnt/opt/IBM/WebSphere/AppServer/profiles/CnxNode01/installedApps/CnxCell/Touchpoint.ear/touchpoint.war/js/startup.js. So it should re-appear only after 6 months. But on my test users, it re-appears just after ~1 hour.
Analyzing the problem: Deleted completed state
To debug/analyze this, I found out that touchpoint stores its data in the PEOPLEDB database, which contains a table EMPINST.PROFILE_EXTENSIONS. It uses PROF_PROPERTY_ID = 'touchpointState' to store the timestamp when touchpoint is completed (= a user confirms all steps). In this case, PROF_VALUE contains JSON like {"state":"complete","timestamp":1599763075000} which means the user completed it on 2020-09-10.
I created the following query to get the name, timestamp and the date in human readable form from the completed users:
SELECT e.PROF_DISPLAY_NAME, ext.PROF_VALUE, replace(REPLACE(ext.PROF_VALUE, '}', ''), '{"state":"complete","timestamp":', '') AS timestamp,
date((((replace(REPLACE(ext.PROF_VALUE, '}', ''), '{"state":"complete","timestamp":', '') / 1000)-5*3600)/86400)+719163) AS date
/*SELECT count(*)*/
FROM EMPINST.PROFILE_EXTENSIONS ext
LEFT JOIN EMPINST.EMPLOYEE e ON (e.PROF_KEY=ext.PROF_KEY)
WHERE PROF_PROPERTY_ID = 'touchpointState'
ORDER BY replace(REPLACE(ext.PROF_VALUE, '}', ''), '{"state":"complete","timestamp":', '') desc
Example result:
While this seemed to work, I re-run this query some time later (about 1h) and all those new rows were gone! They got deleted from the database. As a result, the users are redirected to touchpoint again and have to complete it a second time.
I don't know why they got deleted and how we can stop it. On the first run they were deleted after one admin user completed touchpoint. But later also after normal users ran them.
The problem was, that those attributes were left in ${tdisol}/TDI/conf/LotusConnections-config/tdi-profiles-config.xml:
<simpleAttribute extensionId="recommendedTags" length="256" sourceKey="recommendedTags" />
<simpleAttribute extensionId="departmentKey" length="256" sourceKey="departmentKey" />
<simpleAttribute extensionId="privacyAndGuidelines" length="256" sourceKey="privacyAndGuidelines" />
<simpleAttribute extensionId="touchpointState" length="256" sourceKey="touchpointState" />
<richtextAttribute extensionId="touchpointSession" maxBytes="1000000" sourceKey="touchpointSession" />
Just comment them out with <!-- and -->.
It's also required to remove them from ${tdisol}/TDI/conf/LotusConnections-config/profile-types.xml like this:
<!--
<property>
<ref>recommendedTags</ref>
<updatability>readwrite</updatability>
<hidden>true</hidden>
<fullTextIndexed>false</fullTextIndexed>
</property>
<property>
<ref>departmentKey</ref>
<updatability>read</updatability>
<hidden>true</hidden>
<fullTextIndexed>true</fullTextIndexed>
</property>
<property>
<ref>privacyAndGuidelines</ref>
<updatability>readwrite</updatability>
<hidden>true</hidden>
<fullTextIndexed>false</fullTextIndexed>
</property>
<property>
<ref>touchpointState</ref>
<updatability>readwrite</updatability>
<hidden>true</hidden>
<fullTextIndexed>false</fullTextIndexed>
</property>
<property>
<ref>touchpointSession</ref>
<updatability>readwrite</updatability>
<hidden>true</hidden>
<fullTextIndexed>false</fullTextIndexed>
</property>
-->
Even those attributes were set in the default tdisol configuration from HCL, it's wrong because this would lead TDI to overwrite ALL three touchpoint attributes from the LDAP. Usually those fields were not present in the domain. In this case, touchpoint DELETES all attributes - on every run.
Since our TDI was scheduled to run every 30 minutes, it deleted ALL TP related information on every run.
As an alternative, you could also exclude all listed properties above in map_dbrepos_from_source.properties like this:
extattr.privacyAndGuideLines=null
extattr.touchpointState=null
...

How to get response content in karate if the json response contains german characters?

I am trying to get GET response in Karate but whenever the response contains non english characters , it returns '???' for the field in the response.
Instead I want the actual characters to be returned.
{Id="abc", name="<Some German Text>"}
Actual Response returned :
{Id="abc", name="????"}
Karate script :
Given path 'URL'
When method get
Then status 200
* json convertJSON = response
Just add the following configuration for the maven-surefire-plugin
<configuration>
<argLine>-Dfile.encoding=UTF-8</argLine>
</configuration>
This is how it would look like in a typical pom.xml:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.10</version>
<configuration>
<argLine>-Dfile.encoding=UTF-8</argLine>
</configuration>
</plugin>
Do let me know if this works !

jooq sybase code generation issue

I am using sybase ASE as my database. When I am trying to generate code using jooq it shows following warning.but no code generation is done. Any help thanks in advance!
//--------------------------------warning-----------------------------------//
Jul 19, 2016 5:49:17 PM org.jooq.tools.JooqLogger warn
WARNING: No schemata were loaded : Please check your connection settings, and whether your database (and your database version!) is really supported by jOOQ. Also, check the case-sensitivity in your configured <inputSchema/> elements : [xfuse]
Jul 19, 2016 5:49:17 PM org.jooq.tools.JooqLogger info
INFO: Generating schemata : Total: 0
-->config file------->>>><<<------->>>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration xmlns="http://www.jooq.org/xsd/jooq-codegen-3.8.0.xsd">
<jdbc>
<driver>com.sybase.jdbc3.jdbc.SybDriver</driver>
<url>jdbc:sybase:Tds:192.168.xx.xx:5000/xfuse</url>
<user>xxx</user>
<password>xxx</password>
</jdbc>
<generator>
<database>
<name>org.jooq.util.ase.ASEDatabase</name>
<inputSchema>xfuse</inputSchema>
<includes>.*</includes>
<excludes></excludes>
</database>
<target>
<packageName>com.gen</packageName>
<directory>E:/RD/Test/codegeneration/output</directory>
</target>
</generator>
</configuration>
From your JDBC URL, I take that you're perhaps confusing the database name with the schema name.
jdbc:sybase:Tds:192.168.xx.xx:5000/xfuse
// database name ------------------^^^^^
What you should provide to <inputSchema> is your schema name. If you're unsure what your schema name is, then it is probably dbo in Sybase.
To be sure, you can also remove the <inputSchema/> element and run the code generator. That way, it will generate code for all of your schemas in the xfuse database.

Datastax 4.0 Error: Does not contain a valid host:port authority: ${dse.job.tracker}

Just got a single node cluster up and running with the new datastax 4.0.
Works great. We use hive to build and query our data.
On the server it self. I can start hive
$>dse hive
and query tables just fine.
When I try and use the newest Hive ODBC driver to run the same query I seeing this error.
It connects just fine, i can query the keyspace and see the tables. but when i try to run the query. Looks like the map/red gets in the queue, but then errors out with the following.
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ${dse.job.tracker}
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:147)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2584)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:474)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:402)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:646)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:630)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Job Submission failed with exception 'java.lang.IllegalArgumentException(Does not contain a valid host:port authority: ${dse.job.tracker})'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Any thoughts on what i should try?
Thanks ahead of time for any thoughts and or suggestions/assistance you all can provide.
Cheers,
Eric
I have solved the issue by manually configuring host:port into mapred-site.xml configuration file.
Just add the lines
<property>
<name>mapred.job.tracker</name>
<value>host:port</value>
</property>
depending on the ip address of your hive server and the used port (usually 8012).
This will override the default placeholder $(dse.job.tracker) present in dse-mapred-default.xml configuration file.
The dse.job.tracker property needs to be set in System properties of the JVM that starts Hadoop Jobs. Hadoop will substitute the placeholder with an appropriate system property value if it is defined. Otherwise, it will be just left as is, thus the error you see.
For hive, pig and mahout the mapred.job.tracker property is set in the bin/dse script as follows:
if [ -z "$HADOOP_JT" ]; then
HADOOP_JT=`$BIN/dsetool jobtracker --use-hadoop-config`
fi
if [ -z "$HADOOP_JT" ]; then
echo "Unable to run $HADOOP_CMD: jobtracker not found"
exit 2
fi
#set the JT param as a JVM arg
export HADOOP_OPTS="$HADOOP_OPTS -Ddse.job.tracker=$HADOOP_JT"
So you should do the same for your program using the Hive ODBC driver and I guess it should be fine.
By hardcoding Hadoop JT location you make it harder to move the JT to another node, because then you'd have to update the config file manually. Moreover, the automatic JT failover of dse won't work properly if your primary JT goes down, because your program would still try to connect the old one.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>local</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>localhost:50030</value>
</property>
<property>
<name>mapreduce.jobhhistory.webapp.address</name>
<value>localhost:19888</value>
</property>
</configuration>

Resources