As I understand, GridGain 6 has some customized serialization and also utlizes H2 for various purposes.
We use H2 as a serialized object store. For instance, here is the relevant part of a table schema.
CREATE TABLE IF NOT EXISTS QUEUE (ID IDENTITY PRIMARY KEY, OBJECT OTHER NOT NULL ....)
When attempting to insert a row, I get the following error. The last few lines indicate that the GridH2IndexingSpi is configured and is failing on something (even though my test isn't running on the Grid). I couldn't easily debug further since the spi source and my debugger seem out of sync and the line numbers are meaningless.
From what I was able to debug in Utils.java, It appears that the gridgain serializer has been configured in H2 (statically !!!!) and is being used.
Any thoughts on how to resolve or avoid this situation? I've tried various H2 versions such as 1.3.176 (which gridgain uses) and the newer 1.4.177, but as expected, they don't make any difference since the issue is with the use the indexing spi.
I can try and create a small H2 / Gridgain project project to illustrate the issue, if that would help.
Thanks
Exception in thread "pool-4-thread-1" org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL []; SQL state [90026]; error code [90026]; Serialization failed, cause: "java.lang.NullPointerException"; SQL statement:
INSERT INTO QUEUE (OBJECT....) VALUES (?,?,?,?) [90026-170]; nested exception is org.h2.jdbc.JdbcSQLException: Serialization failed, cause: "java.lang.NullPointerException"; SQL statement:
INSERT INTO QUEUE (OBJECT, ....) VALUES (?,?,?,?) [90026-170]
....
Caused by: org.h2.jdbc.JdbcSQLException: Serialization failed, cause: "java.lang.NullPointerException"; SQL statement:
INSERT INTO QUEUE (OBJECT, ....) VALUES (?,?,?,?) [90026-170]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
....
Caused by: java.lang.NullPointerException
at org.gridgain.grid.spi.indexing.h2.GridH2IndexingSpi.access$100(GridH2IndexingSpi.java:145)
at org.gridgain.grid.spi.indexing.h2.GridH2IndexingSpi$1.serialize(GridH2IndexingSpi.java:201)
at org.h2.util.Utils.serialize(Utils.java:273)
... 27 more
I finally understood what's happening. At the point when GridGain H2 integration was implemented, H2 had only one static serializer. That's why GridGain is using a static property. As a possible workaround for the problem can you try setting in your H2 database your custom serializer which will in fact utilize usual H2 serialization?
Latest H2 version supports specifying per-database serializer and GridGain will fix that in upcoming version.
I am not sure what it is you are doing with H2, but you should not utilize the same H2 database as used by GridGain internally.
GridGain utilizes H2 internally exclusively for SQL indexing and querying capabilities.
Related
I just updated to Spring Boot 2.7.2 and the new H2 2.1.214.
The jOOQ version is 3.16.6 (pro).
Since then, I get a bad grammar SQL exception with a limit query.
If I understand it correctly, the keyword limit is no longer supported in H2 - instead, FETCH FIRST should be used.
dslContext.select( FOO.fields() ).from( FOO ).limit( 1 )
select "FOO".ID" from "FOO" limit ?
Stacktrace:
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Syntax error in SQL statement "select ""FOO"".""ID"" from ""FOO"" limit [*]?"; SQL statement:
select "FOO"."ID" from "FOO" limit ? [42000-214]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.message.DbException.getSyntaxError(DbException.java:247)
at org.h2.command.Parser.getSyntaxError(Parser.java:898)
at org.h2.command.Parser.prepareCommand(Parser.java:572)
at org.h2.engine.SessionLocal.prepareLocal(SessionLocal.java:631)
at org.h2.engine.SessionLocal.prepareCommand(SessionLocal.java:554)
at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1116)
at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:92)
at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:288)
at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
at jdk.internal.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.performQueryExecutionListener(ConnectionProxyLogic.java:112)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.access$000(ConnectionProxyLogic.java:25)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic$1.execute(ConnectionProxyLogic.java:50)
at net.ttddyy.dsproxy.listener.MethodExecutionListenerUtils.invoke(MethodExecutionListenerUtils.java:42)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.invoke(ConnectionProxyLogic.java:47)
at net.ttddyy.dsproxy.proxy.jdk.ConnectionInvocationHandler.invoke(ConnectionInvocationHandler.java:25)
at jdk.proxy2/jdk.proxy2.$Proxy140.prepareStatement(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.performQueryExecutionListener(ConnectionProxyLogic.java:112)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.access$000(ConnectionProxyLogic.java:25)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic$1.execute(ConnectionProxyLogic.java:50)
at net.ttddyy.dsproxy.listener.MethodExecutionListenerUtils.invoke(MethodExecutionListenerUtils.java:42)
at net.ttddyy.dsproxy.proxy.ConnectionProxyLogic.invoke(ConnectionProxyLogic.java:47)
at net.ttddyy.dsproxy.proxy.jdk.ConnectionInvocationHandler.invoke(ConnectionInvocationHandler.java:25)
at jdk.proxy2/jdk.proxy2.$Proxy140.prepareStatement(Unknown Source)
at org.quickperf.sql.connection.QuickPerfDatabaseConnection.prepareStatement(QuickPerfDatabaseConnection.java:62)
at jdk.internal.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:238)
at jdk.proxy2/jdk.proxy2.$Proxy320.prepareStatement(Unknown Source)
at org.jooq.impl.ProviderEnabledConnection.prepareStatement(ProviderEnabledConnection.java:109)
at org.jooq.impl.SettingsEnabledConnection.prepareStatement(SettingsEnabledConnection.java:82)
at org.jooq.impl.AbstractResultQuery.prepare(AbstractResultQuery.java:210)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:307)
... 92 more
Is this a known issue?
Is there a way to rewrite the query?
This seems to be a frequent misconception about how to use jOOQ with H2 as a test database, so I've written a blog post about jOOQ's stance on this. The feature was also requested on the jOOQ issue tracker as #13895
TL;DR: With jOOQ, please don't use H2's compatibility modes.
Valid configurations
There is only really 1 valid configuration when using H2 as a test database product for a different production database product:
Use jOOQ's SQLDialect.H2 and H2 native, without compatibility mode. This is integration tested by jOOQ.
You might be tempted to:
Use jOOQ's SQLDialect.SQLSERVER and H2 with SQL Server compatibility mode. This is not integration tested by jOOQ (see details below). I don't recommend doing this, because it is likely you'll run into a limitation of H2's compatibility mode that jOOQ assumes is there, because jOOQ thinks it's an actual SQL Server instance.
Use SQLDialect.H2 with H2 in compatibility mode doesn't really make sense, because, well, LIMIT is valid in H2, but not in SQL Server, which supports only TOP or OFFSET .. FETCH. So you're not going to get a good SQL dialect match. See e.g.: https://github.com/h2database/h2database/issues/3537
Some background on compatibility modes and using H2 as a test DB
The assumption in jOOQ is when you use H2, then you use H2 natively (e.g. as an in-memory database, also in production). Using H2 as a simple test database isn't the primary use-case for H2, even if it has seen a lot of popularity in recent years. But why not just use a testcontainers based approach instead, and develop / integration test only with your production RDBMS? The benefits are obvious:
You don't have any such infrastructure problems and test artifacts
You get to use all the vendor specific features, among which table valued functions, table valued parameters, XML/JSON, etc.
The H2 compatibility modes are to make sure your native SQL Server queries work on H2 without any changes. This is useful for purely JDBC based applications. But since you're using jOOQ, and SQLDialect.H2, why keep a compatibility mode around in H2? jOOQ already handles the translation between dialects, if you want to continue using H2 as a test database product. But again, I think your life will be simpler if you're using testcontainers. You can even use it to generate jOOQ code as shown here.
I am attempting to add a field to a user defined type in cassandra 2.1.2, using the nodejs driver from datastax. I added the field using ALTER TYPE in cqlsh. When I attempt to add a row containing the udt with a value for the new field, it gets inserted with null value, instead of value I supplied. I strongly suspect this has to do with the way the cluster is caching the prepared statement. Because I recall reading that the prepared statements are indexed by a hash of the query, I tried changing some whitespace in the query to see if it helped.This actually seemed to work, but only once. subsequent inserts result in error:
message: 'Operation timed out - received only 0 responses.',
info: 'Represents an error message from the server',
code: 4352,
consistencies: 10,
received: 0,
blockFor: 1,
writeType: 'SIMPLE',
coordinator: '127.0.0.1:9042',
and it would seem the new rows are not added.. until I restart cassandra, at which point not only do the inserts that I thought had failed show up, but subsequent ones work fine. This is very disconcerting, but fortunately I have only done this in test instances. I do need to make this change in production however, and restarting the cluster to add a single field is not really an option. Is there a better way to get the cluster to evict the cached prepared statement?
I strongly suspect this has to do with the way the cluster is caching the prepared statement.
Put Cassandra log in DEBUG mode to be sure the prepared statement cache is the root cause. If it is, create an JIRA so the dev team can fix it...
Optionally you can also enable tracing to see what is going on server-side
To enable tracing in cqlsh, just type TRACING ON
To enable tracing with the Java driver, just call enableTracing() on the statement object
Houston, we have a problem.
Trying to create a new table with cqlsh on an existing Cassandra (v2.1.3) keyspace results in:
ServerError:
<ErrorMessage code=0000 [Server error] message="java.lang.RuntimeException:
java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found e8c03790-c952-11e4-a753-5981ea73cd7c; expected e8b14370-c952-11e4-a844-8f10bfb9c386)">
After the first create attempt, trying once more will result in:
AlreadyExists: Table 'ks.metrics' already exists
But retrieving the list of existing tables for the keyspace desc tables; will not report the new table.
The issue seems related to Cassandra-8387 except that there's only one client trying to create the table: cqlsh
We do have a bunch of Spark jobs that will create the keyspaces and tables at startup, potentially doing this in parallel. Would this render the keyspace corrupt?
Creating a new keyspace and adding a table to it works as expected.
Any ideas?
UPDATE
Found a workaround: issue a repair on the keyspace and the tables will appear (desc tables) and are also functional.
Short answer: They have a race condition, which they think they resolved in 1.1.8...
Long answer:
I get that error all the time on one of my clusters. I have test machines that have really slow hard drives and creating one or two tables is enough to get the error when I have 4 nodes on two separate computers.
Below I have a copy of the stack trace from my Cassandra 3.7 installation. Although your version was 2.1.3, I would be surprised that this part of the code changed that much.
As we can see, the exception happens in the validateCompatibility() function. This requires that the new and old versions of the MetaData have these equal:
ksName (keyspace name)
cfName (columnfamily name)
cfId (columnfamily UUID)
flags (isSuper, isCounter, isDense, isCompound)
comparator (key sorting comparator)
If any one of these values do not match between the old and new meta data, then the process raises an exception. In our case, the cfId values are different.
Going up the stack, we have the apply() which calls validateCompatibility() immediately.
Next we have updateTable(). Similarly, it calls apply() nearly immediately. First it calls the getCFMetaData() to retrieve the current column family data ("old") that is going to be compared against the new data.
Next we see updateKeyspace(). That function calculates a diff to know what changed. Then it saves that in each type of data. Table is 2nd after Type...
Before that they have the mergeSchema() which calculates what changed at the Keyspace level. It then drops keyspaces that were deleted and generate new keyspaces for those that were updated (and for new keyspaces). Finally, they loop over the new keyspaces calling updateKeyspace() for each one of them.
Next in the stack we see an interesting function: mergeSchemaAndAnnounceVersion(). This one will update the version once the keyspaces were updated in memory and on disk. The version of the schema includes that cfID that is not compatible and thus generates the exception. The Announce part is to send a gossip message to the other nodes about the fact that this node now knows of the new version of a certain schema.
Next we see something called MigrationTask. This is the message used to migrate changes between Cassandra nodes. The message payload is a collection of mutations (those handled by the mergeSchema() function.)
The rest of the stack just shows run() functions that are various types of functions used to handle messages.
In my case, for me the problem gets resolved a little later and all is well. I have nothing to do for the schema to finally get in sync. as expected. However, it prevents me from creating all my tables in one go. So, my take looking at this is that the migration messages do not arrive in the expected order. There must be a timeout which is handled by resending the event and that generates the mix-up.
So, lets look at the code sending the message in the first place, you see that one in the MigrationManager. Here we have a MIGRATION_DELAY_IN_MS parameter in link with an old issue, Schema push/pull race, which was to avoid a race condition. Well... there you go. So they are aware that there is a possible race condition and to try to avoid it, they added a little delay there. One part of that fix includes a version check. If the versions are already equal, avoid the update altogether (i.e. ignore that gossip).
if (Schema.instance.getVersion().equals(currentVersion))
{
logger.debug("not submitting migration task for {} because our versions match", endpoint);
return;
}
The delay we are talking about is one minute:
public static final int MIGRATION_DELAY_IN_MS = 60000;
One would think that one whole minute would suffice, but somehow I still get the error all the time.
The fact is that their code does not expect multiple changes happening one after the other including large delays like I have. So if I were to create one table, and then do other things, I'd be just fine. On the other hand, when I want to create 20 tables in a row on those slow machines, the gossiping message from a previous schema change arrives late (i.e. after the new CREATE TABLE command arrived to that node.) That's when I get that error. The worst part, I guess, is that it is a spurious error (i.e. it is telling me that the gossip was later, and not that my schema is invalid and the schema in the gossip message is an old one.)
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 122a2d20-9e13-11e6-b830-55bace508971; expected 1213bef0-9e
at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1350) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1306) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1256) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) [apache-cassandra-3.9.jar:3.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
I had two different table schemas with the same table name by mistake. so this issue happened (i was using express-cassandra)
I'm evaluating spark-cassandra-connector and i'm struggling trying to get a range query on partition key to work.
According to the connector's documentation it seems that's possible to make server-side filtering on partition key using equality or IN operator, but unfortunately, my partition key is a timestamp, so I can not use it.
So I tried using Spark SQL with the following query ('timestamp' is the partition key):
select * from datastore.data where timestamp >= '2013-01-01T00:00:00.000Z' and timestamp < '2013-12-31T00:00:00.000Z'
Although the job spawns 200 tasks, the query is not returning any data.
Also I can assure that there is data to be returned since running the query on cqlsh (doing the appropriate conversion using 'token' function) DOES return data.
I'm using spark 1.1.0 with standalone mode. Cassandra is 2.1.2 and connector version is 'b1.1' branch. Cassandra driver is DataStax 'master' branch.
Cassandra cluster is overlaid on spark cluster with 3 servers with replication factor of 1.
Here is the job's full log
Any clue anyone?
Update: When trying to do server-side filtering based on the partition key (using CassandraRDD.where method) I get the following exception:
Exception in thread "main" java.lang.UnsupportedOperationException: Range predicates on partition key columns (here: timestamp) are not supported in where. Use filter instead.
But unfortunately I don't know what "filter" is...
i think the CassandraRDD error is telling that the query that you are trying to do is not allowed in Cassandra and you have to load all the table in a CassandraRDD and then make a spark filter operation over this CassandraRDD.
So your code (in scala) should something like this:
val cassRDD= sc.cassandraTable("keyspace name", "table name").filter(row=> row.getDate("timestamp")>=DateFormat('2013-01-01T00:00:00.000Z')&&row.getDate("timestamp") < DateFormat('2013-12-31T00:00:00.000Z'))
If you are interested in making this type of queries you might have to take a look to others Cassandra connectors, like the one developed by Stratio
You have several options to get the solution you are looking for.
The most powerful one would be to use Lucene indexes integrated with Cassandra by Stratio, which allows you to search by any indexed field in the server side. Your writing time will be increased but, on the other hand, you will be able to query any time range. You can find further information about Lucene indexes in Cassandra here. This extended version of Cassandra is fully integrated into the deep-spark project so you can take all the advantages of the Lucene indexes in Cassandra through it. I would recommend you to use Lucene indexes when you are executing a restricted query that retrieves a small-medium result set, if you are going to retrieve a big piece of your data set, you should use the third option underneath.
Another approach, depending on how your application works, might be to truncate your timestamp field so you can look for it using an IN operator. The problem is, as far as I know, you can't use the spark-cassandra-connector for that, you should use the direct Cassandra driver which is not integrated with Spark, or you can have a look at the deep-spark project where a new feature allowing this is about to be released very soon. Your query would look something like this:
select * from datastore.data where timestamp IN ('2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04', ... , '2013-12-31')
, but, as I said before, I don't know if it fits to your needs since you might not be able to truncate your data and group it by date/time.
The last option you have, but the less efficient, is to bring the full data set to your spark cluster and apply a filter on the RDD.
Disclaimer: I work for Stratio :-) Don't hesitate on contacting us if you need any help.
I hope it helps!
I haver cognos on windows server connecting to posgres via postgreSQL ODBC.
I created a report in RS. Whenever I try adding a numeric field to the report, i get this error:
RQP-DEF-0177 An error occurred while performing operation 'sqlPrepareWithOptions' status='-9'.
UDA-SQL-0107 A general exception has occurred during the operation "prepare".
No query has been executed with that handle
If I change the field's Aggregate Function to 'None', everything works fine.
Any ideas, anyone?
Do all measures fail aggregation? What specific data types are there? Are they correctly seen from within Framework Manager as these data types?
I would generate the SQL from the report without the aggregation, edit it to have aggregation, and run it direct against the database to rule out any data overflow similar issue. [You will not be able to generate the SQL with the aggregation in place].
If not that, my next guess would be the driver itself. What version of Cognos? What OS? 32/64 bit for each? What version of postgre?