I am trying to write a UDF in Cassandra using Java.
I have two list of integers with one to one mapping i.e., first item in the first list corresponds to the first item in the second list. This is how my data is stored in Cassandra table under two columns of list type.
My UDF
CREATE OR REPLACE FUNCTION mykeyspace.get_thread(list1 List<int>, list2 List<int>)
CALLED ON NULL INPUT
RETURNS List<int>
LANGUAGE java
AS '
Map<Integer, Integer> intermediatemap = new HashMap<Integer, Integer>();
for (int i = 0; i < list1.size(); i++) {
if (!intermediatemap.containsKey(i))
intermediatemap.put(list1.get(i), list2.get(i));
}
List<Integer> commentids = new ArrayList<Integer>();
intermediatemap.entrySet().stream()
.sorted(Map.Entry.<Integer, Integer>comparingByValue().reversed())
.forEachOrdered(x -> commentids.add(x.getKey()));
if (commentids.size() > 8)
return commentids.subList(0, 8);
else
return commentids;
';
This is running all good as code. However, when I execute this on cqlsh I am getting error saying
InvalidRequest: Error from server: code=2200 [Invalid query] message="Java source compilation failed:
Line 8: The type java.util.stream.Stream cannot be resolved. It is indirectly referenced from required .class files
Line 8: The method stream() from the type Collection<Map.Entry<Integer,Integer>> refers to the missing type Stream
Is there any problem with the Cassandra version I am using? I am running the Cassandra on my local MAC machine inside a docker. I have tried with 3.11.2 and the image with latest tag.
Also, UDF function is enabled I am able to run simple UDF functions.
You can use only classes that are explicitly defined in the allowed classes list inside Cassandra, and not in the disallowed list defined. If you look to the source code for Cassandra 3.11.2, you can see that java.utils.stream is in the disallowed list, so you can't use it inside UDF/UDA.
Related
We are trying to execute dml which deletes records based on ZonedDateTime. We are using following code but running into an error.
dsl.execute ("delete from fieldhistory where createddate <= ? and object = ?", beforeDate.toOffsetDateTime(), objName)
Where beforeDate is ZonedDateTime and objectName is string
We are getting following error from postgres.
org.jooq.exception.DataAccessException: SQL [delete from fieldhistory where createddate <= ? and object = ?]; ERROR: operator does not exist: timestamp with time zone <= character varying
Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
Position: 56
at org.jooq_3.13.1.POSTGRES.debug(Unknown Source)
at org.jooq.impl.Tools.translate(Tools.java:2751)
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:755)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:385)
at org.jooq.impl.DefaultDSLContext.execute(DefaultDSLContext.java:1144)
Questions is, how do we bind datetime value in Jooq?
For historic reasons, jOOQ binds all JSR-310 times as strings, not as the relevant object type. This is because until recently, JDBC drivers did not support the JSR-310 types natively, and as such, using a string was not a bad default.
Unfortunately, this leads to type ambiguities, which you would not have if either:
jOOQ didn't bind a string
you were using the code generator and thus type safe DSL API methods
As a workaround, you can do a few things, including:
Casting your bind variable explicitly
dsl.execute("delete from fieldhistory where createddate <= ?::timestamptz and object = ?",
beforeDate.toOffsetDateTime(),
objName)
Using the DSL API
dsl.deleteFrom(FIELDHISTORY)
.where(FIELDHISTORY.CREATEDDATE.lt(beforeDate.toOffsetDateTime()))
.and(FIELDHISTORY.OBJECT.eq(objName))
.execute();
By writing your own binding
You can write your own data type binding and attach that to generated code, or to your plain SQL query, in case of which you would be in control of how the bind variable is sent to the JDBC driver. See:
https://www.jooq.org/doc/latest/manual/sql-building/queryparts/custom-bindings/
For example:
DataType<OffsetDateTime> myType = SQLDataType.OFFSETDATETIME
.asConvertedDataType(new MyBinding());
dsl.execute ("delete from fieldhistory where createddate <= {0} and object = {1}",
val(beforeDate.toOffsetDateTime(), myType),
val(objName))
There will be a fix in the future for this, so this won't be necessary anymore: https://github.com/jOOQ/jOOQ/issues/9902
We have an HDInsight cluster running HBase (Ambari)
We have created a table using Pheonix
CREATE TABLE IF NOT EXISTS Results (Col1 VARCHAR(255) NOT NULL,Col1
INTEGER NOT NULL ,Col3 INTEGER NOT NULL,Destination VARCHAR(255)
NOT NULL CONSTRAINT pk PRIMARY KEY (Col1,Col2,Col3) )
IMMUTABLE_ROWS=true
We have filled some data into this table (using some java code)
Later, we decided we want to create a local index on the destination column as follows
CREATE LOCAL INDEX DESTINATION_IDX ON RESULTS (destination) ASYNC
We have run the index tool to fill the index as follows
hbase org.apache.phoenix.mapreduce.index.IndexTool --data-table
RESULTS --index-table DESTINATION_IDX --output-path
DESTINATION_IDX_HFILES
When we run queries and filter using the destination columns everything is ok. For example
select /*+ NO_CACHE, SKIP_SCAN */ COL1,COL2,COL3,DESTINATION from
Results where COL1='data' DESTINATION='some value' ;
But, if we do not use the DESTINATION in the where query, then we will get a NullPointerException in BaseResultIterators.class
(from phoenix-core-4.7.0-HBase-1.1.jar)
This exception is thrown only when we use the new local index. If we query ignoring the index like this
select /*+ NO_CACHE, SKIP_SCAN ,NO_INDEX */ COL1,COL2,COL3,DESTINATION from
Results where COL1='data' DESTINATION='some value' ;
we will not get the exception
Showing some relevant code from the area where we get the exception
...
catch (StaleRegionBoundaryCacheException e2) {
// Catch only to try to recover from region boundary cache being out of date
if (!clearedCache) { // Clear cache once so that we rejigger job based on new boundaries
services.clearTableRegionCache(physicalTableName);
context.getOverallQueryMetrics().cacheRefreshedDueToSplits();
}
// Resubmit just this portion of work again
Scan oldScan = scanPair.getFirst();
byte[] startKey = oldScan.getAttribute(SCAN_ACTUAL_START_ROW);
byte[] endKey = oldScan.getStopRow();
====================Note the isLocalIndex is true ==================
if (isLocalIndex) {
endKey = oldScan.getAttribute(EXPECTED_UPPER_REGION_KEY);
//endKey is null for some reason in this point and the next function
//will fail inside it with NPE
}
List<List<Scan>> newNestedScans = this.getParallelScans(startKey, endKey);
We must use this version of the Jar since we run inside Azure HDInsight
and we can not select a newer jar version
Any ideas how to solve this?
What does "recover from region boundary cache being out of date" mean? it seems to be related to the problem
It appears that the version that azure HDInsight has for phoenix core (phoenix-core-4.7.0.2.6.5.3004-13.jar) has the bug but if i use a bit newer version (phoenix-core-4.7.0.2.6.5.8-2.jar, from http://nexus-private.hortonworks.com:8081/nexus/content/repositories/hwxreleases/org/apache/phoenix/phoenix-core/4.7.0.2.6.5.8-2/) we do not see the bug any more
note that it is not possible to take a much newer version like 4.8.0 since in this case the server will throw a version mismatch error
I updated these lines of code to support for spring-data-cassandra-2.0.7.RELEASE:
CassandraOperations cOps = new CassandraTemplate(session);
From:
Insert insertStatement = (Insert)statement;
CqlTemplate.addWriteOptions(insertStatement, queryWriteOptions);
cOps.execute(insertStatement);
To:
Insert insertStatement = (Insert)statement;
insertStatement = QueryOptionsUtil.addWriteOptions(insertStatement,
queryWriteOptions);
cOps.insert(insertStatement);
Above changes are throwing below error:
Caused by: org.springframework.dao.InvalidDataAccessApiUsageException: Unknown type [interface com.datastax.driver.core.policies.RetryPolicy] for property [retryPolicy] in entity [com.datastax.driver.core.querybuilder.Insert]; only primitive types and Collections or Maps of primitive types are allowed
at org.springframework.data.cassandra.core.mapping.BasicCassandraPersistentProperty.getDataType(BasicCassandraPersistentProperty.java:170)
at org.springframework.data.cassandra.core.mapping.CassandraMappingContext.lambda$null$10(CassandraMappingContext.java:552)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.data.cassandra.core.mapping.CassandraMappingContext.lambda$getDataTypeWithUserTypeFactory$11(CassandraMappingContext.java:542)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.data.cassandra.core.mapping.CassandraMappingContext.getDataTypeWithUserTypeFactory(CassandraMappingContext.java:527)
at org.springframework.data.cassandra.core.mapping.CassandraMappingContext.getDataType(CassandraMappingContext.java:486)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.getPropertyTargetType(MappingCassandraConverter.java:689)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.lambda$getTargetType$0(MappingCassandraConverter.java:682)
at java.util.Optional.orElseGet(Optional.java:267)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.getTargetType(MappingCassandraConverter.java:670)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.getWriteValue(MappingCassandraConverter.java:711)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.writeInsertFromWrapper(MappingCassandraConverter.java:403)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.writeInsertFromObject(MappingCassandraConverter.java:360)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.write(MappingCassandraConverter.java:345)
at org.springframework.data.cassandra.core.convert.MappingCassandraConverter.write(MappingCassandraConverter.java:320)
at org.springframework.data.cassandra.core.QueryUtils.createInsertQuery(QueryUtils.java:78)
at org.springframework.data.cassandra.core.CassandraTemplate.insert(CassandraTemplate.java:442)
at org.springframework.data.cassandra.core.CassandraTemplate.insert(CassandraTemplate.java:430)
Query that is passed as input is of type com.datastax.driver.core.querybuilder.Insert containing:
INSERT INTO person (name,id,age) VALUES ('name01','123',23) USING TIMESTAMP 1528922717378000 AND TTL 60;
And the queryoptions containing RetryPolicy and consistency level is passed.
Based on documentation followed above changes are not working. Can anyone let me know what is wrong here?
I'm using Spring 2.0.7.RELEASE with Cassandra driver 3.5.0
I was able to work with it using below changes:
cOps.getCqlOperations().execute(insertStatement);
How can i check the consistency level if it got applied?
For me, this works:
batchOps.insert(ImmutableSet.of(entity), insertOptions);
I am not able to return map from User Defined Function in CQL 3.x (Cassandra 2.2.3). Here is my code:
CREATE OR REPLACE FUNCTION issaviktempkeyspace.myfilter1(begindate timestamp,argtimestamp timestamp,columnname text,columnvalue text,companyid int)
CALLED ON NULL INPUT
RETURNS map<text,text> LANGUAGE java AS
'
if(begindate.before(argtimestamp)){
Map<String,String> map= Collections.<String,String>emptyMap();
map.put("columnname",columnname);
map.put("columnvalue",columnvalue);
map.put("companyid",Integer.toString(companyid));
return map;
}
else
return null;
';
When I try to create the function, Cassandra says that,
InvalidQueryException: Could not compile function 'issaviktempkeyspace.myfilter1' from Java source: org.apache.cassandra.exceptions.InvalidRequestException: Java source compilation failed:
Line 3: Map cannot be resolved to a type
Line 3: Collections cannot be resolved
com.datastax.driver.core.exceptions.InvalidQueryException: InvalidQueryException: Could not compile function 'issaviktempkeyspace.myfilter1' from Java source: org.apache.cassandra.exceptions.InvalidRequestException: Java source compilation failed:
Line 3: Map cannot be resolved to a type
Line 3: Collections cannot be resolved
-- Any idea how to do this.
I have one more query. Can I have tuple type to be returned as a part of UDF ?
--Update: it seems it was a glitch with the Virtual Machine. I restarted Cassandra service and it works as expected.
--Update: It seems that the problem is not in the code, I tried to execute the insert statement in a Cassandra client and I get the same behavior, No error is displayed, nothing is inserted.
The column that causes this behavior is of type timestamp. When I set this column value to some values (ex. '2015-08-25 22:15:12')
The table is:
create table player
(msisdn varchar primary key,
game int,keyword varchar, inserted timestamp,lang int,mo int,mt int,qid int,score int)
I am new to Cassandra, downloaded the VirtualBox snapshot to test it.
I am trying the example code of the batch and it did nothing, so I tried as people suggested to execute the prepared statement directly.
var addPlayer = casDemoSession.Prepare("INSERT INTO player (msisdn,qid,keyword,mo,mt,game,score,lang,inserted) Values (?,?,?,?,?,?,?,?,?)");
for (int i = 0; i < 20; i++) {
var bs = addPlayer.Bind(getRandMSISDN(), 1, "", 1, 0, 0, 10, 0, DateTime.Now);
bs.EnableTracing(true);
casDemoSession.Execute(bs);
}
The code above does not throw any exceptions nor insert any data. I tried to trace the query but it does not show the actual cql query.
PlannetCassandra V0.1 VM running Cassandra 2.0.1
datastax driver 2.6 https://github.com/datastax/csharp-driver
One thing that might be missing is the keyspace name for your player table. Usually you would have "INSERT INTO <keyspace>.name (..."
If you're able to run cqlsh, could you add the output from "DESCRIBE TABLE <keyspace>.player" to your question, and show what happens when you attempt to do the insert in cqlsh.