I am trying to implement search query using dynamic query
I created dynamic query like this
DynamicQuery dynamicQuery=DynamicQueryFactoryUtil.forClass(OrderSubscription.class,PortletClassLoaderUtil.getClassLoader());
DynamicQuery dynamicQueryCount=DynamicQueryFactoryUtil.forClass(OrderSubscription.class,PortletClassLoaderUtil.getClassLoader());
Criterion criterion = RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.ORDERNO, StringPool.PERCENT+searchKeyword+StringPool.PERCENT);
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.STATUS, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.ORDERDATE, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.PAYMENTSTATUS, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.ORDERSTATUS, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.ORDERID, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.PAYMENTTRANSACTIONTYPE, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.STARTDATE, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
criterion = RestrictionsFactoryUtil.or(criterion, RestrictionsFactoryUtil.like(DBConstants.OrderSubscription.CANCELDATE, StringPool.PERCENT+searchKeyword+StringPool.PERCENT));
dynamicQuery.add(junctionOR);
dynamicQueryCount.add(junctionOR);
List<OrderSubscription> orders = OrderSubscriptionLocalServiceUtil.dynamicQuery(dynamicQuery, start, end);
Now its throwing error of casting
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
at org.hibernate.type.descriptor.java.JdbcTimestampTypeDescriptor.unwrap(JdbcTimestampTypeDescriptor.java:42)
at org.hibernate.type.descriptor.sql.TimestampTypeDescriptor$1.doBind(TimestampTypeDescriptor.java:53)
at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:91)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:283)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:278)
at org.hibernate.loader.Loader.bindPositionalParameters(Loader.java:1873)
at org.hibernate.loader.Loader.bindParameterValues(Loader.java:1844)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1716)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2542)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:119)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1716)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
at com.liferay.portal.dao.orm.hibernate.DynamicQueryImpl.list(DynamicQueryImpl.java:94)
at com.liferay.portal.dao.orm.hibernate.DynamicQueryImpl.list(DynamicQueryImpl.java:88)
at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.findWithDynamicQuery(BasePersistenceImpl.java:166)
... 148 more
Jan 08, 2016 6:39:51 AM org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet ordermanagment Servlet threw exception
summary :
i want to get list based on searched keyword from DB
and my DB column type id datetime.
Can any one help me sort out this problem
Thanks
Ravi Darji
Related
I have a data frame with 3 columns: journal, abstract, and word count. I'm trying to calculate the average word count of the abstract for each journal and then order them in descending order to show the journals with the longest average abstract.
I've managed to get to a point where I have just the journal and average word count. That was done by doing:
newDF = marchDF.select("journal", "abstract").withColumn("wordcount", lit("0").cast("integer")).withColumn("wordcount", sql.size(sql.split(sql.col("abstract"), " ")))
nonullDF = newDF.filter(col("journal").isNotNull()).filter(col("abstract").isNotNull())
groupedDF = nonullDF.select("journal", "wordcount").groupBy("journal").avg("wordcount")
however, when I try to order it by wordcount, it throws the error:
cannot resolve '`wordcount`' given input columns: [avg(wordcount), journal];;
I've tried:
orderedDF = groupedDF.orderBy(col("wordcount")).desc().show(5)
and:
orderedDF = groupedDF.sort(col("wordcount")).desc.show(5)
but both throw that same error and I don't understand why.
That's because as the error says, there is no column named wordcount. The column you want to order by is called avg(wordcount), so you can do
orderedDF = groupedDF.orderBy("avg(wordcount)", ascending=False).show(5)
Alternatively, you can rename the avg column to wordcount during the aggregation:
import pyspark.sql.functions as F
groupedDF = nonullDF.select("journal", "wordcount").groupBy("journal").agg(F.avg("wordcount").alias("wordcount"))
orderedDF = groupedDF.orderBy("wordcount", ascending=False).show(5)
Note also the correct syntax for ordering in descending order.
I'm trying to use Cross-validation on using spark but it throws an error:
gbtClassifier = GBTClassifier(featuresCol= "features", labelCol="is_goal")
lr = LogisticRegression(featuresCol= "features" ,labelCol="is_goal")
pipelineStages = stringIndexers + encoders + [featureAssembler]
pipeline = Pipeline(stages=pipelineStages)
param_grid_lr = ParamGridBuilder().addGrid(lr.regParam, [0.1,0.01]).addGrid(lr.elasticNetParam, [0,0.5,1]).build()
crossval = CrossValidator(estimator=lr, estimatorParamMaps=param_grid_lr ,evaluator=BinaryClassificationEvaluator(), numFolds=3)
cross_model = crossval.fit(df_tr)
IllegalArgumentException: label does not exist. Available: event_type_str, event_team, shot_place_str, location_str, assist_method_str, situation_str, country_code, is_goal, event_type_str_idx, event_team_idx, shot_place_str_idx, location_str_idx, assist_method_str_idx, situation_str_idx, country_code_idx, event_type_str_vec, event_team_vec, shot_place_str_vec, location_str_vec, assist_method_str_vec, situation_str_vec, country_code_vec, features, CrossValidator_2fc516202d9d_rand, rawPrediction, probability, prediction
[here is who my features look like1
Your BinaryClassificationEvaluator expects by default that the label column is called label as you can see from the docs https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.evaluation.BinaryClassificationEvaluator .
You'll need to specify rawPredictionCol and labelCol according to the columns given in your dataframe
I have a spring boot java application that talks to cassandra .
However one of my queries is failing .
public class ParameterisedListItemRepository {
private PreparedStatement findByIds;
public ParameterisedListItemRepository(Session session, Validator validator, ParameterisedListMsisdnRepository parameterisedListMsisdnRepository ) {
this.findByIds = session.prepare("SELECT * FROM mep_parameterisedListItem WHERE id IN ( :ids )");
}
public List<ParameterisedListItem> findAll(List<UUID> ids){
List<ParameterisedListItem> parameterisedListItemList = new ArrayList<>();
BoundStatement stmt =this.findByIds.bind();
stmt.setList("ids", ids);
session.execute(stmt)
.all()
.stream()
.map(parameterisedListItemMapper)
.forEach(parameterisedListItemList::add);
return parameterisedListItemList;
}
}
the following is the stack trace
java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.UUID
at com.datastax.driver.core.TypeCodec$AbstractUUIDCodec.serialize(TypeCodec.java:1626)
at com.datastax.driver.core.AbstractData.setList(AbstractData.java:358)
at com.datastax.driver.core.AbstractData.setList(AbstractData.java:374)
at com.datastax.driver.core.BoundStatement.setList(BoundStatement.java:681)
at com.openmind.primecast.repository.ParameterisedListItemRepository.findAll(ParameterisedListItemRepository.java:128)
at com.openmind.primecast.repository.ParameterisedListItemRepository$$FastClassBySpringCGLIB$$46ffc15e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.openmind.primecast.repository.ParameterisedListItemRepository$$EnhancerBySpringCGLIB$$b2db3c41.findAll(<generated>)
at com.openmind.primecast.service.impl.ParameterisedListItemServiceImpl.findByParameterisedList(ParameterisedListItemServiceImpl.java:102)
at com.openmind.primecast.web.rest.ParameterisedListItemResource.getParameterisedListItemsByParameterisedList(ParameterisedListItemResource.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Any idea what is going wrong. I know this query is the problem
SELECT * FROM mep_parameterisedListItem WHERE id IN ( :ids )
any idea how I can change the findAll function to achieve the query?
this is the table definition
CREATE TABLE "Openmind".mep_parameterisedlistitem (
id uuid PRIMARY KEY,
data text,
msisdn text,
ordernumber int,
parameterisedlist uuid
) WITH COMPACT STORAGE;
Thank you.
Without knowing the table schema, my guess is that a change was made to the table so the schema no longer match the bindings in the prepared statement.
A big part of the problem is your query with SELECT *. Our recommendation for best practice is to explicitly name all the columns you're retrieving from the table. By specifying the columns in your query, you avoid surprises when the table schema changes.
In this instance, either a new column was added or an old column was dropped. With the cached prepared statement, it was expecting one column type and got another -- the ArrayList doesn't match UUID.
The solution is to re-prepare the statement and name all the columns. Cheers!
I'm using the H2ODRF and H2OGridSearch models to create a random forest pipeline with random discrete grid search hyper-parameter optimization. However, when I set nfolds to any number greater than 1 and call fit(), I get an error. My code looks like this:
val drf = new H2ODRF()
.setFeaturesCols(featuresCols)
.setLabelCol(labelCol)
.setColumnsToCategorical(categoricalCols)
.setSplitRatio(splitRatio)
.setNfolds(4)
val nps = Map(
"ntrees" -> Array(10, 50).map(_.asInstanceOf[AnyRef]))
val search = new H2OGridSearch()
.setHyperParameters(hyperParams)
.setAlgo(drf)
val model = search.fit(data) // data is a Spark DataFrame
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 608096 path $.cross_validation_metrics_summary[0].data[0][0]
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:224)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:129)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:220)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:129)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:220)
at com.google.gson.Gson.fromJson(Gson.java:887)
at com.google.gson.Gson.fromJson(Gson.java:852)
at com.google.gson.Gson.fromJson(Gson.java:801)
at ai.h2o.sparkling.backend.utils.RestCommunication$class.ai$h2o$sparkling$backend$utils$RestCommunication$$deserialize(RestCommunication.scala:164)
at ai.h2o.sparkling.backend.utils.RestCommunication$$anonfun$request$1.apply(RestCommunication.scala:147)
at ai.h2o.sparkling.backend.utils.RestCommunication$$anonfun$request$1.apply(RestCommunication.scala:145)
at ai.h2o.sparkling.utils.ScalaUtils$.withResource(ScalaUtils.scala:28)
at ai.h2o.sparkling.backend.utils.RestCommunication$class.request(RestCommunication.scala:145)
at ai.h2o.sparkling.ml.algos.H2OGridSearch.request(H2OGridSearch.scala:46)
at ai.h2o.sparkling.backend.utils.RestCommunication$class.query(RestCommunication.scala:54)
at ai.h2o.sparkling.ml.algos.H2OGridSearch.query(H2OGridSearch.scala:46)
at ai.h2o.sparkling.ml.algos.H2OGridSearch.getGridModels(H2OGridSearch.scala:129)
at ai.h2o.sparkling.ml.algos.H2OGridSearch.fit(H2OGridSearch.scala:163)
at ai.h2o.sparkling.ml.algos.H2OGridSearch.fit(H2OGridSearch.scala:46)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:153)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:149)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableViewLike$Transformed$class.foreach(IterableViewLike.scala:44)
at scala.collection.SeqViewLike$AbstractTransformed.foreach(SeqViewLike.scala:37)
at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:149)
... 59 elided
Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 608096 path $.cross_validation_metrics_summary[0].data[0][0]
at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:385)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:213)
... 90 more
The error seems to be caused by a cross_validation_metrics_summary field which is only returned when Nfolds is greater than 1. Is there a work around to this problem?
Edit:
I'm using the Prostate Data and using Spark version 2.4.4, Scala Version 2.11.12, and using the following sparkling water version ai.h2o:sparkling-water-package_2.11:3.30.0.4-1-2.4.
Edit:
After scouring through the Sparkling Water source code, it's starting to look like the issue is with a misconfigured schema in GridSchemaV99. Is there a setting/configuration I am supposed to update that looks for a different schema?
I am using Spark 1.5.0 and I have this issue:
val df = paired_rdd.reduceByKey {
case (val1, val2) => val1 + "|" + val2
}.toDF("user_id","description")
Here is sample data for df, as you can see the column description has
this format (text1#text3#weight | text1#text3#weight|....)
user1
book1#author1#0.07841217886795074|tool1#desc1#0.27044260397331488|song1#album1#-0.052661673730870676|item1#category1#-0.005683148395350108
I want to sort this df based on weight in descending order here is what I tried:
First split the contents at "|" and then for each of those strings, split them at "#" and get the 3rd string which is weight and then convert that into a double value
val getSplitAtWeight = udf((str: String) => {
str.split("|").foreach(_.split("#")(2).toDouble)
})
Sort based on the weigh value returned by the udf (in descending manner)
val df_sorted = df.sort(getSplitAtWeight(col("description")).desc)
I get the following error:
Exception in thread "main" java.lang.UnsupportedOperationException:
Schema for type Unit is not supported at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:153)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
at
org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:64)
at
org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:29)
at org.apache.spark.sql.functions$.udf(functions.scala:2242)
Change foreach in your udf to map as following will eliminate the exception:
def getSplitAtWeight = udf((str: String) => {
str.split('|').map(_.split('#')(2).toDouble)
})
The problem with your method is that foreach method on List doesn't return anything, i.e., its result is of type Unit that's why you get the Exception. To understand more about the foreach, check this blog.