I'm trying to add a new measure to an existing ActivePivot cube.
I started adding a measure to the Sanbox project. It worked fine.
I was able to see the new measure and i got the sum of all element.
I added following lines:
EquityDerivativesCube.xml:
<measure name="test" aggregationFunctions="SUM"/>
SandboxFields.xml:
<field name="test" type="double" defaultValue="0" />
PNLCalculator#PNLCalculatorResult.java
private Double test;
public Double getTest() {
return test;
}
public void setTest(Double test) {
this.test = test;
}
PNLCalculator.enrichTrade()
...
result.setTest(2.);
...
In the Real-World-Application this approach doesn't work so good.
I added the following lines:
Into the
RealWorldApplicationCube.xml
<measure name="Test" aggregationFunctions="SUM" folder="Dev\Test" />
Into
RealWorldApplicationSchema.xml
<field name="Test" type="double" defaultValue="0" />
The data is loaded from CSV files. There is a CSV file that defines all fields for each file that the Applcation can handle.
...;Test=N/A;
There is also a calculator that handles all the other measures. I extended it like this:
public void computeTest(IRelationalEntry entry) {
double price = org.apache.commons.collections.MapUtils.getDouble(entry, "price", 0.);
double test = price * 2;
entry.put("Test", test);
}
When i run the application i get couple exceptions and i cannot see the new measure.
Here is the exception:
com.quartetfs.fwk.transaction.TransactionException: [Transaction manager: ActivePivot] Prepare commit operation failed because an exception has been raised. You must now rollback the transaction. at com.quartetfs.fwk.transaction.impl.ATransactionManager.prepareCommit(ATransactionManager.java:130) at com.quartetfs.fwk.transaction.impl.ATransactionManager.commit(ATransactionManager.java:142) at com.quartetfs.tech.store.impl.ARelationalTransactionManager.commit(ARelationalTransactionManager.java:139) at com.quartetfs.tech.store.impl.ARelationalTransactionManager.commit(ARelationalTransactionManager.java:51) at com.real.world.application.impl.ATransactionExceptionAwareHandler.doSubmit(ATransactionExceptionAwareHandler.java:97) at com.real.world.application.impl.ATransactionExceptionAwareHandler.submit(ATransactionExceptionAwareHandler.java:51) at com.quartetfs.tech.store.impl.TransactionHandlerListener.receive(TransactionHandlerListener.java:61) at com.quartetfs.tech.store.csv.impl.FilteredSource.receive(FilteredSource.java:56) at com.quartetfs.fwk.messaging.impl.ParserContext.publishChunk(ParserContext.java:457) at com.quartetfs.fwk.messaging.impl.ParserContext.awaitTermination(ParserContext.java:382) at com.quartetfs.fwk.messaging.impl.CSVSource.process(CSVSource.java:308) at com.quartetfs.fwk.messaging.impl.CSVSource.onFileAction(CSVSource.java:282) at com.quartetfs.fwk.messaging.impl.AFileWatcher.filesAction(AFileWatcher.java:277) at com.quartetfs.fwk.messaging.impl.AFileWatcher.doInterval(AFileWatcher.java:267) at com.quartetfs.fwk.messaging.impl.AFileWatcher.startScheduling(AFileWatcher.java:125) at com.quartetfs.fwk.messaging.impl.AFileWatcher.start(AFileWatcher.java:107) at com.quartetfs.fwk.messaging.impl.CSVSource.start(CSVSource.java:182) at com.quartetfs.tech.store.csv.impl.ExtendedCSVDataModelFactory$1.run(ExtendedCSVDataModelFactory.java:217) at java.lang.Thread.run(Thread.java:662) Caused by: com.quartetfs.fwk.transaction.TransactionException: [Transaction manager: RealWorldApplicationSchema] Prepare commit operation failed because an exception has been raised. You must now rollback the transaction. at com.quartetfs.fwk.transaction.impl.ATransactionManager.prepareCommit(ATransactionManager.java:130) at com.quartetfs.tech.store.impl.ARelationalTransactionManager.doPrepareCommit(ARelationalTransactionManager.java:113) at com.quartetfs.fwk.transaction.impl.ATransactionManager.prepareCommit(ATransactionManager.java:128) ... 18 more Caused by: com.quartetfs.fwk.transaction.TransactionException: One of the schema transaction contribution tasks failed. at com.quartetfs.biz.pivot.transaction.impl.ActivePivotSchemaTransaction.prepareCommit(ActivePivotSchemaTransaction.java:239) at com.quartetfs.biz.pivot.impl.ActivePivotSchemaTransactionManager.doPrepareCommit(ActivePivotSchemaTransactionManager.java:194) at com.quartetfs.fwk.transaction.impl.ATransactionManager.prepareCommit(ATransactionManager.java:128) ... 20 more Caused by: java.lang.RuntimeException: com.quartetfs.biz.pivot.ClassificationException: The calculator has thrown an exception during the evaluation of the object: RelationalEntry [type=ActivePivot, key=Key [ ... ,Test=null, ...]] at jsr166y.ForkJoinTask.completeExceptionally(ForkJoinTask.java:1116) at jsr166y.cancellable.impl.CancellableCountedCompleter.onCompletion(CancellableCountedCompleter.java:132) at jsr166y.CountedCompleter.tryComplete(CountedCompleter.java:391) at com.quartetfs.biz.pivot.transaction.impl.ActivePivotSchemaTransaction$ContributeAction.afterCompute(ActivePivotSchemaTransaction.java:492) at jsr166y.cancellable.impl.CancellableCountedCompleter.compute(CancellableCountedCompleter.java:96) at jsr166y.CountedCompleter.exec(CountedCompleter.java:437) at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:265) at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:976) at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1480) at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:105) Caused by: com.quartetfs.biz.pivot.ClassificationException: The calculator has thrown an exception during the evaluation of the object: RelationalEntry [type=ActivePivot, key=Key [ ... ,Test=null, ...]] at com.quartetfs.biz.pivot.classification.impl.Classifier.classifyAll(Classifier.java:114) at com.quartetfs.biz.pivot.transaction.impl.ActivePivotSchemaTransaction$ContributeAction.compute(ActivePivotSchemaTransaction.java:446) at com.quartetfs.biz.pivot.transaction.impl.ActivePivotSchemaTransaction$ContributeAction.computeSafely(ActivePivotSchemaTransaction.java:487) at jsr166y.cancellable.impl.CancellableCountedCompleter.compute(CancellableCountedCompleter.java:91) ... 5 more Caused by: com.quartetfs.biz.pivot.ClassificationException: Classification failure for level "Test": the mandatory property "Test" is not found. Contribution key: "Key [ ... ,Test=null, ...]]. at com.quartetfs.biz.pivot.classification.impl.ResultClassificationProcedure.execute(ResultClassificationProcedure.java:192) at com.quartetfs.biz.pivot.classification.impl.ClassificationTree$ClassificationNode.execute(ClassificationTree.java:215) at com.quartetfs.biz.pivot.classification.impl.ClassificationTree$ClassificationNode.execute(ClassificationTree.java:235) at com.quartetfs.biz.pivot.classification.impl.ClassificationTree.execute(ClassificationTree.java:57) at com.quartetfs.biz.pivot.classification.impl.Classifier.classify(Classifier.java:161) at com.quartetfs.biz.pivot.classification.impl.Classifier.classifyAll(Classifier.java:110) ... 8 more
06.02.2013 16:09:24 com.real.world.application.impl.ATransactionExceptionAwareHandler doSubmit
What am i missing? How can i add a new measure to the cube?
What causes the error in the calculator?
While debugging the code is visiting the calculators method (computeTest()) and is setting the value to the RelationalEntry.
Cheers
The steps to add a new measure in ActivePivot are not more complicated than what you describe in your post. Declare the field that holds the value to aggregate, reference that field as a measure in your cube, together with the aggregation function(s) you want to apply on it. And optionally the post processors you have written if the base aggregation functions are not enough.
But something else in your project seems to cause an issue and you need support (the data, the loading of the data, another component...). StackOverflow is not meant for issue tracking or troubleshooting support, it is more about "I would like to do that, how can I do it?". My advice is to bring your issue directly to the Quartet FS customer support ( http://support.quartetfs.com )
Related
Why isn't there an equivalent updateSQL() (as in JdbcPollingChannelAdapter) for JpaInboundChannelAdapterSpec? Please confirm if manual persist through handle() should be called to update polled records as NOT to pick the same again?
UPDATE
#Bean
public JpaInboundChannelAdapterSpec dBJpaInboundSpec() {
return Jpa.inboundAdapter(entityManagerFactory)
.entityClass(XYZ.class)
.jpaQuery("SELECT a FROM XYZ a WHERE a.status is null")
// trick to update the polled record
.deleteAfterPoll(true)
.maxResults(maxMessagesPerPoll);
}
#Bean
public PollerSpec dBPollerSpec() {
return Pollers.fixedDelay(Duration.ofMinutes(pollInterval))
.maxMessagesPerPoll(maxMessagesPerPoll)
.transactional()
.advice(loggingAdvice);
}
2022-10-21 20:05:57,559 [scheduling-2 ] ERROR o.s.i.h.LoggingHandler - b46f904d98859f6f org.springframework.messaging.MessagingException: nested exception is org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not execute statement, failedMessage=GenericMessage [payload=[REMOVED_FOR_OBVIOUS_REASONS], headers={REMOVED_FOR_OBVIOUS_REASONS}]
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:427)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$4(AbstractPollingEndpoint.java:348)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$5(AbstractPollingEndpoint.java:341)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.con
I think it is not implement like you'd expect just because JPA is not just about DB, but persistent entity state in the memory. So, if you just did an update on the entity, then downstream logic is going to deal with that updated data - not an original you have polled from persistent layer before.
Yes, it is better to do such an update already after processing in this case manually.
I also always recommend a trick with an org.hibernate.annotations.SQLDelete where you indeed can specify an UPDATE query. That JpaInboundChannelAdapterSpec has this option then:
/**
* If set to 'true', the retrieved objects are deleted from the database upon
* being polled. May not work in all situations, e.g. for Native SQL Queries.
* #param deleteAfterPoll Defaults to 'false'.
* #return the spec
*/
public JpaInboundChannelAdapterSpec deleteAfterPoll(boolean deleteAfterPoll) {
I am following Doc to try out how to enrich an unbounded stream by directly looking up from a IMap. I have two Maps:
Product: Map<String, Product> (ProductId as key)
Seller: Map<String, Seller> (SellerId as key)
Both Product and Seller are very simple classes:
public class Product implements DataSerializable {
String productId;
String sellerId;
int price;
...
public class Seller implements DataSerializable {
String sellerId;
int revenue;
...
I have two data generators keep pushing data to the two maps. The event-journal are enabled for both maps. I have verified the event-journal works fine.
I want to enrich the stream event of Product map with Seller map. Here is a snippet of my code:
IMap<String, Seller> sellerIMap = jetClient.getMap(SellerDataGenerator.SELLER_MAP);
StreamSource<Product> productStreamSource = Sources.mapJournal(ProductDataGenerator.PRODUCT_MAP, Util.mapPutEvents(), Util.mapEventNewValue(), START_FROM_CURRENT);
p.drawFrom(productStreamSource)
.withoutTimestamps()
.groupingKey(Product::getSellerId)
.mapUsingIMap(sellerIMap, (product, seller) -> new EnrichedProduct(product, seller))
.drainTo(getSink());
try {
JobConfig jobConfig = new JobConfig();
jobConfig.addClass(TaskSubmitter.class).addClass(Seller.class).addClass(Product.class).addClass(ExtendedProduct.class);
jobConfig.setName(Constants.BASIC_TASK);
Job job = jetClient.newJob(p, jobConfig);
} finally {
jetClient.shutdown();
}
When job was submitted, I got following error:
com.hazelcast.spi.impl.operationservice.impl.Invocation - [172.31.33.212]:80 [jet] [3.1] Failed asynchronous execution of execution callback: com.hazelcast.util.executor.DelegatingFuture$DelegatingExecutionCallback#77ac0407for call Invocation{op=com.hazelcast.map.impl.operation.GetOperation{serviceName='hz:impl:mapService', identityHash=1939050026, partitionId=70, replicaIndex=0, callId=-37944, invocationTime=1570410704479 (2019-10-07 01:11:44.479), waitTimeout=-1, callTimeout=60000, name=sellerMap}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1570410704479, firstInvocationTime='2019-10-07 01:11:44.479', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[172.31.33.212]:80, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}
I tried to put one and two instances in my cluster and got the same error message. I couldn't figure out what was the root cause.
It seems that your problem is a ClassNotFoundException, even though you added the appropriate classes to the job. The objects you store in the IMap exist independent of your Jet job and when the event journal source asks for them, Jet's IMap code tries to deserialize them and fails because Jet doesn't have your domain model classes on its classpath.
To move on, add a JAR with the classes you use in the IMap to Jet's classpath. We are looking for a solution that will remove this requirement.
The reason you haven't got the exception stacktrace in the log output is due to the default java.util.logging setup you end up with when you don't explicitly add a more flexible logging module, such as Log4j.
The next version of Jet's packaging will improve this aspect. Until that time you can follow these steps:
Go to the lib directory of Jet's distribution package and download Log4j into it:
$ cd lib
$ wget https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar
Edit bin/common.sh to add the module to the classpath. Towards the end of the file there is a line
CLASSPATH="$JET_HOME/lib/hazelcast-jet-3.1.jar:$CLASSPATH"
You can duplicate this line and replace hazelcast-jet-3.1 with log4j-1.2.17.
At the end of commons.sh there is a multi-line command that constructs the JAVA_OPTS variable. Add "-Dhazelcast.logging.type=log4j" and "-Dlog4j.configuration=file:$JET_HOME/config/log4j.properties" to the list.
Create a file log4j.properties in the config directory:
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %5p [%c{1}] [%t] - %m%n
# Change this level to debug to diagnose failed cluster formation:
log4j.logger.com.hazelcast.internal.cluster=info
log4j.logger.com.hazelcast.jet=info
log4j.rootLogger=info, stdout
Background:
We are using Spring.Net 2.0 to manage our transactions with nHibernate 3.3.1. The application follows the the publisher-subscriber pattern. Each subscriber runs in a separate thread and is invoked by RabbitMQ message broker system once the queue, it is bound to, got a message published from some other thread.
Each time the subscriber begins processing it starts a new transaction (using Spring.Net declarative transactions support) in a separate thread. All the subsequent operations invoked by subscriber usually use the same transaction. But there are cases where we need a new transaction e.g. when we need to update the shared data, like oracle sequence. The methods doing such operations are decorated with Transaction aspect and with TransactionPropagation value set as RequiresNew.
Based on spring.Net documentation and source code we assume that for such special scenarios where a method is invoked with TransactionPropagation value as RequiresNew, the outer transaction is suspended until the method is completed. The outer transaction resumes after the completion of the method and follows the execution path of the thread.
Problem:
We are encountering a random scenario where once a subscriber is handed over the message for processing, some times it completes the task with out any problem but some times it throws following exception:
Exception message:
Cannot activate transaction synchronization - already active
Stacktrace: at
Spring.Transaction.Support.TransactionSynchronizationManager.InitSynchronization()
at
Spring.Transaction.Support.AbstractPlatformTransactionManager.DoResumeSynchronization(IList
suspendedSynchronizations) at
Spring.Transaction.Support.AbstractPlatformTransactionManager.Resume(Object
transaction, Object suspendedResources) at
Spring.Transaction.Support.AbstractPlatformTransactionManager.HandleExistingTransaction(ITransactionDefinition
definition, Object transaction, Boolean debugEnabled) at
Spring.Transaction.Support.AbstractPlatformTransactionManager.GetTransaction(ITransactionDefinition
definition) at
Spring.Transaction.Interceptor.TransactionAspectSupport.CreateTransactionIfNecessary(ITransactionAttribute
sourceAttr, String joinpointIdentification) at
Spring.Transaction.Interceptor.TransactionAspectSupport.CreateTransactionIfNecessary(MethodInfo
method, Type targetType) at
Spring.Transaction.Interceptor.TransactionInterceptor.Invoke(IMethodInvocation
invocation) at
Spring.Aop.Framework.AbstractMethodInvocation.Proceed() in
e:\spring.net\spring-net-master\src\Spring\Spring.Aop\Aop\Framework\AbstractMethodInvocation.cs:line
291 at Spring.Aop.Framework.DynamicProxy.AdvisedProxy.Invoke(Object
proxy, Object target, Type targetType, MethodInfo targetMethod,
MethodInfo proxyMethod, Object[] args, IList interceptors) in
e:\spring.net\spring-net-master\src\Spring\Spring.Aop\Aop\Framework\DynamicProxy\AdvisedProxy.cs:line
210
The frequency of this issue increases with number of messages in the message broker queues.
Sample Code/Configuration:
The spring transaction is configured in our spring context file as:
<object id="transactionManager" type="Spring.Data.NHibernate.HibernateTransactionManager, Spring.Data.NHibernate3">
<property name="DbProvider" ref="DbProvider" />
<property name="SessionFactory" ref="NHibernateSessionFactory" />
<property name="TransactionSynchronization" value="Always" />
</object>
The code is written like:
public class MyMessageHandler
{
ServiceOne MyServiceOne { get; set;}
ServiceTwo MyServiceTwo { get; set;}
[Transaction]
public void Execute()
{
var unprocessedObjects = MyServiceOne.FindUnprocessedObjects();
foreach (var unprocessedObject in unprocessedObjects)
{
MyServiceTwo.UpdateObject(unprocessedObject); //----> This is where exception is being thrown
}
}
}
public class ServiceOne
{
[Transaction]
public List<MyType> FindUnprocessedObjects()
{
return TestRepository.FindUnprocessedObjects();
}
}
public class ServiceTwo
{
[Transaction(TransactionPropagation.RequiresNew)]
public List<MyType> UpdateObject(MyType obj)
{
var myObj = this.GetCurrentSession().Merge(obj);
myObj.Processed = true;
}
}
We are stuck as we could not see a real reason whey above code is running some times without problem but throwing exception randomly. Please help.
Regards
How does the code within the 'shouldFail?' block work? I understand that it is a closure, but the code gets called regardless of whether or not I call is using its signature. Besides, what's the deal with the "ReadOnlyPropertyException" showing up in the parenthesis? If it is a parameter, then it is not setup as listed in the official documentation!!
Questions: What is 'shouldFail'? How should it be invoked? How to handle the exception purportedly thrown by this method/function/closure?
void test02_ReadOnlyFieldInGroovyBean() {
// You've probably noticed how Groovy automatically generates getters/setters for you. But what if you don't
// want to generate a setter because it's a read-only field? Just mark it with 'final'. Groovy will understand.
// Try to modify Ken's ssn. You should get a ReadOnlyPropertyException.
def person = new GroovyPerson('Ken', 'Kousen', '7878')
def failed = false
shouldFail (ReadOnlyPropertyException) {
// ------------ START EDITING HERE ----------------------
System.out.println(" i am in should fail")
person.ssn='8332';
// ------------ STOP EDITING HERE ----------------------
failed = false
System.out.println(" exiting should fail")
}
//def foobar=shouldFail("hjh");
//def foobar=true;
failed=shouldFail('abc');
//System.out.println("Failed: "+failed);
assert failed
// The code wrapping your additions verifies that the ReadOnlyProperty exception has been thrown.
// The curly brackets ({}) represent a closure. We'll get into what that means very soon.
}
shouldFail() (in this variant) takes a class and a closure. It runs the closure and reports a test failure if the closure does NOT exit by throwing an exception of that type. As to catching the exception, you don't - shouldFail() does that for you.
See: http://docs.groovy-lang.org/latest/html/gapi/groovy/test/GroovyAssert.html#shouldFail%28java.lang.Class,%20groovy.lang.Closure%29
(Reading the comments and the code around them, it looks like this unit test should pass because setting the .ssn property of GroovyPerson will fail as it is a read-only property, causing a ReadOnlyPropertyException.)
I just received an exception when I try to reference a static variable in another class, which is also statically initialized. This worked before, and for some reason it fails now. The only changes I made were resetting Visual Studio (2010) to its default setting, which I can't imagine to be the reason for this. Any other code I added didn't touch any of the affected parts either.
This is my code
WinForms class 'MainForm':
partial class MainForm : Form
{
// ...
private RefClass convMan;
private Dictionary<EnumType, string> LogNames = RefClass.LogNames;
// ...
public MainForm() { .... }
}
Referenced class 'RefClass':
class RefClass
{
// ...
public enum EnumType { TypeOne = 0, TypeTwo = 1, TypeThree = 2 };
public static Dictionary<EnumType, string> LogNames = new Dictionary<EnumType, string>()
{
{ EnumType.TypeOne, "Text0" },
{ EnumType.TypeTwo, "Text1" },
{ EnumTypy.TypeThree, "Text2" }
};
}
The error I get now is (translated from German):
An unhandled exception of type "System.TypeInitializationException" occurred.
Additional information: The type initializer for "RefClass" threw an exception.
which has the InnerException
System.ArgumentException
So, as far as I'm concerned, my static dictionary should be initialized once it gets accessed, thus when my Form class references it. I tried debugging to see if the static dictionary is initialized before it gets referenced in the Form class, which is not the case. Also, when I stop at a breakpoint for the reference line, the variable LogNames is null.
I'm really confused as to why this happens, it all worked before.
I found my error, the exceptions I got were quite misleading though. It was a problem with a different dictionary than the one I referenced. It probably didn't get initialized in the first place because something before that failed (If someone can clear this up, please feel free to do so!). Basically what I did wrong was using a two-directional dictionary and adding a value twice. This should normally produce a normal exception, but since it was done statically it got wrapped into a TypeInitializationException. I had a deeper look into the exact stacktrace of the inner exception and found where the exception originated from. Maybe this helps someone in the future...
I had a simular issue getting the same exception. Found that my static constructor for my utility class was generating the exception. Took some time locating since the description of the exception was misleading.
As #Yeehaw mentioned, it appears that the exception gets wrapped, so the common denominator here I would say is that the class/object is static.