<context:annotation-config/>
<context:component-scan base-package="ru.*"/>
<int:poller id="defaultPoller" default="true"
fixed-delay="1000" task-executor="taskExecutor"/>
<task:executor id="taskExecutor" pool-size="1-5" queue-capacity="200" rejection-policy="CALLER_RUNS"/>
<int:channel id="listFiles">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="pages">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="resultWithHeader">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="toBD">
<int:queue capacity="1000"/>
</int:channel>
<int:channel id="unparsedToUniq">
<int:queue capacity="1000"/>
</int:channel>
<task:scheduled-tasks>
<task:scheduled ref="getFilesList"
method="getList"
cron="0 0 12 * * WED"/>
</task:scheduled-tasks>
<int:splitter id="splitterPdf" ref="split" method="receive" input-channel="listFiles" output-channel="pages"/>
<int:transformer id="transfMain" input-channel="pages" ref="firstTransform" method="transform"
output-channel="resultWithHeader"/>
<int:header-value-router id="routedParsingDocument" input-channel="resultWithHeader" header-name="parsed">
<int:mapping value="yes" channel="toBD"/>
<int:mapping value="no" channel="unparsedToUniq"/>
</int:header-value-router>
<int:transformer id="transfUniq" input-channel="unparsedToUniq" ref="secondTransform" method="transform"
output-channel="toBD"/>
<int:service-activator id="bd_activator" input-channel="toBD" method="receive" ref="bd" > <int:poller task-executor="taskBD" fixed-delay="1500"/></int:service-activator>
<bean id="getFilesList" class="ru.*.GetAndReadFiles"/>
<bean id="split" class="ru.*.SplitDocument"/>
<bean id="firstTransform" class="ru.*.MainParser"/>
<bean id="secondTransform" class="ru.*.UniqParser"/>
<bean id="bd" class="ru.*.BDWriter"/>
<task:executor id="taskBD" pool-size="1-10" queue-capacity="100" rejection-policy="CALLER_RUNS"/>
Hello.
Program read all files on directory, split pages, transform information(main or uniq) and write info to base.
I have problem with this config.
06-03-2019 13:01:24.773 [task-scheduler-7] ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#19c520db[Running, pool size = 5, active threads = 5, queued tasks = 200, completed tasks = 1353]] did not accept task: org.springframework.integration.util.ErrorHandlingTaskExecutor$$Lambda$241/1065676784#6991f347
What am I doing wrong? If i don't use taskExecutor, program work. But I need to increase the speed of execution. In other configuration work< but i have this problem:ERROR o.s.i.handler.LoggingHandler.handleMessageInternal - java.lang.OutOfMemoryError: Java heap space
Thanks for you help.
You should consider do not use queue channels in between. There is just enough to distribute polling results into that taskExecutor from the poller.
You can leave a queue channel after splitter though, since you are going to have one-to-many after splitting, so would be great to make them parallel. Although you may also consider to use there an ExecutorChannel instead.
Not clear, though, why you have a rejected task since your policy is CALLER_RUNS...
Related
Requirement is that the message payload received in a particular direct channel has to be converted to a json and stored in a temporary file using the spring integration.
enter code here
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns="http://www.springframework.org/schema/beans"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-4.3.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.3.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/task
http://www.springframework.org/schema/task/spring-task-4.3.xsd">
<int:channel id="requestChannel">
<int:dispatcher task-executor="preValidationWorkers"/>
</int:channel>
<task:executor id="preValidationWorkers" pool-size="6" queue-capacity="6" rejection-policy="CALLER_RUNS"/>
<int:channel id="validaitonSuccessChannel">
<int:dispatcher task-executor="validationWorkers"/>
</int:channel>
<task:executor id="validationWorkers" pool-size="3" queue-capacity="3" rejection-policy="CALLER_RUNS"/>
<int:channel id="transformationSuccessChanel">
<int:dispatcher task-executor="transformationWorkers"/>
</int:channel>
<task:executor id="transformationWorkers" pool-size="3" queue-capacity="3" rejection-policy="CALLER_RUNS"/>
<int:service-activator id="requestDistributor" input-channel="requestChannel"
ref="personValidatorService"
method="validate"
output-channel="validaitonSuccessChannel">
</int:service-activator>
<int:service-activator id="requestTransformer"
input-channel="validaitonSuccessChannel"
ref="personTransformationService"
method="transform"
output-channel="transformationSuccessChanel">
</int:service-activator>
<int:service-activator id="requestInjector"
input-channel="transformationSuccessChanel"
ref="personSqlService"
method="inject">
</int:service-activator>
<bean id="personValidatorService"
class="com.naveendc.taskexecutor.service.validation.PersonValidatorServiceImpl" lazy-init="true">
</bean>
<bean id="personTransformationService"
class="com.naveendc.taskexecutor.service.transformation.PersonTransformationServiceImpl" lazy-init="true">
</bean>
<bean id="personSqlService"
class="com.naveendc.taskexecutor.service.sql.PersonSqlServiceImpl" lazy-init="true">
</bean>
<int:gateway id="personGateway" service-interface="com.naveendc.taskexecutor.service.integration.PersonGateway"></int:gateway>
<context:annotation-config/>
<int:annotation-config/>
<context:component-scan base-package="com.naveendc.taskexecutor.service, com.naveendc.taskexecutor.service.integration"/>
</beans>
Service activators method is something like this.
#Override
public void inject(Person person) {
System.out.println("Person injected ==>" + person.toString()));
return;
}
Is there a way to convert the payload received in transformationSuccessChannel into json format and store it in a temp file using the spring integration?
First of all you have a typo in your code snippet - transformationSuccessChanel. It's hard to perform search in such a big domain specific code.
Not sure what is your goal, but that's definitely not a channel responsibility.
For the transformation purpose there is a <transformer> in Spring Integration: https://docs.spring.io/spring-integration/docs/5.0.6.RELEASE/reference/html/messaging-transformation-chapter.html
So, if your <int:service-activator id="requestInjector"> expect a tmp file with the JSON content, you just need to place before that and after <int:service-activator id="requestTransformer"> a couple those <transformer>: one to convert a payload to the JSON (the <object-to-json-transfrormer> should do the trick for your) and another to create a file with that content. I think for the file it would be better to use a <int-file:outbound-gateway> to write a content into the file and get a java.io.File as a reply: https://docs.spring.io/spring-integration/docs/5.0.6.RELEASE/reference/html/files.html#file-writing.
You can combine all the logic into the <chain> also: https://docs.spring.io/spring-integration/docs/5.0.6.RELEASE/reference/html/messaging-routing-chapter.html#chain
I have a message driven channel adapter configured to pick messages from queue with acknowledge set to transacted. Is it possible to send different messages to different queues down stream holding the same transaction also have database inserts between in the flow?. If message delivery fails to any queue , the transaction must roll back (including database entries) as well the message send to other queues.
Example : queue(receive)--->insert Db-->send to(queue1,queue2.. etc sending different message to each queue)
if any send call fails for queue1, queue2 etc the transaction should roll back..
I am able to do the configuration with single queue (ie only with queue1). But how to do if multiple queues are involved and holding the transaction boundaries.
Thanks
Vaidya
Below is the configuration
<int-jms:message-driven-channel-adapter
id="MessageDrivenAdapter" channel="injmsChannel" destination="testQ"
concurrent-consumers="5" max-concurrent-consumers="10" acknowledge="transacted"
error-channel="Error" />
<int:channel id="injmsChannel" />
<int:chain input-channel="injmsChannel" id="Chain1">
<int-jpa:retrieving-outbound-gateway
entity-class="entity.TestTable8"
entity-manager-factory="entityManagerFactory" id-expression="payload.getSno()">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:retrieving-outbound-gateway>
<int:recipient-list-router id="ROUTE_1_2">
<int:recipient channel="SuccessChannel"
selector-expression="payload.getSno()==1" />
/> -->
</int:recipient-list-router>
</int:chain>
<int:channel id="SuccessChannel" />
<int:chain id="MessageProcessingChain" input-channel="SuccessChannel"
output-channel="putMsgChannel">
<int:service-activator id="a1" ref="taskexe"
method="processTable8_1" requires-reply="true" />
<int-jpa:retrieving-outbound-gateway
id="table7" entity-class="entity.TestTable7"
entity-manager-factory="entityManagerFactory" id-expression="payload.getSno()">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:retrieving-outbound-gateway>
<int:service-activator id="a2" ref="taskexe"
method="processTable8_2" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="table6" entity-class="entity.TestTable6"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator id="a3" ref="taskexe"
method="processTable6_1" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="uptable6" entity-class="entity.TestTable6"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator id="a4" ref="taskexe"
method="processTable6_2" requires-reply="true" />
<int-jpa:updating-outbound-gateway
id="uptable4" entity-class="entity.TestTable4"
entity-manager-factory="entityManagerFactory" flush="true">
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:updating-outbound-gateway>
<int:service-activator ref="taskexe" method="processTable4_1"
requires-reply="true" />
</int:chain>
<int:channel id="putMsgChannel" />
<int-jms:outbound-channel-adapter id="sendsomemsg"
channel="putMsgChannel" connection-factory="connectionFactory"
session-transacted="true" destination-expression="headers['somequeue']" />
How to add another int-jms:outbound-channel-adapte for other queue having same transaction boundaries of message driven adapter?. Also flush=true been set so that message is not passed downstream if there is any jpa adapter exception.
As long as the queue sends are performed using a JmsTemplate (including the use of a JMS outbound channel adapter), on the same thread, they will be performed in the same transactional session as the message-driven adapter's message delivery.
If you add the JDBC transaction manager to the message-driven adapter, its transaction will be synchronized with the JMS transaction.
This provides "Best Effort 1PC" as discussed in Dave Syer's JavaWorld article: Distributed transactions in Spring, with and without XA.
There is a small possibility that the DB commit will succeed and the JMS commit fails, so you need to deal with duplicates. To avoid that you need a full XA solution.
I am getting the following error while shutting down tomcat
SEVERE: The web application [/TestService] appears to have started a thread named [SimpleAsyncTaskExecutor-3] but has failed to stop it. This is very likely to create a memory leak.
<gateway id="testGateway" service-interface="org.example.TestGateway"
default-request-channel="requestChannel" error-channel="errorChannel"/>
public interface TestGateway {
Future execute(Request request);
}
<int:chain input-channel="requestChannel" output-channel="routerChannelA">
<int:service-activator ...
<int:transformer ....
</int:chain>
<int:router input-channel="routerChannelA" expression="payload.name" resolution-required="true">
<int:mapping value="B" channel="channelB" />
....
</int:router>
<int:chain input-channel="channelB" output-channel="channelD">
<int:transformer ......
<int:gateway request-channel="channelC" .....
<int:filter expression="headers['RELEASE'] != null" discard-="nullChannel"/>
</int:chain>
<int:recipient-list-router id="customRouter" input-channel="channelD"
timeout="1234"
ignore-send-failures="true"
apply-sequence="false" >
<int:recipient channel="splitterRequestChannel"/>
<int:recipient channel="completeChannel"/>
</int:recipient-list-router>
<int:splitter expression="payload" input-channel="splitterRequestChannel"
output-channel="splitterResponseChannel" ></int:splitter>
<int:channel id="splitterResponseChannel">
<int:dispatcher task-executor="splitterChannelTaskExecutor"/>
</int:channel>
<bean id="splitterChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:chain input-channel="splitterResponseChannel">
......
......
</int:chain>
The client sends requests to TestGateway. I dont have to send any reply back to client, but I want to return back immediately.
The Future return type serves my purpose of returning immediately. However I feel it blocks the main thread.
The request moves from a series of chains and finally reaches splitterRequestChannel. This channel delegates its work to the threads initiated by splitterChannelTaskExecutor, they do their respective jobs now. I feel the main thread should be released now as it has delegated its task, but it doesn't look its getting released.
Edit:
public interface TestGateway {
void execute(Request request);
}
<bean id="requestChannelTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" >
<property name="corePoolSize" value="10" />
<property name="daemon" value="false"/>
</bean>
<int:channel id="requestChannel">
<int:dispatcher task-executor="requestChannelTaskExecutor"/>
</int:channel>
Some thoughts...
Just return void if you don't care about the result.
How are you instantiating the application context? If using normal spring web, spring should take care of closing the context; if you are creating it yourself, it's up to you to close it.
Take a thread dump to see what the thread is doing.
EDIT:
If your flow never returns a reply, the TE within the gateway is stuck waiting for the reply.
So; return void and you won't have the problem.
My Spring Integration Xml.
<int:channel id="request-write-to-PMSQueueChannel" >
<int:queue message-store="channelStore" />
</int:channel>
<int:bridge input-channel="request-write-to-PMSQueueChannel" output-channel="writetoPMSChannel">
<int:poller fixed-rate="5000" max-messages-per-poll="-1">
<int:transactional propagation="REQUIRED" transaction-manager="transactionManager"/>
</int:poller>
</int:bridge>
<int:channel id="redBlue-error-channel"/>
<int:service-activator id="errorServiceActivator" input-channel ="redBlue-error-channel">
<bean id="errorSVC"
class="com.sds.redBlue.core.module.analyzer.sample.ErrorServiceActivator"/>
</int:service-activator>
<bean id="channelStore" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="dataSource" />
<property name="channelMessageStoreQueryProvider" ref="queryProvider" />
</bean>
<bean id="queryProvider" class="org.springframework.integration.jdbc.store.channel.DerbyChannelMessageStoreQueryProvider"/>
<int:publish-subscribe-channel id="writetoPMSChannel" ignore-failures = "false"/>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS001Channel">
<int:service-activator method="exectue">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS002Channel">
<int:service-activator method="exectue002">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS003Channel">
<int:service-activator method="exectue003">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS004Channel">
<int:service-activator method="exectue004">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
<int:chain input-channel="writetoPMSChannel"
output-channel="writetoPMS005Channel">
<int:service-activator method="exectue005">
<bean id=""
class="com.sds.redBlue.core.module.analyzer.convert.ModelingConvertSVC">
</bean>
</int:service-activator>
<int:splitter ref="fromListToRowSplitter" />
</int:chain>
I want to 5 subscribers have to be executed at once or have to be rollback if one subscribe throws any exception. But I can't find the way to sovle.
related article:
Pub-Sub error handling strategy
I already asked with the the eip pattern pic and Gary helped. but, I stucked.
(Spring Integration : How to guarantee the transaction two more jdbc-outbound-gateway?)
Since transaction is bounded to the Thread, you should be sure that all your subscribers are withing the same direct flow. In this case they will be invoked one by one.
However I see that you use ignore-failures = "false". Having you ignore all downstream exception and allow for your subscribers to do their work. Although you lose TX rollback, of course.
So, revise your use-case if you really want to have automatic rallback.
There is an async trick to control TX, but it is a bit complex and require some logic with aggregator.
I've got for a JPA Outbound-channel-adapter both transactional and request-handler-advice-chain. In the advice-chain I try to log the Exception, when it happens.
It iss not logged, but I know that it actually happend since the Message was sent to failover clickDbFailoverChannel . What can be a problem with it? Is it a bug in Spring Integration?
<int:channel id="clickDbWithFailoverChannelSite-1">
<int:dispatcher load-balancer="none" task-executor="clickDbSiteRouterExecutor"/>
</int:channel>
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="jpaOutboundChannelSite-1" order="1" send-timeout="100" />
<int:bridge input-channel="clickDbWithFailoverChannelSite-1"
output-channel="clickDbFailoverChannel" order="2" />
<int-jpa:outbound-channel-adapter id="jpaOutboundChannelSite-1"
persist-mode="PERSIST" flush-size="100" entity-manager-factory="emfSite-1">
<int-jpa:transactional transaction-manager="transactionManagerSite-1" />
<int-jpa:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="failureChannel" ref="clickDbFailureLogger"/>
<property name="onFailureExpression" value="#exception"/>
</bean>
</int-jpa:request-handler-advice-chain>
</int-jpa:outbound-channel-adapter>
OK. I can guess where is your issue. The real exception to rollback the transaction is caused before an internal logic, where <request-handler-advice-chain> does the stuff. That's why your ExpressionEvaluatingRequestHandlerAdvice doesn't get a failure message.
To workaround your rollback issue, you should replace <int-jpa:transactional> with <tx:advice> within <int-jpa:request-handler-advice-chain>.
You should understand here that <int-jpa:transactional> is for entire MessageHandler.handleMessage, but <int-jpa:request-handler-advice-chain> is just for its part - AbstractReplyProducingMessageHandler.handleRequestMessage.
UPDATE
TX Advice should be like this:
<tx:advice transaction-manager="transactionManagerSite-1"/>
<tx:attributes>
<tx:method name="*"/>
</tx:attributes>
</tx:advice>