I am working on an existing Spring Integration code that is getting stuck.
Below code is getting struck. Around 20,000 records are fetched from the sql query and sent to splitter.
code:
<int-jdbc:outbound-gateway query="..." />
<int:splitter input-channel="..." output-channel="queueChannel"/>
<int:channel id="queueChannel">
<int:queue capacity="25" />
</int:channel>
<int:service-activator ref="..."
input-channel="queueChannel" output-channel="..." method="xxx">
<int:poller max-messages-per-poll="25" fixed-delay="100"
receive-timeout="30000" task-executor="reqExecutor"/>
</int:service-activator>
<task:executor id="reqExecutor" pool-size="25" queue-capacity="5" rejection-policy="CALLER_RUNS" />
After doing some search on the internet, here is my understanding of the code. Please correct me, If I am wrong:
splitter output channel is queue channel with capacity of 25 that means it will fetch batch of 25 records from the query.
Now, the code written in service activator will be polled every 100 ms and fetch 25 messages from the queue channel. Service Activator runs in multi-threaded environment with task executor.
Task executor has a pool size of 25. Thats the max number of threads that can run at a time. and the queue-capacity of 5. If all threads are busy then Executor will put them in queue. If queue capacity is reached to 5 then executor will reject the task.
The rejection policy is CALLER_RUNS. The Executor will use the main process on rejection.
Application Performance may get affected by CALLER_RUNS rejection policy. But other policies discard or abort the thread. So, changing rejection policy is not a solution.
Shall I change pool-size or queue-capacity of the task executor to resolve the issue. Is there any preferred value. What shall be the impact of it.
Or, Shall I change the fixed-delay of poller?
EDIT 1:
In the logs, below line is repeatedly coming and the process is not finishing:
Received no Message during the poll, returning 'false'
EDIT 2:
Is my issue related to issue mentioned in the link below:
http://docs.spring.io/autorepo/docs/spring-integration/3.0.x/reference/html/messaging-endpoints-chapter.html#async-polling
Also, I don't clearly understand the receive-timeout attribute of poller.
The receive-timeout is how long the poller (reqExecutor) thread waits in the queue channel for a message to arrive. If it expires after no message arrives, the thread is returned to the pool.
If a message arrives, it is processed on the thread and then the thread goes back to the pool.
If you can't work it out from the thread dump, post it someplace (not here - probably too big) - pastebin or a github gist.
It is resolved after doing some code optimization. There were two major problems in the code:
hashcode and equals method of the key of a HashMap were not overrided. The hashmap was used to cache the queries but in abscent of overrided methods it was not working correctly.
There was an insert query in the Spring Integration code that was inserting records around 20,000 times. Since, AFAIK we cannot do batch update in Spring Integration. So, I extract insert query into java class and did batch update.
However, I am wondering, why thread dump and memory dump did not give any hint about it?
Thanks Geek and Gary for helping me out.
Related
I have a flow that starts with a poller and hands off the message to several async flows downstream using task-executors to execute in parallel for a given dataset. A downstream aggregator completes the flow and notifies the poller that the flow is complete.
I would like to track every execution of the poller by using MDC so that the logs can be mapped to a particular execution of the flow.
I started by adding MDC to the poller thread (using Advice), however with this approach there could be a couple of issues:
How do I stamp the MDC on the executor thread when the async hand off happens?
Since executor uses a a thread pool, do I need to clear the MDC before the thread returns to the pool? Will there be any side effects?
Another approach would be to add MDC to the Message header and set it manually on the new thread during the async handoff. How to do that? For example, if I turn on the debug logs, the MDC should be stamped right from the beginning of the new thread execution and not from the point where my logic starts in the service activator.
How to set this on the task-executor thread (and probably also remove before returning to the pool) using XML configuration? Something like an MdcAwareThreadPoolExecutor seen here.
Also, I would not want the MDC logic to be spread across all the async handoff endpoints, may be there is some generic way to configure it?
Is there a better way to achieve this? Any known solutions?
I would like to track every execution of the poller by using MDC so that the logs can be mapped to a particular execution of the flow.
It is fully sound as "you would like to track the message journey in your flow". As you noticed there is the way to set some message header. So, why just don't map your logs by this specific header?
You can take a look into Message History pattern how to gather the whole path for the message, so then in logs you can track it back looking into message headers.
See here: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/system-management.html#message-history
If you really still insist on the MDC, then you definitely need to take a look into some MDCDelegatingExecutorDecorator. Some sample you can borrow from Spring Security and its DelegatingSecurityContextExecutor`: https://docs.spring.io/spring-security/site/docs/5.4.0/reference/html5/#concurrency
I have a jdbc inbound endpoint that selects tens of thousands of records. Mule automatically spits them up and process each of them concurrently. The problem is, the process involves calling another service that cannot handle so many requests at the same time.
Is there a way to limit the number of concurrent threads so that not that many requests happen at the same time?
I am reading http://www.mulesoft.org/documentation/display/current/Configuring+a+Transport#ConfiguringaTransport-receiver but I cannot grasp how to do it. I tried:
<jdbc-ee:inbound-endpoint doc:name="db" connector-ref="testConnector" exchange-pattern="one-way" pollingFrequency="60000" queryTimeout="-1" queryKey="findAllPersonIds">
<receiver-threading-profile maxThreadsActive="2" />
</jdbc-ee:inbound-endpoint>
But when I try to start it, Mule complains that 'receiver-threading-profile' isn't valid.
A JDBC inbound endpoint is a poller endpoint, which is backed by a single thread per Mule instance (or per cluster if you run EE).
The parallelism you're experiencing comes from the flow processing strategy, which by default has a threading profile that will use multiple concurrent threads.
You need to limit this parallelism for the flow that performs the remote HTTP invocation. You haven't shown your config so I can't tell if it's the same flow where the inbound JDBC endpoint is.
With the information you have provided, the best I can do is direct you to the reference documentation: http://www.mulesoft.org/documentation/display/current/Flow+Processing+Strategies
From this documentation, here is a flow that will use 500 concurrent threads:
<queued-asynchronous-processing-strategy name="allow500Threads"
maxThreads="500"/>
<flow name="manyThreads" processingStrategy="allow500Threads">
<vm:inbound-endpoint path="manyThreads" exchange-pattern="one-way"/>
<vm:outbound-endpoint path="output" exchange-pattern="one-way"/>
</flow>
The SI workflow starts with an inbound-channel-adapter which runs under a new transaction started by a pollar.
The adapter triggers a data processing flow which kind of fans out, for example the adapter polls a database and get few rows, splits and next adapter makes another db call (hence for each row) as input which produces several other rows and so on.
Right now its running as single threaded behavior as I only want to commit the original transaction when everything went well.
Now I want to to speed up the processing by running it under more threads, so if one of my original adapter produced 3 rows, I want to process them simultaneously for downstream flow and so on.
is this possible and if yes, how can I define a global task executor with some configuration and allow the processing at various stages to be executed under this.
Thank you
You could insert a <gateway/> in the flow between the inbound adapter and the rest of the flow; the poller thread will be suspended in the gateway awaiting a reply. You would likely need an aggregator to send the reply to the gateway when the other tasks are complete. You would discard the reply.
<service-activator input-channel="fromAdapter" ref="gw" output-channel="nullChannel" />
<gateway default-request-channel="toRestOfFlow" />
The problem is if the reply is never received the thread will wait forever. If you add a reply-timeout, the transaction will commit after the timeout. So you may need some additional logic to handle that, perhaps adding your own bean that invokes the gateway and detects a null reply.
Of course, in all these scenarios; the async tasks cannot participate in the original transaction.
I have a spring integration configuration where I am polling files from a directory and then processing those files in later steps.
I am using a dispatcher channel for processing steps which allows the poller thread to be released and return back to continue to do polling while processing steps are happening.
This works fine until the thread pool limit is reached, at that point because of CALLER_RUNS rejection policy when the poller submits the file to processing step it ends up doing the processing in the same thread as the poller which stops the polling process to be put on halt until this step is completed.
How can I c*ompletely decouple the two stages* of this process?
My thinking was to use two task-executors, one for polling and another for dispatcher but that doesn't seems to help.
<!-- Poll files from landing zone directory -->
<int-file:inbound-channel-adapter id="files" directory="${lz.dir.${ft}}" filename-regex=".*\.txt$">
<int:poller fixed-delay="3000" max-messages-per-poll="2" task-executor="pollingExecutor" />
</int-file:inbound-channel-adapter>
<int:bridge input-channel="files" output-channel="sourceFiles" />
<!-- Dispatch retrieved files -->
<int:channel id="sourceFiles">
<int:dispatcher task-executor="processingExecutor" />
</int:channel>
Basically I want the polling process to never stop. ANy help would be appreciated.
Thanks
Use a QueueChannel; the poller dumps the file into the queue an you have a poller on the other side pulling messages out of the queue.
The queue is unbounded by default.
Or, of course, you can use an unbounded queue in your executor.
But if your consumer(s) can't keep up, you'll eventually get into trouble.
My requirement is to use transactional session with message-driven-channel-adapter (JmsMessageDrivenEndpoint). I am able to setup the configuration buy using sessionTransacted = true for DefaultMessageListenerContainer.
Work flow: receive a message -> call the service activator -> service activator calls dao class
On successful commit to database a commit() is called by spring framework and on any runtime exception a rollback() is called by spring framework. Which works just fine. When a rollback happens JMS Broker sends the message back again to my application.
For a specific type of exception in dao I want to add a message header (i.e redelivery time) so that JMS Broker will not send the message again right away. How can I do it?
For another specific type of exception in dao I want to use control bus to stop the end point (message-driven-channel-adapter) and before stopping it rollback the previous transaction. How can I do it?
Any one can help me out?
There is no wonder, how to use Control Bus for start/stop endpoints:
<int:control-bus input-channel="controlChannel"/>
<int-jms:message-driven-channel-adapter id="jmsInboundEndpoint"/>
<int:transformer input-channel="stopImsInboundEndpointChannel"
outbound-channel="controlChannel"
expression="'#jmsInboundEndpoint.stop()'"/>
Or you can send to the controlChannel the same command string from any place of your code.
But it doesn't matter that the last transaction will be rollbacked. It depends on your 'unit of work' (in other words - the behaviour of your service).
However you can, at the same time when you send 'stop command', mark current transaction for rollback:
TransactionAspectSupport.currentTransactionStatus().setRollbackOnly();
Another your question about 'adding some message header' is abnormal for Messaging at all.
If you change the message it will be a new one and you can't rollback message to the queue with some new info.
Of course, you can do it anyway and have new message. But you should resend it, not rollback. So, you should commit transaction anyway and send that new message somewhere (or to the same queue), but it will be new message as for Broker as well for your application. And one more time: for this case you have to commit transaction.
Not sure that it is very clear and I go right way in my asnwer, but hope it helps you a bit.
You cannot modify the message (add a header) before rollback. You could, of course, requeue it as a new message after catching the exception. Some brokers (e.g. ActiveMQ) provide a back-off retry policy after a rollback. That might be a better solution if your broker supports it.
You can use the control bus to stop the container, but you will probably have to do it asynchronously (invoke the stop on another thread, e.g. by using an ExecutorChannel on the control bus). Otherwise, depending on your environment you might have problems with the stop waiting for the container thread to exit, so you shouldn't execute the stop on the container thread itself.
Best thing to do is experiment.
Thanks Gary and Artem. The solution is working. I am using the below configuration:
<jms:message-driven-channel-adapter id="jmsMessageDrivenChannelAdapter" connection-factory="connectionFactory"
destination="destination" transaction-manager="jmsTransactionManager" channel="serviceChannel" error-channel="ultimateErrorChannel" />
<si:service-activator input-channel="ultimateErrorChannel" output-channel="controlChannel">
<bean class="play.spring.integration.TestErrorHandler">
<property name="adapterNeedToStop" value="jmsMessageDrivenChannelAdapter" />
<property name="exceptionWhenNeedToStop" value="play.spring.integration.ShutdownException" />
</bean>
</si:service-activator>
<si:channel id="controlChannel">
<si:dispatcher task-executor="controlBusExecutor" />
</si:channel>
<task:executor id='controlBusExecutor' pool-size='10' queue-capacity='50' />
<si:control-bus input-channel="controlChannel" />
Now my question is if I want to stop multiple inbound adapters how can I send a single message to control-bus for all these adapters?
I am going to study SpEL. Would appreciate if someone already know it.
Thanks