I have a spring integration configuration where I am polling files from a directory and then processing those files in later steps.
I am using a dispatcher channel for processing steps which allows the poller thread to be released and return back to continue to do polling while processing steps are happening.
This works fine until the thread pool limit is reached, at that point because of CALLER_RUNS rejection policy when the poller submits the file to processing step it ends up doing the processing in the same thread as the poller which stops the polling process to be put on halt until this step is completed.
How can I c*ompletely decouple the two stages* of this process?
My thinking was to use two task-executors, one for polling and another for dispatcher but that doesn't seems to help.
<!-- Poll files from landing zone directory -->
<int-file:inbound-channel-adapter id="files" directory="${lz.dir.${ft}}" filename-regex=".*\.txt$">
<int:poller fixed-delay="3000" max-messages-per-poll="2" task-executor="pollingExecutor" />
</int-file:inbound-channel-adapter>
<int:bridge input-channel="files" output-channel="sourceFiles" />
<!-- Dispatch retrieved files -->
<int:channel id="sourceFiles">
<int:dispatcher task-executor="processingExecutor" />
</int:channel>
Basically I want the polling process to never stop. ANy help would be appreciated.
Thanks
Use a QueueChannel; the poller dumps the file into the queue an you have a poller on the other side pulling messages out of the queue.
The queue is unbounded by default.
Or, of course, you can use an unbounded queue in your executor.
But if your consumer(s) can't keep up, you'll eventually get into trouble.
Related
<int:channel id="emailInputChannel"/>
<!-- Email Poller. Only one poller thread -->
<task:executor id="emailPollingExecutor" pool-size="1" />
<int-mail:inbound-channel-adapter id="pop3EmailAdapter" store-uri="pop3://${pop3.user}:${pop3.pwd}#${pop3.server.host}/Inbox"
channel="emailInputChannel" should-delete-messages="true" auto-startup="true" java-mail-properties="javaMailProperties">
<int:poller max-messages-per-poll="1" fixed-delay="${email.poller.delay}" task-executor="emailPollingExecutor"/>
</int-mail:inbound-channel-adapter>
<!-- Java Mail POP3 properties -->
<util:properties id="javaMailProperties">
<beans:prop key="mail.debug">true</beans:prop>
<beans:prop key="mail.pop3.port">${pop3.server.port}</beans:prop>
</util:properties>
This application polls for emails containing application file attachments which contain the data to process. The email attachments are sent typically a few a day and are relatively sporadic. Since the files contain data for bulk load, we resorted to this configuration with a single poller for the Inbound POP3 mail adapter. Having multiple pollers caused duplicate poller invocations to pull the same email while another poller is processing it. With this configuration, however, the single poller hangs after some time with no indications of the problem in the logs. Please review what is wrong with this configuration. Also, is there is an alternative way to trigger the email adapter (e.g cron etc at a periodic interval)? I am using Spring Integration 2.1
A hung poller is most likely caused by the the thread stuck in user code. I see you have mail.debug=true. If that shows no activity then a hung thread is probably the cause. Use us jstack to take a thread dump.
Yes, you can use a cron expression but that's unlikely to change things.
2.1 is extremely old but I still think a hung thread is the cause.
Issue:
Have 1 active MDP (jmsIn below) attached to a single queue and keep the second clustered MDP on server 2 passive. I require only a single active process because I want to perform aggregation and not lose messages.
I have been reading about the control bus but since its a clustered environment , the channel id and jms:message-driven-channel-adapter id would have the save name on both servers. Is it possible for the control bus to deactivate on another server using JMX even though they have the id's? Or possibly have a check done first by the message-driven-channel-adapter to determine if there is already a connection active on the queue itself.
Message-driven-channel-adapter Sample Code:
<jms:message-driven-channel-adapter id="jmsIn"
destination="requestQueue"
channel="jmsInChannel" />
<channel id="jmsInChannel" />
<beans:beans profile="default">
<stream:stdout-channel-adapter id="stdout" channel="jmsInChannel" append-newline="true"/>
</beans:beans>
<beans:beans profile="testCase">
<bridge input-channel="jmsInChannel" output-channel="queueChannel"/>
<channel id="queueChannel">
<queue />
</channel>
</beans:beans>
There is no reason to worry about clustered singleton for the <message-driven-channel-adapter>, just because any queuing system (like JMS) is designed for the single consumer for the particular message. In other words independently of the number of processes on the queue, only one of them picks up the message and process it.
The Transaction semantics on JMS help you do not lose messages. If there is some Exception, the TX rallbacks and the message is returned to the queue and can be picked up by another consumer.
For the aggregator you really can use some distributed persistent MessageStore. Where all your consumer send their messages to the same <aggregator> component which just deals with the shared MS to do its own aggregation logic.
I am working on an existing Spring Integration code that is getting stuck.
Below code is getting struck. Around 20,000 records are fetched from the sql query and sent to splitter.
code:
<int-jdbc:outbound-gateway query="..." />
<int:splitter input-channel="..." output-channel="queueChannel"/>
<int:channel id="queueChannel">
<int:queue capacity="25" />
</int:channel>
<int:service-activator ref="..."
input-channel="queueChannel" output-channel="..." method="xxx">
<int:poller max-messages-per-poll="25" fixed-delay="100"
receive-timeout="30000" task-executor="reqExecutor"/>
</int:service-activator>
<task:executor id="reqExecutor" pool-size="25" queue-capacity="5" rejection-policy="CALLER_RUNS" />
After doing some search on the internet, here is my understanding of the code. Please correct me, If I am wrong:
splitter output channel is queue channel with capacity of 25 that means it will fetch batch of 25 records from the query.
Now, the code written in service activator will be polled every 100 ms and fetch 25 messages from the queue channel. Service Activator runs in multi-threaded environment with task executor.
Task executor has a pool size of 25. Thats the max number of threads that can run at a time. and the queue-capacity of 5. If all threads are busy then Executor will put them in queue. If queue capacity is reached to 5 then executor will reject the task.
The rejection policy is CALLER_RUNS. The Executor will use the main process on rejection.
Application Performance may get affected by CALLER_RUNS rejection policy. But other policies discard or abort the thread. So, changing rejection policy is not a solution.
Shall I change pool-size or queue-capacity of the task executor to resolve the issue. Is there any preferred value. What shall be the impact of it.
Or, Shall I change the fixed-delay of poller?
EDIT 1:
In the logs, below line is repeatedly coming and the process is not finishing:
Received no Message during the poll, returning 'false'
EDIT 2:
Is my issue related to issue mentioned in the link below:
http://docs.spring.io/autorepo/docs/spring-integration/3.0.x/reference/html/messaging-endpoints-chapter.html#async-polling
Also, I don't clearly understand the receive-timeout attribute of poller.
The receive-timeout is how long the poller (reqExecutor) thread waits in the queue channel for a message to arrive. If it expires after no message arrives, the thread is returned to the pool.
If a message arrives, it is processed on the thread and then the thread goes back to the pool.
If you can't work it out from the thread dump, post it someplace (not here - probably too big) - pastebin or a github gist.
It is resolved after doing some code optimization. There were two major problems in the code:
hashcode and equals method of the key of a HashMap were not overrided. The hashmap was used to cache the queries but in abscent of overrided methods it was not working correctly.
There was an insert query in the Spring Integration code that was inserting records around 20,000 times. Since, AFAIK we cannot do batch update in Spring Integration. So, I extract insert query into java class and did batch update.
However, I am wondering, why thread dump and memory dump did not give any hint about it?
Thanks Geek and Gary for helping me out.
I have a scenario where I would like to separate the flow into a number of transactions. I am using queue channels based on a JdbcChannelMessageStore to do so and that works excellent. Its robust and it just works. But because these Jdbc based queues (the database) are polled by the executors, I get a natural limitation on the throughput (I don't really want to configure the poller to poll every 1 millisecond). So my question is this, is there a way for the queue channel to notify the consumer of that channel that a new messages has been queued, and then trigger the "poller" to have a look in the database to see what has to be consumed?
So the simple scenario:
1. A queue channel where someone puts a message
2. A service activator that will process that message (in parallel)
<int:channel id="InputChannel">
<int:queue message-store="jdbcChannelStore"/>
</int:channel>
<task:executor id="TradeTransformerExecutor" pool-size="2-20" queue-capacity="20" rejection-policy="CALLER_RUNS"/>
<int:service-activator id="TradeConverter" input-channel="InputChannel" output-channel="TradeChannel" method="transform">
<beans:bean class="com.service.TradeConverter"/>
<int:poller task-executor="TradeTransformerExecutor" max-messages-per-poll="-1" receive-timeout="0" fixed-rate="100">
<int:transactional transaction-manager="dbTransactionManager"/>
</int:poller>
</int:service-activator>
<int:channel id="TradeChannel"></int:channel>
So how could I make this InputChannel notify the poller (or something else) to start executing the message right away and not wait for 100ms?
Also I don't want to use DirectChannels as I do want some persistence between defined flows for robustness reasons.
Cheers guys.
Jonas
There's no way to change a trigger on demand; you can have a dynamic trigger, but changes only take effect after the next poll.
Instead of using a JDBC-backed channel, consider using an outbound channel adapter to store the data and a JDBC outbound gateway (with just a query, no update).
Use a pub-sub channel and, after storing, send the message (perhaps via a bridged ExecutorChannel) to the gateway.
Alternatively, simply inject your queue channel into a service and invoke it via a <service-activator/>. You would need a pub-sub channel bridged to your queue channel, with the second subscriber being the service activator which, when it receives the message calls receive() on the channel.
Finally, consider using a JMS, or RabbitMQ backed-channel for high performance persistence instead - they are much better a queueing than a database.
I want to configure a spring integration application so that if I put a number of tasks, each represented by one message, on a channel then one of a group of endpoints will pick the next task and process it. This would entail some thread pool executor service I guess.
Yes, use a dispatcher + task executor with the channel (aka ExecutorChannel). That way any endpoint (e.g. service-activator) consuming from the channel will be invoked asynchronously using dispatcher's thread pool.
In the following example, any messages landing on channel channel01 will be consumed by the jobLauncher service within one of the taskExecutor threads.
<int:channel id="channel01">
<int:dispatcher task-executor="taskExecutor">
</int:channel>
<task:executor id="taskExecutor" pool-size="2"/>
<int:service-activator input-channel="channel01" ref="jobLauncher">