We are using Spring Batch and partitioning of jobs in a 10 server JBoss EAP 5.2 cluster. Because of a problem in JBoss messaging, we needed to use a topic for the reply message from the partitioned steps. All has been working fine until we see JBoss Messaging glitches (on the server that launches the job) and that drops it from the cluster. It recovers but the main partition does no``t pick up the messages sent from the partition steps. I see the messages in the topic in the JMX-Console but also see that the subscription and the messages are non-durable. Therefore I would like to make the communication for the partition step reply into a durable subscription. I can't seem to fine a document way to do this. This is my current configuration of the partitioned step and associated bean.
Inbound Gateway Configuration
<int:channel id="springbatch.slave.jms.request"/>
<int:channel id="springbatch.slave.jms.response" />
<int-jms:inbound-gateway
id="springbatch.master.inbound.gateway"
connection-factory="springbatch.listener.jmsConnectionFactory"
request-channel="springbatch.slave.jms.request"
request-destination="springbatch.partition.jms.requestsQueue"
reply-channel="springbatch.slave.jms.response"
concurrent-consumers="${springbatch.partition.concurrent.consumers}"
max-concurrent-consumers="${springbatch.partition.concurrent.maxconsumers}"
max-messages-per-task="${springbatch.partition.concurrent.maxmessagespertask}"
reply-time-to-live="${springbatch.partition.reply.time.to.live}"
/>
Outbound Gateway Configuration
<int:channel id="jms.requests">
<int:dispatcher task-executor="springbatch.partitioned.jms.taskExecutor" />
</int:channel>
<int:channel id="jms.reply" />
<int-jms:outbound-gateway id="outbound-gateway"
auto-startup="false" connection-factory="springbatch.jmsConnectionFactory"
request-channel="jms.requests"
request-destination="springbatch.partition.jms.requestsQueue"
reply-channel="jms.reply"
reply-destination="springbatch.partition.jms.repliesQueue"
correlation-key="JMSCorrelationID">
<int-jms:reply-listener />
</int-jms:outbound-gateway>
</code>
Further to Michael's comment; there is currently no way to configure a topic for the <reply-listener/> - it's rather unusual to use a topic in a request/reply scenario and we didn't anticipate that requirement.
Feel free to open a JIRA Issue against Spring Integration.
An alternative would be to wire in an outbound-channel-adapter for the requests and an inbound-channel-adapter for the replies. However, some special handling of the replyChannel header is needed when doing that - see the docs here for more information about that.
Related
<int:channel id="emailInputChannel"/>
<!-- Email Poller. Only one poller thread -->
<task:executor id="emailPollingExecutor" pool-size="1" />
<int-mail:inbound-channel-adapter id="pop3EmailAdapter" store-uri="pop3://${pop3.user}:${pop3.pwd}#${pop3.server.host}/Inbox"
channel="emailInputChannel" should-delete-messages="true" auto-startup="true" java-mail-properties="javaMailProperties">
<int:poller max-messages-per-poll="1" fixed-delay="${email.poller.delay}" task-executor="emailPollingExecutor"/>
</int-mail:inbound-channel-adapter>
<!-- Java Mail POP3 properties -->
<util:properties id="javaMailProperties">
<beans:prop key="mail.debug">true</beans:prop>
<beans:prop key="mail.pop3.port">${pop3.server.port}</beans:prop>
</util:properties>
This application polls for emails containing application file attachments which contain the data to process. The email attachments are sent typically a few a day and are relatively sporadic. Since the files contain data for bulk load, we resorted to this configuration with a single poller for the Inbound POP3 mail adapter. Having multiple pollers caused duplicate poller invocations to pull the same email while another poller is processing it. With this configuration, however, the single poller hangs after some time with no indications of the problem in the logs. Please review what is wrong with this configuration. Also, is there is an alternative way to trigger the email adapter (e.g cron etc at a periodic interval)? I am using Spring Integration 2.1
A hung poller is most likely caused by the the thread stuck in user code. I see you have mail.debug=true. If that shows no activity then a hung thread is probably the cause. Use us jstack to take a thread dump.
Yes, you can use a cron expression but that's unlikely to change things.
2.1 is extremely old but I still think a hung thread is the cause.
I have four subflows each sitting in its own application and packaged in jar file.
Each subflow is build as spring boot application. Each subflow has input channel and output channel.
I would like the to have the main flow sitting in its own spring-boot application that will sequentially called those 4 subflows.
Is it possible with spring integration?
If it is not possible what would be the best approach that is possible?
I searched the internet and look at How to handle subflows
https://github.com/spring-projects/spring-integration-flow and everything else that I can find but still not sure how to proceed.
https://github.com/spring-projects/spring-integration-flow states that to use subflow I need to have in my xml
int-flow:flow id="subflow1"
How do I tie this subflow1 to my subflow packaged in the separate jar. Then how would I invoke this subflows in my main flow that is spring boot application? Do I sent message to the input channel to each subflow to start subflow or do something else?
Thanks,David
Example of subflow xml file. I removed irrelevant parts of the file to shorten it and put ... in place of removed info
<int:channel id="createTwo"/>
<int:service-activator input-channel="createOne" output-channel="createTwo"
ref="automationUtilities" method="createTwo"/>
...
<int:service-activator input-channel="createFive"
ref="automationUtilities" method="createSix"/>
<bean id="automationUtilities" class="package.BeanName" />
<bean id="validator" class="package.anotherBeanName" />
<util:properties id="config" location="classpath:application.properties"/>
If all your applications are Spring Boot, then they are Microservices and each of them lives in its onw JVM. Therefore just having input and output channel isn't enough. Because you can't just send message from one JVM to another.
Since all of them are Spring Boot, consider to add some REST capabilities for them. At least simple <int-http:inbound-gateway> to receive messages from external world. And <int-http:outbound-gateway> to perform REST request from one application to another.
OTOH consider to use Spring Cloud Stream project, which is founded exactly for Messaging Microservices communication. It is based on the Binder concept, currently Kafka or RabbitMQ.
Issue:
Have 1 active MDP (jmsIn below) attached to a single queue and keep the second clustered MDP on server 2 passive. I require only a single active process because I want to perform aggregation and not lose messages.
I have been reading about the control bus but since its a clustered environment , the channel id and jms:message-driven-channel-adapter id would have the save name on both servers. Is it possible for the control bus to deactivate on another server using JMX even though they have the id's? Or possibly have a check done first by the message-driven-channel-adapter to determine if there is already a connection active on the queue itself.
Message-driven-channel-adapter Sample Code:
<jms:message-driven-channel-adapter id="jmsIn"
destination="requestQueue"
channel="jmsInChannel" />
<channel id="jmsInChannel" />
<beans:beans profile="default">
<stream:stdout-channel-adapter id="stdout" channel="jmsInChannel" append-newline="true"/>
</beans:beans>
<beans:beans profile="testCase">
<bridge input-channel="jmsInChannel" output-channel="queueChannel"/>
<channel id="queueChannel">
<queue />
</channel>
</beans:beans>
There is no reason to worry about clustered singleton for the <message-driven-channel-adapter>, just because any queuing system (like JMS) is designed for the single consumer for the particular message. In other words independently of the number of processes on the queue, only one of them picks up the message and process it.
The Transaction semantics on JMS help you do not lose messages. If there is some Exception, the TX rallbacks and the message is returned to the queue and can be picked up by another consumer.
For the aggregator you really can use some distributed persistent MessageStore. Where all your consumer send their messages to the same <aggregator> component which just deals with the shared MS to do its own aggregation logic.
I have a scenario where I would like to separate the flow into a number of transactions. I am using queue channels based on a JdbcChannelMessageStore to do so and that works excellent. Its robust and it just works. But because these Jdbc based queues (the database) are polled by the executors, I get a natural limitation on the throughput (I don't really want to configure the poller to poll every 1 millisecond). So my question is this, is there a way for the queue channel to notify the consumer of that channel that a new messages has been queued, and then trigger the "poller" to have a look in the database to see what has to be consumed?
So the simple scenario:
1. A queue channel where someone puts a message
2. A service activator that will process that message (in parallel)
<int:channel id="InputChannel">
<int:queue message-store="jdbcChannelStore"/>
</int:channel>
<task:executor id="TradeTransformerExecutor" pool-size="2-20" queue-capacity="20" rejection-policy="CALLER_RUNS"/>
<int:service-activator id="TradeConverter" input-channel="InputChannel" output-channel="TradeChannel" method="transform">
<beans:bean class="com.service.TradeConverter"/>
<int:poller task-executor="TradeTransformerExecutor" max-messages-per-poll="-1" receive-timeout="0" fixed-rate="100">
<int:transactional transaction-manager="dbTransactionManager"/>
</int:poller>
</int:service-activator>
<int:channel id="TradeChannel"></int:channel>
So how could I make this InputChannel notify the poller (or something else) to start executing the message right away and not wait for 100ms?
Also I don't want to use DirectChannels as I do want some persistence between defined flows for robustness reasons.
Cheers guys.
Jonas
There's no way to change a trigger on demand; you can have a dynamic trigger, but changes only take effect after the next poll.
Instead of using a JDBC-backed channel, consider using an outbound channel adapter to store the data and a JDBC outbound gateway (with just a query, no update).
Use a pub-sub channel and, after storing, send the message (perhaps via a bridged ExecutorChannel) to the gateway.
Alternatively, simply inject your queue channel into a service and invoke it via a <service-activator/>. You would need a pub-sub channel bridged to your queue channel, with the second subscriber being the service activator which, when it receives the message calls receive() on the channel.
Finally, consider using a JMS, or RabbitMQ backed-channel for high performance persistence instead - they are much better a queueing than a database.
Having defined a channel adapter as:
<int:channel id="target">
<int:queue />
</int:channel>
<int-jdbc:inbound-channel-adapter id="adapter" channel="target" query="${int.poll.query}" update="${int.update.query}" data-source="mock-datasource">
<int:poller fixed-rate="5000"/>
</int-jdbc:inbound-channel-adapter>
I wonder why I cannot modify the polling rate on runtime, as follows:
SourcePollingChannelAdapter adapter = applicationContext.getBean("adapter",SourcePollingChannelAdapter.class);
adapter.setTrigger(new PeriodicTrigger(1000));
When i debug this solution, I can see that the adapter has this new trigger attached to it, however the polling rate remains unchanged (every 5 secs). I tried also to stop() and start() the adapter, with similar luck.
Anyone can point me out what I am doing wrong?
Thanks
[RESOLVED]
It has been confirmed by members of Spring team, that a trigger cannot be modified on runtime. So if you want to modify the polling rate dynamically, for example to throttle inbound messages, you will have to roll your own Trigger implementation and add a setter for the interval polling.
I leave here the changes done in my configuration:
<int-jdbc:inbound-channel-adapter id="bancsAdapter" channel="target" query="${int.bancs.poll.query}" update="${int.bancs.update.query}" data-source="bancsMockDB">
<int:poller trigger="dynamicTrigger" />
</int-jdbc:inbound-channel-adapter>
<bean id="dynamicTrigger" class="directlabs.integration.DynamicTrigger">
<constructor-arg value="5000" />
</bean>
So for throttling, you only need to do the following:
applicationContext.getBean("dynamicTrigger",DynamicTrigger.class).setPeriod(1000);
The implementation of the DynamicTrigger can be found here
The original comments from the Spring team members can be found here.
While space here does not allow for a full example, we created a Service that uses Quartz Scheduler as a triggering mechanism. It accepts an XML document with the Quartz Jobs and Triggers defined (this stack overflow describes the process Use simple xml to drive the Quartz Sheduler )
The input channel will accept the XML to be used for setting the Schedules in Quartz. The input channel then can be used to accept dynamic updates of jobs and triggers.
The xml entries in the job-map-data will have an "output" channel defined and one can add other job-map-data that can be set in the output message header to allow for routing.
We constantly re-use this Service in many of our Spring Integration contexts.
Hope this helps.