I am migrating Spring Batch Partition Jobs with XML configuration to Spring batch 4.x. I am trying to take advantage to an improvement in the MessageChannelPartitionHandler where it looks for completion of remote steps with both a reply channel and a datasource polling.
When I use this configuration:
<int:channel id="partitioned.jms.requests">
<int:dispatcher task-executor="springbatch.partitioned.jms.taskExecutor"/>
</int:channel>
<int:channel id="partitioned.jms.reply" />
<bean id="partitioned.jms.handler" class="org.springframework.batch.integration.partition.MessageChannelPartitionHandler">
<property name="messagingOperations">
<bean class="org.springframework.integration.core.MessagingTemplate">
<property name="defaultChannel" ref="partitioned.jms.requests"/>
</bean>
</property>
<property name="stepName" value="process.partitioned.step"/>
<property name="gridSize" value="${process.step.partitioned.gridSize}"/>
<property name="dataSource" ref="springbatch.repositoryDataSource" />
<property name="pollInterval" value="${springbatch.partition.verification.interval}"/>
</bean>
The step completes but I see an error in the logs.
no output-channel or replyChannel header available
I looked at the class and see I can add a replyChannel property to the MessageChannelPartitionHandler class. If I add the following:
<property name="replyChannel" ref="claim.acp.process.partitioned.jms.reply"/>
I get error back that a pollable channel is needed.
How do I create a pollable channel (assuming from the same JMS queue)?
You need to show the rest of your configuration.
If you are using DB polling for the results, set the output-channel on the jms outbound gateway to "nullChannel" and the replies received over JMS will be discarded.
Or, use an outbound channel adapter (instead of a gateway) (and an inbound-channel-adapter on the slaves). That avoids the replies being returned altogether.
You have to set pollRepositoryForResults to true.
To answer your specific question
<int:channel id="replies>
<int:queue />
<int:channel>
Related
I have configured queueconnection factory in the WebSphere admin console and using jndi lookup to use it. MQ Queue Connection Factories in WebSphere Application Server has settings. Connection pool Max size to 30 and session pool Max size to 20
These are than used in several jms:message-driven-channel-adapter or jms channels or jms:inbound-channel-adapter as part of various spring integration workflows that I have in my application.
Over a period we see that the connection count on MQ channel keeps increasing upto maximum allowed (about 1800).Once we bounce the server the count comes back to normal below 50.
Is there any setting missing ?
How can I be sure if the JMS session pools are being closed/released ?
Any help is appreciated
<jee:jndi-lookup id="queueConnectionFactory" jndi-name="$env{Queue.ConnectionFactory}" />
<si-jms:message-driven-channel-adapter
id="messageDrivenAdapter" channel="routingChannel"
container="messageListenerContainer" />
<bean id="messageListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="queueConnectionFactory" />
<property name="destination" ref="inQueue" />
<property name="transactionManager" ref="txManager" />
<property name="taskExecutor" ref="MQExecutor" />
</bean>
<si-jms:channel id="regChannel" queue="regQueue" connection-factory="queueConnectionFactory" transaction-manager="txManager" task-executor="regtaskExecutor" />
In my old days when we use Spring Integration on WebSphere we had a bean like this:
<bean id="connectionFactory" class="org.springframework.jms.connection.DelegatingConnectionFactory">
<property name="targetConnectionFactory">
<jee:jndi-lookup jndi-name="someConnectionFactory"/>
</property>
<property name="shouldStopConnections" value="true"/>
</bean>
Let's see if that can help you!
We need to persist message from a JMS Queue to a database in a transaction ensuring that JMS message is not acknowledged in case any error is thrown during DB persist.
Based on the solution provided here- Transaction handling while using message driven channel adapter & service activator
Below is the approach I arrived at
<int-jms:message-driven-channel-adapter id="jmsIn"
transaction-manager="transactionManager"
connection-factory="sConnectionFactory"
destination-name="emsQueue"
acknowledge="transacted" channel="jmsInChannel"/>
<int:channel id=" jmsInChannel " />
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:testdb" />
</bean>
<!-- Transaction manager for a datasource -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="sconnectionFactory" class="org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy">
<property name="targetConnectionFactory">
<bean class="project.TestConnectionFactory">
</bean>
</property>
<property name="synchedLocalTransactionAllowed" value="true" />
</bean>
Please confirm if the understanding is correct. Also, have the below queries
Is the TransactionAwareConnectionFactoryProxy necessary in this scenario
JMS Queue and JDBC are two separate transactional resources. Is injecting jdbc transaction manager into JMS adapter as shown in this example equivalent to using a ChainedTransactionManager(chaining JMS Transaction Mananger and JDBC transaction manager)
For TransactionAwareConnectionFactoryProxy, please, consult its JavaDocs:
* Proxy for a target CCI {#link javax.resource.cci.ConnectionFactory}, adding
* awareness of Spring-managed transactions. Similar to a transactional JNDI
* ConnectionFactory as provided by a Java EE server.
*
* <p>Data access code that should remain unaware of Spring's data access support
* can work with this proxy to seamlessly participate in Spring-managed transactions.
* Note that the transaction manager, for example the {#link CciLocalTransactionManager},
* still needs to work with underlying ConnectionFactory, <i>not</i> with this proxy.
I'm not sure how is that related to your request about JMS+DB transaction, but I think you should worry about transaction manager which supports both those transactional resources.
For this purpose Spring suggests JtaTransactionManager for XA transactions.
Or you can consider to use ChainedTransactionManager based on the Dave Syer article: http://www.javaworld.com/article/2077963/open-source-tools/distributed-transactions-in-spring--with-and-without-xa.html
UPDATE
If local transaction are fine for you, you really can go ahead with the TransactionAwareConnectionFactoryProxy and its synchedLocalTransactionAllowed to true. And, of course, use that DataSourceTransactionManager directly from the <int-jms:message-driven-channel-adapter>.
Otherwise you have to go with the ChainedTransactionManager. This solution really adds some overhead over the first TransactionAwareConnectionFactoryProxy-based one.
I am using Spring AMQP and Spring Integration. I am currently using a service-activator from integration as my handler. I have tried to get an amqp listener working as well but I never receive the message. Is using a service-activator accepted practice or should I be using a listener?
<!-- Receive Inbound messages and process them -->
<int-amqp:inbound-channel-adapter channel="sda.text.analytics.process.channel" queue-names="${sda.text.analytics.process.queue}"
connection-factory="sda.text.analytics.connectionFactory" prefetch-count="${sda.mule.prefetchCount}"
concurrent-consumers="${uima.process.threads}" task-executor="analyticsExecutor" receive-timeout="5000"/>
<int:json-to-object-transformer input-channel="sda.text.analytics.process.channel" type="com.issinc.sda.ingest.impl.IngestBean"/>
<bean id="analyticsExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="${uima.process.threads}" />
<property name="maxPoolSize" value="${uima.process.threads}" />
<property name="queueCapacity" value="${sda.mule.prefetchCount}" />
</bean>
<!-- Service that listens for an inbound messages, processes the text and sends the processed text back -->
<int:service-activator input-channel="sda.text.analytics.process.channel" output-channel="sda.text.analytics.response.channel"
ref="asyncExtractionService" method="processMessage" >
</int:service-activator>
<!--<rabbit:listener-container connection-factory="sda.connectionFactory" message-converter="jsonMessageConverter">
<rabbit:listener ref="asyncExtractionService" method="processMessage" queue-names="${sda.process.queue}" />
</rabbit:listener-container>-->
You're not really "using" the service activator at all; you are invoking the same bean/method as the service activator.
Consider using an AMQP inbound channel adapter to send messages to the service activator via the sda.process.channel.
I am new to spring integration. My requirement is that if there is a connection problem to the jms q then it should try to connect 3 times then log it and exit the process. I am not able to do it. It throws an error saying it needs the ref attribute for service:activator. But I don't have/know reference of which class to provide here. Is there any other way of doing it?
<int-jms:message-driven-channel-adapter id="msgIn" channel="toRoute" container="messageListenerContainer" />
<int:service-activator id="service" input-channel="toRoute" >
<int:request-handler-advice-chain>
<bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<property name="recoveryCallback">
<bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<constructor-arg ref=“errorChannel" />
</bean>
</property>
</bean>
</request-handler-advice-chain>
</int:service-activator>
You seem to have completely misunderstood what the framework does.
The service-activator gets a message when one is received from JMS (which implies the connection is good), and needs "something" (a reference to a bean or an expression) to invoke as a result of receiving that message.
The retry advice is to retry calling that service if it fails to process the message for some reason. It is unrelated to whatever is the source of the message (JMS in this case).
It's not clear why you are trying to use Spring Integration for something as simple as testing whether a JMS broker is available.
Perhaps if you can provide some larger context someone might be able to help.
I'm using spring-integration-kafka 1.1.0 with the following config. I don't quite understand about the streams config. When I increase this, does Spring automatically spawn more threads to handle the messages? e.g. when I have streams=2, does the correlated transformer and service-activator all run in 2 threads? I feel like missing some thread-executor configurations, but not sure how. Any hint is appreciated. Thanks.
<int:poller default="true" fixed-delay="10"/>
<int:channel id="tag.track">
</int:channel>
<int-kafka:inbound-channel-adapter id="kafkaInboundChannelAdapterForTagTrack" kafka-consumer-context-ref="consumerContextForTagTrack" auto-startup="true" channel="tag.track">
</int-kafka:inbound-channel-adapter>
<int-kafka:consumer-context id="consumerContextForTagTrack"
consumer-timeout="${kafka.consumer.timeout}" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration group-id="${kafka.consumer.group.track}" max-messages="200">
<int-kafka:topic id="tag.track" streams="2" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int:channel id="tag.track.transformed">
<int:interceptors>
<int:wire-tap channel="event.logging" />
</int:interceptors>
</int:channel>
<int:transformer id="kafkaMessageTransformerForTagTrack"
ref="kafkaMessageTransformer" input-channel="tag.track" method="transform"
output-channel="tag.track.transformed" />
<int:service-activator input-channel="tag.track.transformed" ref="tagTrackMessageHandler" method="handleTagMessage">
<int:request-handler-advice-chain>
<ref bean="userTagRetryAdvice" />
</int:request-handler-advice-chain>
</int:service-activator>
Tried message-driven-channel-adapter, but can't get it work, the following config doesn't pick up any message. Also tried the org.springframework.integration.kafka.listener.KafkaTopicOffsetManager , it complains Offset management topic cannot have more than one partition. Also, in this adapter, how to configure the consumer group?
Is there any detailed example on how to use the message-driven-channel-adapter? The instruction on the project page is pretty high level.
<int:channel id="tag.track">
<int:queue capacity="100"/>
</int:channel>
<bean id="kafkaConfiguration" class="org.springframework.integration.kafka.core.ZookeeperConfiguration">
<constructor-arg ref="zookeeperConnect"/>
</bean>
<bean id="connectionFactory" class="org.springframework.integration.kafka.core.DefaultConnectionFactory">
<constructor-arg ref="kafkaConfiguration"/>
</bean>
<bean id="decoder" class="org.springframework.integration.kafka.serializer.common.StringDecoder"/>
<int-kafka:message-driven-channel-adapter
id="adapter"
channel="tag.track"
connection-factory="connectionFactory"
key-decoder="decoder"
payload-decoder="decoder"
max-fetch="100"
topics="tag.track"
auto-startup="true"
/>
The streams property has nothing to do with Spring itself; it's simply passed to Kafka when invoking ConsumerConnector.createMessageStreams() (each topic/streams entry is passed in the map argument).
Refer to the kafka documentation.
EDIT:
When using the high-level consumer, the kafka inbound channel adapter is polled, so the threads on which the downstream integration flow runs are not related to the kafka client threads; they are managed in the poller configuration.
You could consider using the message-driven channel adapter instead.