we used to configure the poller in int:chain like below, the inboundChannel is configured with queue
<int:chain id="messageProcessChain" input-channel="inboundChannel" >
<int:poller fixed-delay="1" max-messages-per-poll="1"/>
This time, we'd like to programmatically initialize int flows, so using DSL. my test code as below:
Jms.messageDrivenChannelAdapter(inConnFactory).destination(flowProperties.getInputQueue()))
.channel(MessageChannels.queue(mongoDbChannelMessageStore,"groupId"))
// .enrichHeaders(testHeaders)
.handle(m -> {
System.out.println("test handler");
logger.info("test dsl flow:"+m.getPayload().toString());
},c -> c.poller(Pollers.fixedRate(flowProperties.getInboudFixedRate()).maxMessagesPerPoll(10)))
.get()
it cannot work with the ".enrichHeaders(testHeaders)", due to'No poller has been defined for endpoint', but neither can I configure a poller for the header richer because it has an implicit SubscribableChannel?
In such case, can only use a bridge to connect the two channel? Is there some other approaches?
Use .enrichHeaders(headers, e -> e.poller(...)).
However, you should NOT use a queue channel with a message-driven channel adapter, you will likely lose messages in the event of a failure.
To achieve concurrency, increase concurrency on the adapter.
Related
I'm writing a Spring Integration application which receives messages from an input system, transforms them and sends them to an output system. The connection to the output system is not always available. The messages from the input system can come at any moment. In case if they are coming when the output system is not available, they shouldn't be lost and send eventually when the output system is available. So I store the messages from the input system in a QueueChannel:
#Configuration
class TcInFlowConfiguration {
#Bean
fun tcInFlow(
#Qualifier(TC_MESSAGE_LISTENER) listener: MessageProducerSupport,
#Qualifier(TC_MESSAGE_CHANNEL) messageChannel: MessageChannel
): IntegrationFlow {
return IntegrationFlow
.from(listener)
.transform { msg: ByteArray -> RamsesTcFrame.deserialize(msg) }
.channel(messageChannel)
.get()
}
#Bean
#Qualifier(TC_MESSAGE_CHANNEL)
fun tcMessageChannel(): MessageChannel {
return MessageChannels.queue().get()
}
The app receives an API call to open/close the connection to the output system, so I create and remove the output integration flow programmatically via IntegrationFlowContext:
val outFlow = IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender())
.get()
integrationFlowContext.registration(outFlow).register()
When the messages are polled from the queue to be processed by the outFlow, the default Spring task executor is used (I see "scheduling-1" as a thread name in logs). The problem is that I have multiple independent integration flows in the app with the multiple queue channels, so they all got mixed up by being processed by the same task executor. What I want is to process each flow in its own dedicated thread, so the flows won't block each other. How can I achieve this?
Spring Boot v3.0.2, Spring v6.0.4
I tried setting a task scheduler for my QueueChannel:
val queueChannel = MessageChannels.queue().get()
queueChannel.setTaskScheduler(taskScheduler)
It didn't have any effect, taskScheduler seems to be simply not used by QueueChannel implementation.
I tried using ExecutorChannel instead of QueueChannel which supports setting a custom Executor. Unfortunately, ExecutorChannel doesn't buffer messages in memory, so if there are no subscribers to the channel the messages are lost.
Finally, I tried defining a poller in the outFlow to poll the messages from the QueueChannel:
IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender()) { e -> e.poller(Pollers.fixedDelay(10).taskExecutor(taskExecutor)) }
.get()
This didn't work either. After the connection to the output system is closed and the outFlow is removed, the intermediate channel created by the poller remains in Spring context. So when the new message arrives in QueueChannel it goes to that intermediate channel which is a subscribable channel without subscribers, so the message is lost.
That's correct. The QueueChannel is just a buffer for messages. It really only matters how you consume messages from there. And the common way is to use a PollingConsumer like you do with that e.poller(). It is also correct to configure a taskExecutor() if you don't like to have your messages to be consumed by a TaskScheduler's thread. Not sure what you talk about an "intermediate channel" since it doesn't look like you have one declared in your outFlow. You have that .from(TC_MESSAGE_CHANNEL) and then immediately a handle() with a proper poller. So, no any extra channel in between or after. Unless you do something else in your createMessageSender().
I would suggest do not have a dynamic flow, but rather singleton one for that output system. The QueueChannel can be configured for a persistent message store and poller can be transactional. So, if no connection to a target system, the transaction is going to be rolled back and message remains in the store: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store.
You also can just stop() a polling consumer for that handle() when no connection, so no messages are polled from the queue at that moment.
I am trying to test a Spring Integration flow that starts off from a message-driven-channel-adapter configured as:
<int-jms:message-driven-channel-adapter id="myAdapter" ... />
My test goes like:
#SpringJUnitConfig(locations = {"my-app-context.xml"})
#SpringIntegrationTest(noAutoStartup = {"myAdapter"})
public class MyIntegrationFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Test
public void myTest() {
...
MessageSource<String> messageSource = () -> new GenericMessage<>("foo");
mockIntegrationContext.substituteMessageSourceFor("myAdapter", messageSource);
...
}
}
I am however getting the following error:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'myAdapter' is expected to be of type 'org.springframework.integration.endpoint.SourcePollingChannelAdapter' but was actually of type 'org.springframework.integration.jms.JmsMessageDrivenEndpoint'
How should one specify an alternate source for the channel adapter for testing using the MockIntegrationContext, or by some other method?
The Message Driver Channel Adapter is really not a Source Polling Channel Adapter. So, the substituteMessageSourceFor() is indeed cannot be used for that type of components, which, essentially is a MessageProducerSupport implementation, not a SourcePollingChannelAdapter for a MessageSource.
The difference exists because not all protocols provides a listener-like hooks to spawn some self-managed task to subscribe to. The good example is JDBC, which is only passive system expecting requests. Therefore a polling channel adapter with a JdbcPollingChannelAdapter (which is a MessageSource) implementation must be used to interact with DB in event-driven manner.
Other systems (like JMS in your case) provides some listener (or consumer) API for what we can spawn a while task (see MessageListenerContainer in spring-jms) and let its MessageProducerSupport to emit messages to the channel.
Therefore you need to distinguish for yourself with what type of component you interact before choosing a testing strategy.
Since there is no extra layer in case of message-driver channel adapter, but rather some specific, self-managed MessageProducerSupport impl, we not only provide a particular mocking API, but even don't require to know anything else, but just standard unit testing feature and a message channel this endpoint is producing in the configuration.
So, the solution for you is something like:
#SpringIntegrationTest(noAutoStartup = {"myAdapter"}) - that's fully correct in your code: we really have to stop the real channel adapter to not pollute our testing environment.
You just need to inject into your test class a MessageChannel that id="myAdapter" is producing to. In your test code you just build a Message and send it into this channel. No need to worry about a MockIntegrationContext at all.
A very simple example with Spring Integration 5.3.2: a single integration flow, registered using the spring-integration DSL with the IntegrationFlowContext:
`
MessageChannel startChannel = MessageChannels.direct("channel").get();
StandardIntegrationFlow flow = IntegrationFlows.from(startChannel)
.log().get();
flowContext.registration(flow)
.autoStartup(false)
.register();
flow.start();
startChannel.send(MessageBuilder.withPayload("test").build());`
Yields org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel.
Why? Adding a single .transform(e->e) (for example) resolves the problem, but obviously is a strange workaround.
That’s definitely an expected behavior. You declare a MessageChannel and send a message into it, but there are no subscribers declared for that channel . Therefore you really see a difference when you add that transform(). You should decide for yourself what is your logic and what you would like to do when there is the message in that channel. You may consider to use other channel types, but still consider to process messages somehow.
I need to throttle the movement of messages between some JMS (activeMQ) queues to ensure I dont overrun an external service used during message processing.
I have done this in the past with Camel but given that this project is otherwise entirely Spring based, I figured I'd give spring-integration a whirl.
I am happy to see that in 5.0.7 the Java DSL is in core and would really like to use it instead of xml.
But...I cant seem to find good/current docs for using the DSL to do even simple things like create the input and output messageChannels for JMS.
Could anyone point me to any current example of using the java DSL to create channels that I can use to consume and produce messages with...and then later use in a bridge with some throttling applied?
Well, looks like our JMS chapter in the Reference Manual leaks of the Java DSL samples, similar to what we have so far with AMQP, for example:
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_2
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_4
I believe we can add similar paragraphs into the JMS chapter as well. Please, raise a JIRA on the matter and we will address it soon. Meanwhile I suggest you to open a org.springframework.integration.jms.dsl.Jms factory for appropriate builder to use.
On the other hand I can suggest you to take a look into the existing test-case of some possible configurations: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jms/src/test/java/org/springframework/integration/jms/dsl/JmsTests.java
For example to read from the queue you need a configuration like this:
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory(), DefaultMessageListenerContainer.class)
.outputChannel(jmsMessageDrivenInputChannel())
.destination("jmsMessageDriven")
.configureListenerContainer(c -> c.clientId("foo")))
.<String, String>transform(String::toLowerCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}
To send to the JMS you need something like this:
#Bean
public IntegrationFlow jmsOutboundFlow() {
return f -> f
.handle(Jms.outboundAdapter(jmsConnectionFactory())
.destinationExpression("headers." + SimpMessageHeaderAccessor.DESTINATION_HEADER)
.configureJmsTemplate(t -> t.id("jmsOutboundFlowTemplate")));
}
1) Server sends a message to client.
2) Inbound channel adapter is configured to wait for "MANUAL" acknowledge mode operation from consumer
3) "TaskBundlereceiver" bean is implementing "ChannelAwareMessageListener" and in the implementation method, I am performing message acknowledgement.
I don't see "TaskBundlereceiver" getting executed. Am I missing something ?
Below is the configuration details of the steps that I have explained.
Appreciate your inputs.
#Override
public void onMessage(org.springframework.amqp.core.Message message, Channel channel) throws Exception
{
logger.debug("In onMessage method of the channel aware listener. message =["+message.getBody().toString()+"]");
channel.basicAck(message.getMessageProperties().getDeliveryTag(), true);
}
XML Configuration :
<!-- Channel that receives the task bundle from the server for execution -->
<int:channel id="fromKServerChannel"/>
<int-amqp:inbound-channel-adapter id="taskBundleReceiverAdapter"
channel="fromKServerChannel"
error-channel="taskBundleErrorChannel"
acknowledge-mode="MANUAL"
expose-listener-channel="true"
queue-names="kanga_task_queue"
connection-factory="connectionFactory"
concurrent-consumers="20"/>
<int:chain input-channel="fromKServerChannel" output-channel="nullChannel">
<int:service-activator ref="taskBundleReceiver" method="onMessage"/>
<int:service-activator ref="taskBundleExecutor" method="executeBundle"/>
</int:chain>
It doesn't work that way; the listener is the adapter, not the service invoked via the service-activator. The adapter currently does not support passing the channel to the client for manual acks. The expose-listener-channel attribute is for use when using transactions, so a down-stack rabbit template can participate in the transaction.
Why do you want MANUAL acks? AUTO (default) means the ack will be done automatically by the container when the thread returns normally; if your service throws an exception, the message will be nacked.
So, that's how to control the ack.
If you really want to use MANUAL acks, you'll have to use a <rabbit:listener-container/> to invoke your taskBundleReceiver directly. It could then send a message to the executor using a messaging gateway.