I am trying to test a Spring Integration flow that starts off from a message-driven-channel-adapter configured as:
<int-jms:message-driven-channel-adapter id="myAdapter" ... />
My test goes like:
#SpringJUnitConfig(locations = {"my-app-context.xml"})
#SpringIntegrationTest(noAutoStartup = {"myAdapter"})
public class MyIntegrationFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Test
public void myTest() {
...
MessageSource<String> messageSource = () -> new GenericMessage<>("foo");
mockIntegrationContext.substituteMessageSourceFor("myAdapter", messageSource);
...
}
}
I am however getting the following error:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'myAdapter' is expected to be of type 'org.springframework.integration.endpoint.SourcePollingChannelAdapter' but was actually of type 'org.springframework.integration.jms.JmsMessageDrivenEndpoint'
How should one specify an alternate source for the channel adapter for testing using the MockIntegrationContext, or by some other method?
The Message Driver Channel Adapter is really not a Source Polling Channel Adapter. So, the substituteMessageSourceFor() is indeed cannot be used for that type of components, which, essentially is a MessageProducerSupport implementation, not a SourcePollingChannelAdapter for a MessageSource.
The difference exists because not all protocols provides a listener-like hooks to spawn some self-managed task to subscribe to. The good example is JDBC, which is only passive system expecting requests. Therefore a polling channel adapter with a JdbcPollingChannelAdapter (which is a MessageSource) implementation must be used to interact with DB in event-driven manner.
Other systems (like JMS in your case) provides some listener (or consumer) API for what we can spawn a while task (see MessageListenerContainer in spring-jms) and let its MessageProducerSupport to emit messages to the channel.
Therefore you need to distinguish for yourself with what type of component you interact before choosing a testing strategy.
Since there is no extra layer in case of message-driver channel adapter, but rather some specific, self-managed MessageProducerSupport impl, we not only provide a particular mocking API, but even don't require to know anything else, but just standard unit testing feature and a message channel this endpoint is producing in the configuration.
So, the solution for you is something like:
#SpringIntegrationTest(noAutoStartup = {"myAdapter"}) - that's fully correct in your code: we really have to stop the real channel adapter to not pollute our testing environment.
You just need to inject into your test class a MessageChannel that id="myAdapter" is producing to. In your test code you just build a Message and send it into this channel. No need to worry about a MockIntegrationContext at all.
Related
I'm writing a Spring Integration application which receives messages from an input system, transforms them and sends them to an output system. The connection to the output system is not always available. The messages from the input system can come at any moment. In case if they are coming when the output system is not available, they shouldn't be lost and send eventually when the output system is available. So I store the messages from the input system in a QueueChannel:
#Configuration
class TcInFlowConfiguration {
#Bean
fun tcInFlow(
#Qualifier(TC_MESSAGE_LISTENER) listener: MessageProducerSupport,
#Qualifier(TC_MESSAGE_CHANNEL) messageChannel: MessageChannel
): IntegrationFlow {
return IntegrationFlow
.from(listener)
.transform { msg: ByteArray -> RamsesTcFrame.deserialize(msg) }
.channel(messageChannel)
.get()
}
#Bean
#Qualifier(TC_MESSAGE_CHANNEL)
fun tcMessageChannel(): MessageChannel {
return MessageChannels.queue().get()
}
The app receives an API call to open/close the connection to the output system, so I create and remove the output integration flow programmatically via IntegrationFlowContext:
val outFlow = IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender())
.get()
integrationFlowContext.registration(outFlow).register()
When the messages are polled from the queue to be processed by the outFlow, the default Spring task executor is used (I see "scheduling-1" as a thread name in logs). The problem is that I have multiple independent integration flows in the app with the multiple queue channels, so they all got mixed up by being processed by the same task executor. What I want is to process each flow in its own dedicated thread, so the flows won't block each other. How can I achieve this?
Spring Boot v3.0.2, Spring v6.0.4
I tried setting a task scheduler for my QueueChannel:
val queueChannel = MessageChannels.queue().get()
queueChannel.setTaskScheduler(taskScheduler)
It didn't have any effect, taskScheduler seems to be simply not used by QueueChannel implementation.
I tried using ExecutorChannel instead of QueueChannel which supports setting a custom Executor. Unfortunately, ExecutorChannel doesn't buffer messages in memory, so if there are no subscribers to the channel the messages are lost.
Finally, I tried defining a poller in the outFlow to poll the messages from the QueueChannel:
IntegrationFlow
.from(TC_MESSAGE_CHANNEL)
.handle(createMessageSender()) { e -> e.poller(Pollers.fixedDelay(10).taskExecutor(taskExecutor)) }
.get()
This didn't work either. After the connection to the output system is closed and the outFlow is removed, the intermediate channel created by the poller remains in Spring context. So when the new message arrives in QueueChannel it goes to that intermediate channel which is a subscribable channel without subscribers, so the message is lost.
That's correct. The QueueChannel is just a buffer for messages. It really only matters how you consume messages from there. And the common way is to use a PollingConsumer like you do with that e.poller(). It is also correct to configure a taskExecutor() if you don't like to have your messages to be consumed by a TaskScheduler's thread. Not sure what you talk about an "intermediate channel" since it doesn't look like you have one declared in your outFlow. You have that .from(TC_MESSAGE_CHANNEL) and then immediately a handle() with a proper poller. So, no any extra channel in between or after. Unless you do something else in your createMessageSender().
I would suggest do not have a dynamic flow, but rather singleton one for that output system. The QueueChannel can be configured for a persistent message store and poller can be transactional. So, if no connection to a target system, the transaction is going to be rolled back and message remains in the store: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store.
You also can just stop() a polling consumer for that handle() when no connection, so no messages are polled from the queue at that moment.
I am interested in using Spring Integration to fetch files from various endpoints (FTP servers, email inboxes, S3, etc.) and load them into my system (essentially, ETL).
There are times when I will want these channels active and running, and other times when I will want them paused/stopped. Meaning, even if there are files available at the source, I do not want the channel consuming the data and doing anything with it.
Is a control bus an appropriate start/stop solution here:
#Bean
public IntegrationFlow controlBusFlow() {
return IntegrationFlow.from("controlBus")
.controlBus()
.get();
}
If so, how would I stop/restart a specific channel (route between an S3 bucket and the rest of my system) using the Java DSL/API? And if not, then what is the recommended practice/EIP to apply here?
Yes, the Control Bus is exactly a pattern and tool designed for your goal: https://www.enterpriseintegrationpatterns.com/ControlBus.html.
Yes, to use it you need to send messages to input channel of that control bus endpoint. The payload of message to sent must be a command to do some control activity for endpoint. Typically we call start and stop.
So, let's imagine you have an S3 source polling channel adapter:
#Bean
IntegrationFlow s3Flow(S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource) {
return IntegrationFlow.from(s3InboundFileSynchronizingMessageSource, e -> e.id("myS3SourceEndpoint"))
...;
}
So, to stop that myS3SourceEndpoint via Control Bus, you need to send a message with a payload #myS3SourceEndpoint.stop().
Pay attention that we don't talk here about message channels neither message sources. The active components in the flow are really endpoints.
UPDATE
The Control Bus component utilizes a Command Message pattern. So, you need to build a respective message and send it to the input channel of that control bus endpoint. Something like this is OK:
#Autowired
MessageChannel controlBus;
...
this.controlBus.send(new GenericMessage<>("#myS3SourceEndpoint.stop()"));
You can use a MessagingTemplate.convertAndSend() if you don't like creating message yourself. Or you also can expose high-lever API via #MessagingGateway interface.
Everything you can find in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/index.html
A very simple example with Spring Integration 5.3.2: a single integration flow, registered using the spring-integration DSL with the IntegrationFlowContext:
`
MessageChannel startChannel = MessageChannels.direct("channel").get();
StandardIntegrationFlow flow = IntegrationFlows.from(startChannel)
.log().get();
flowContext.registration(flow)
.autoStartup(false)
.register();
flow.start();
startChannel.send(MessageBuilder.withPayload("test").build());`
Yields org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel.
Why? Adding a single .transform(e->e) (for example) resolves the problem, but obviously is a strange workaround.
That’s definitely an expected behavior. You declare a MessageChannel and send a message into it, but there are no subscribers declared for that channel . Therefore you really see a difference when you add that transform(). You should decide for yourself what is your logic and what you would like to do when there is the message in that channel. You may consider to use other channel types, but still consider to process messages somehow.
I'm trying to learn how to handle MQTT Messages in Spring-Integration.
Have created a converter, that subscribes with a single MqttPahoMessageDrivenChannelAdapter per MQTT Topic for consuming and converting the messages.
The problem is our data provider is planning to "speed-up" publishing messages on his side. So instead of having a few(<=10) topics each of which has messages with about 150 fields it is planned to publish each of those fields to the separate MQTT topic.
This means my converter would have to consume ca. 1000 mqtt topics, but I do not know whether:
Is spring-integration still a good choice for it. Cause afaik. the mentioned adapter uses the PAHO MqttClient that will consume the messages from all of the topics it is subscribed to in one single thread and creating 1000 instances of those adapters is an overkill.
If we stick further to spring-integration and use the provided components, would it be a good idea to create a single inbound adapter for all of the fields, that previously were in messages of one topic but moving the conversion away from the adapter bean to a separate bean ( that does the conversion) connected with an executer-channel to the adapter and thus executing the conversion of those fields on some threadpool in parallel.
Thanks in advance for your answers!
I think your idea makes sense.
For that purpose you need to implement a passthrough MqttMessageConverter and provide an MqttMessage as a payload and topic as a header:
public class PassThroughMqttMessageConverter implements MqttMessageConverter {
#Override
public Message<?> toMessage(String topic, MqttMessage mqttMessage) {
return MessageBuilder.withPayload(mqttMessage)
.setHeader(MqttHeaders.RECEIVED_TOPIC, topic)
.build();
}
#Override
public Object fromMessage(Message<?> message, Class<?> targetClass) {
return null;
}
#Override
public Message<?> toMessage(Object payload, MessageHeaders headers) {
return null;
}
}
So, you really will be able to perform a target conversion downstream, after a mentioned ExecutorChannel in the custom transformer.
You also may consider to implement a custom MqttPahoClientFactory (an extension of the DefaultMqttPahoClientFactory may work as well) and provide a custom ScheduledExecutorService to be injected into the MqttClient you are going create in the getClientInstance().
I did one simple DSL which retrieves the data from database and doing simple conversion in the service activator.
#Bean
public IntegrationFlow mainFlow() {
return IntegrationFlows.from("userChannel")
.channel("queryChannel")
.handle("sampleConvertor","convertUser")
.get();
queryChannel is a jdbc outbound gateway and sampleConverter is the service Activator.
<int-jdbc:outbound-gateway query="select * from employee where employee_id=:payload"
request-channel="queryChannel" data-source="dataSource"/>
The issue is after retrieving the data from database, the flow is not going to serviceActivator and it simply returns back the database response.
In xml configuration, I used to invoke gateway inside the chain like below.
<int:gateway id="query.gateway" request-channel="queryChannel"/>
Please suggest what I am doing wrong here. Thanks in advance.
That's a bit unusual to combine Java DSL and XML configuration, but they still work together.
Your problem I think that you are missing the fact that your queryChannel has two subscriber at runtime, not a chain of call.
The first one is <int-jdbc:outbound-gateway> and the second is that .handle("sampleConvertor","convertUser"). Right, when you declare a channel in the IntegrationFlow, the next EIP-method produces a subscriber for this channel. At the same time when you use a channel like request-channel or input-channel in the XML configuration that brings a subscriber as well.
So, you have two subscriber on the DirectChannel with the RoundRobinLoadBalancingStrategy and therefore only one of them will handle a message and if it is a request-replly component, like that <int-jdbc:outbound-gateway> it will produce a message into the output-channel or to the replyChannel in the headers. In your case the story is exactly about a replyChannel and therefore you don't go to the .handle("sampleConvertor","convertUser") because it's not the next in the chain, but just a parallel universe by the round-robin algorithm.
If you really would like to reach that .handle("sampleConvertor","convertUser") after calling the <int-jdbc:outbound-gateway>, you should consider to use .gateway("queryChannel") instead of that .channel().