How to make the call to service activator transactional after split - spring-integration

I am using following to define my integration flow:
#Bean
public IntegrationFlow pollingFlow(MessageSource<Object> jdbcMessageSource) {
return IntegrationFlows.from(jdbcMessageSource,
c -> c.poller(Pollers.fixedRate(250, TimeUnit.MILLISECONDS)
.maxMessagesPerPoll(1)
.transactional()))
.split()
.channel(taskSourceChannel())
.get();
}
I would like to make call to service activator that reads from taskSourceChannel as transactional. Also, I want to use following with my transaction.
#Bean
public TransactionSynchronizationFactory transactionSynchronizationFactory() {
ExpressionEvaluatingTransactionSynchronizationProcessor syncProcessor
= new ExpressionEvaluatingTransactionSynchronizationProcessor();
syncProcessor.setAfterCommitChannel(successChannel());
syncProcessor.setAfterRollbackChannel(failureChannel());
return new DefaultTransactionSynchronizationFactory(syncProcessor);
}
The taskSourceChannel is an executor channel.
#Bean
public MessageChannel taskSourceChannel() {
return new ExecutorChannel(executor());
}
How can I add transaction support after split while using TransactionSynchronizationFactory. I don't want to make polling transacational. The only solution I can think of is putting transactional on activator but that won't solve my problem. I would like to make it applicable to any service activator uses this channel.

You question is not so clear, but you definitely need to consider to add transaction into the service activator. Although you don't show what is the subscriber for that taskSourceChannel, but you need to think do not have several subscribers on it.
Nevertheless I think your point is to apply TX into the service activator on this taskSourceChannel and everything after that one.
For this purpose Spring Integration provides a TransactionHandleMessageAdvice. See more info the Reference Manual: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#tx-handle-message-advice.
The TransactionSynchronizationFactory is only used from the AbstractPollingEndpoint implementations. However you can still utilize it in your transactional context relying on the TransactionSynchronizationManager.registerSynchronization().

Related

Spring integration aws (sqs) to trigger spring integration flow

I have to listen a queue using spring integration flow and intgeration sqs. Once message is received from queue it should trigger a integration flow. Below is the things which I am trying but everythings fine in but afater receiving test it is not triggering any Integration flow. Please let me know where I am doing wrong:
UPDATED as per comment from Artem
Adapter for SQS.
#Bean
public MessageProducerSupport sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter adapter = new SqsMessageDrivenChannelAdapter(amazonSQSAsync, "Main");
adapter.setOutputChannel(inputChannel());
adapter.setAutoStartup(true);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.NEVER);
adapter.setMaxNumberOfMessages(10);
adapter.setVisibilityTimeout(5);
return adapter;
}
Queue configured:
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
Now the main integration flow trigger point:
#Bean
public IntegrationFlow inbound() {
return IntegrationFlows.from("inputChannel").transform(i -> "TEST_FLOW").get();
}
}
Appreciate any type of help.
The sqsMessageDrivenChannelAdapter() must be declared as a #Bean
The inbound() must be declared as a #Bean
This one fully does not make sense IntegrationFlows.from(MessageChannels.queue()). What is the point to start the flow from anonymous channel? Who and how is going to produce messages to that channel?
Make yourself familiar with different channels: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
Pay attention that QueueChannel must be consumed via polling endpoint.
Right, there is a default poller auto-configured by Spring Boot, but it is based on a single thread in the TaskScheduler and has a polling period as 10 millis.
I wouldn't recommend to hand off SQS messages to the QueueChannel: when consumer fails, you lose the data. It is better to process those messages in the consumer thread.
Otherwise your intention is not clear in the provided code.
Can you, please, share with us what error you get or anything else?
You also can turn on DEBUG logging level for org.springframework.integration to see how your messages are processed.

How to test message-driven-channel-adapter with MockIntegrationContext

I am trying to test a Spring Integration flow that starts off from a message-driven-channel-adapter configured as:
<int-jms:message-driven-channel-adapter id="myAdapter" ... />
My test goes like:
#SpringJUnitConfig(locations = {"my-app-context.xml"})
#SpringIntegrationTest(noAutoStartup = {"myAdapter"})
public class MyIntegrationFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Test
public void myTest() {
...
MessageSource<String> messageSource = () -> new GenericMessage<>("foo");
mockIntegrationContext.substituteMessageSourceFor("myAdapter", messageSource);
...
}
}
I am however getting the following error:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'myAdapter' is expected to be of type 'org.springframework.integration.endpoint.SourcePollingChannelAdapter' but was actually of type 'org.springframework.integration.jms.JmsMessageDrivenEndpoint'
How should one specify an alternate source for the channel adapter for testing using the MockIntegrationContext, or by some other method?
The Message Driver Channel Adapter is really not a Source Polling Channel Adapter. So, the substituteMessageSourceFor() is indeed cannot be used for that type of components, which, essentially is a MessageProducerSupport implementation, not a SourcePollingChannelAdapter for a MessageSource.
The difference exists because not all protocols provides a listener-like hooks to spawn some self-managed task to subscribe to. The good example is JDBC, which is only passive system expecting requests. Therefore a polling channel adapter with a JdbcPollingChannelAdapter (which is a MessageSource) implementation must be used to interact with DB in event-driven manner.
Other systems (like JMS in your case) provides some listener (or consumer) API for what we can spawn a while task (see MessageListenerContainer in spring-jms) and let its MessageProducerSupport to emit messages to the channel.
Therefore you need to distinguish for yourself with what type of component you interact before choosing a testing strategy.
Since there is no extra layer in case of message-driver channel adapter, but rather some specific, self-managed MessageProducerSupport impl, we not only provide a particular mocking API, but even don't require to know anything else, but just standard unit testing feature and a message channel this endpoint is producing in the configuration.
So, the solution for you is something like:
#SpringIntegrationTest(noAutoStartup = {"myAdapter"}) - that's fully correct in your code: we really have to stop the real channel adapter to not pollute our testing environment.
You just need to inject into your test class a MessageChannel that id="myAdapter" is producing to. In your test code you just build a Message and send it into this channel. No need to worry about a MockIntegrationContext at all.

SI subscription to multiple mqtt topics

I'm trying to learn how to handle MQTT Messages in Spring-Integration.
Have created a converter, that subscribes with a single MqttPahoMessageDrivenChannelAdapter per MQTT Topic for consuming and converting the messages.
The problem is our data provider is planning to "speed-up" publishing messages on his side. So instead of having a few(<=10) topics each of which has messages with about 150 fields it is planned to publish each of those fields to the separate MQTT topic.
This means my converter would have to consume ca. 1000 mqtt topics, but I do not know whether:
Is spring-integration still a good choice for it. Cause afaik. the mentioned adapter uses the PAHO MqttClient that will consume the messages from all of the topics it is subscribed to in one single thread and creating 1000 instances of those adapters is an overkill.
If we stick further to spring-integration and use the provided components, would it be a good idea to create a single inbound adapter for all of the fields, that previously were in messages of one topic but moving the conversion away from the adapter bean to a separate bean ( that does the conversion) connected with an executer-channel to the adapter and thus executing the conversion of those fields on some threadpool in parallel.
Thanks in advance for your answers!
I think your idea makes sense.
For that purpose you need to implement a passthrough MqttMessageConverter and provide an MqttMessage as a payload and topic as a header:
public class PassThroughMqttMessageConverter implements MqttMessageConverter {
#Override
public Message<?> toMessage(String topic, MqttMessage mqttMessage) {
return MessageBuilder.withPayload(mqttMessage)
.setHeader(MqttHeaders.RECEIVED_TOPIC, topic)
.build();
}
#Override
public Object fromMessage(Message<?> message, Class<?> targetClass) {
return null;
}
#Override
public Message<?> toMessage(Object payload, MessageHeaders headers) {
return null;
}
}
So, you really will be able to perform a target conversion downstream, after a mentioned ExecutorChannel in the custom transformer.
You also may consider to implement a custom MqttPahoClientFactory (an extension of the DefaultMqttPahoClientFactory may work as well) and provide a custom ScheduledExecutorService to be injected into the MqttClient you are going create in the getClientInstance().

Spring Integration 5.07 Using Java DSL to bridge between two JMS queues

I need to throttle the movement of messages between some JMS (activeMQ) queues to ensure I dont overrun an external service used during message processing.
I have done this in the past with Camel but given that this project is otherwise entirely Spring based, I figured I'd give spring-integration a whirl.
I am happy to see that in 5.0.7 the Java DSL is in core and would really like to use it instead of xml.
But...I cant seem to find good/current docs for using the DSL to do even simple things like create the input and output messageChannels for JMS.
Could anyone point me to any current example of using the java DSL to create channels that I can use to consume and produce messages with...and then later use in a bridge with some throttling applied?
Well, looks like our JMS chapter in the Reference Manual leaks of the Java DSL samples, similar to what we have so far with AMQP, for example:
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_2
https://docs.spring.io/spring-integration/docs/5.0.7.RELEASE/reference/html/amqp.html#_configuring_with_the_java_dsl_4
I believe we can add similar paragraphs into the JMS chapter as well. Please, raise a JIRA on the matter and we will address it soon. Meanwhile I suggest you to open a org.springframework.integration.jms.dsl.Jms factory for appropriate builder to use.
On the other hand I can suggest you to take a look into the existing test-case of some possible configurations: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-jms/src/test/java/org/springframework/integration/jms/dsl/JmsTests.java
For example to read from the queue you need a configuration like this:
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory(), DefaultMessageListenerContainer.class)
.outputChannel(jmsMessageDrivenInputChannel())
.destination("jmsMessageDriven")
.configureListenerContainer(c -> c.clientId("foo")))
.<String, String>transform(String::toLowerCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}
To send to the JMS you need something like this:
#Bean
public IntegrationFlow jmsOutboundFlow() {
return f -> f
.handle(Jms.outboundAdapter(jmsConnectionFactory())
.destinationExpression("headers." + SimpMessageHeaderAccessor.DESTINATION_HEADER)
.configureJmsTemplate(t -> t.id("jmsOutboundFlowTemplate")));
}

Howto execute a success action tied to inbound flow after message processing via DirectChannel

This question is more of a design question than a real problem. Given following basic flow:
#Bean
public DirectChannel getFileToSftpChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow sftpOutboundFlow() {
return IntegrationFlows.from(getFileToSftpChannel())
.handle(Sftp.outboundAdapter(this.sftpSessionFactory)
.useTemporaryFileName(false)
.remoteDirectory("test")).get();
}
#Bean
public IntegrationFlow filePollingInboundFlow() {
return from(s -> s.file(new File("path")).patternFilter("*.ext"),
e -> e.poller(fixedDelay(60, SECONDS).channel(getFileToSftpChannel()).get();
}
There is an inbound file polling flow which publishes messages via a DirectChannel to an outbound SFTP flow uploading the file.
After the entire flow finishes, I want to execute a "success" action: move the original file (locally) to an archive folder.
Using the DirectChannel, I understand that the upload will happen in the same thread as the file polling.
In other words, the file poller blocks untill the upload completes (or an error message is returned which is then pushed to the error channel).
Knowing this, I want to place the 'success' action (= moving the original file) on the inbound flow. Things I already know about and don't want to use:
Another 'handle' on the sftpOutbound. Reason: moving the file is tied to the inboud flow not the outbound flow. For ex. if I would introduce another, 2nd, producer later on (eg. a JMS inbound flow) publishing to the same channel, there would be no 'file' to be moved.
Adding an interceptor on the DirectChannel and use the 'afterSendCompletion'. Reason: same as above, I want to logic to be tied to the inbound flow
Add transaction semantics on the inbound flow and react on 'commit'. Reason: as all of this is non transactional (file system/SFTP based) I want to avoid using this.
Another thing I tried was adding an 'handle' on the inbound flow. However, I learned as the inbound flow has no real 'reply', the handle is executed before the message is sent, so this doesn't work as the move has to be executed after successful processing of the message.
Question in short: what is the standard way of executing an action supplied by the producer (=inbound flow) after the message was successfully processed by a consumer (=outbound flow) via the DirectChannel?
Well, the standard way to do something similar is transaction and that's why we some time ago introduced the PseudoTransactionManager and the XML sample for similar task looks like:
<int-file:inbound-channel-adapter id="realTx" channel="txInput" auto-startup="false"
directory="${java.io.tmpdir}/si-test2">
<int:poller fixed-rate="500">
<int:transactional synchronization-factory="syncFactory"/>
</int:poller>
</int-file:inbound-channel-adapter>
<bean id="transactionManager" class="org.springframework.integration.transaction.PseudoTransactionManager"/>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.delete()"/>
</int:transaction-synchronization-factory>
As you see we remove the file in the end of transaction which is caused really after your move to SFTP.
I'd say it is the best way to be tied with only the producer.
Another way is to introduce one more channel before getFileToSftpChannel() and apply the ChannelInterceptor.afterSendCompletion which will be invoked in the end too, by the same single-thread reason. With this approach you should just bridge all your producers with their specific DirectChannels to that single getFileToSftpChannel() for the SFTP adapter.
So, it's up to you what to choose. You have good argument from the architectural perspective to divide the logic by the responsibility levels, but as you see there is no so much choice...
Any other ideas are welcome!
You can try something like the following
#Bean
public DirectChannel getFileToSftpChannel() {
DirectChannel directChannel = new DirectChannel();
directChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void afterSendCompletion(final Message<?> message,
final MessageChannel channel, final boolean sent, final Exception ex) {
if (ex == null) {
new Archiver().archive((File) message.getPayload());
}
}
});
return directChannel;
}

Resources