I am using spring integration TcpInboundGateway in my app that receives requests(packets) from remote clients over TCP channel.The configuration as follows:
#Bean
ByteArrayCrLfSerializer ser()
{
return new ByteArrayCrLfSerializer();
}
#Bean
TcpNetServerConnectionFactory cf(){
TcpNetServerConnectionFactory connectionFactory=new TcpNetServerConnectionFactory(45456);
connectionFactory.setSingleUse(true);
connectionFactory.setSerializer(ser());
connectionFactory.setDeserializer(ser());
return connectionFactory;
}
#Bean
TcpInboundGateway tcpGate(){
TcpInboundGateway gateway=new TcpInboundGateway();
gateway.setConnectionFactory(cf());
gateway.setRequestChannel(requestChannel());
gateway.setRequestTimeout(TIMEOUT);
return gateway;
}
#Bean
public MessageChannel requestChannel(){
return new DirectChannel();
}
As per my knowledge it supports multithreading. But when lot of requests comes at same time apllication stop working with following error message.
2016-06-13 06:12:52.330 INFO 1431 --- [pool-2-thread-1] .s.i.i.t.c.TcpNetServerConnectionFactory : Server Socket closed
Is there any any configuration missing to support multithreading to handle large number of requests concurrently.
Not enough StackTrace, though... Although that may be as a reason of that gateway.setRequestTimeout(TIMEOUT); when your downstream process can't catch up all the incoming messages. Consider to make it infinite.
Or take a look to your services and try to figure out why they don't work in multithreading properly.
You might want to use Non-Blocking I/O by means of TcpNioServerConnectionFactory instead of TcpNetServerConnectionFactory.
Some docs: https://docs.spring.io/spring-integration/docs/5.2.1.RELEASE/reference/html/ip.html#note-nio
Related
I have to listen a queue using spring integration flow and intgeration sqs. Once message is received from queue it should trigger a integration flow. Below is the things which I am trying but everythings fine in but afater receiving test it is not triggering any Integration flow. Please let me know where I am doing wrong:
UPDATED as per comment from Artem
Adapter for SQS.
#Bean
public MessageProducerSupport sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter adapter = new SqsMessageDrivenChannelAdapter(amazonSQSAsync, "Main");
adapter.setOutputChannel(inputChannel());
adapter.setAutoStartup(true);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.NEVER);
adapter.setMaxNumberOfMessages(10);
adapter.setVisibilityTimeout(5);
return adapter;
}
Queue configured:
#Bean
public MessageChannel inputChannel() {
return new DirectChannel();
}
Now the main integration flow trigger point:
#Bean
public IntegrationFlow inbound() {
return IntegrationFlows.from("inputChannel").transform(i -> "TEST_FLOW").get();
}
}
Appreciate any type of help.
The sqsMessageDrivenChannelAdapter() must be declared as a #Bean
The inbound() must be declared as a #Bean
This one fully does not make sense IntegrationFlows.from(MessageChannels.queue()). What is the point to start the flow from anonymous channel? Who and how is going to produce messages to that channel?
Make yourself familiar with different channels: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
Pay attention that QueueChannel must be consumed via polling endpoint.
Right, there is a default poller auto-configured by Spring Boot, but it is based on a single thread in the TaskScheduler and has a polling period as 10 millis.
I wouldn't recommend to hand off SQS messages to the QueueChannel: when consumer fails, you lose the data. It is better to process those messages in the consumer thread.
Otherwise your intention is not clear in the provided code.
Can you, please, share with us what error you get or anything else?
You also can turn on DEBUG logging level for org.springframework.integration to see how your messages are processed.
Now I accept an http request to handle time-consuming tasks. So i want to reply the requester first and then use other flow async to handler it.
I try to use CompletableFuture.runAsync in a handler() method and then use MessageChannel.send to do . But I think there should be a more elegant way
#Bean
public IntegrationFlow testFlow(){
return IntegrationFlows.from("requestChannel")//accept an request
.handle(handlerSomeThing())//do something
.handle()?// how to send message to main and reply quickly
.enrichHeaders(c->c.header(HttpHeaders.STATUS_CODE,HttpStatus.OK))
.get();
}
#Bean
public IntegrationFlow mainFlow(){
return IntegrationFlows.from("mainChannel")//accept an request
.handleMainLogic()//handler main logic
.get();
}
User a publishSubscribeChannel.
See this answer.
Add a task executor so that the two subflows run in parallel.
I am using following to define my integration flow:
#Bean
public IntegrationFlow pollingFlow(MessageSource<Object> jdbcMessageSource) {
return IntegrationFlows.from(jdbcMessageSource,
c -> c.poller(Pollers.fixedRate(250, TimeUnit.MILLISECONDS)
.maxMessagesPerPoll(1)
.transactional()))
.split()
.channel(taskSourceChannel())
.get();
}
I would like to make call to service activator that reads from taskSourceChannel as transactional. Also, I want to use following with my transaction.
#Bean
public TransactionSynchronizationFactory transactionSynchronizationFactory() {
ExpressionEvaluatingTransactionSynchronizationProcessor syncProcessor
= new ExpressionEvaluatingTransactionSynchronizationProcessor();
syncProcessor.setAfterCommitChannel(successChannel());
syncProcessor.setAfterRollbackChannel(failureChannel());
return new DefaultTransactionSynchronizationFactory(syncProcessor);
}
The taskSourceChannel is an executor channel.
#Bean
public MessageChannel taskSourceChannel() {
return new ExecutorChannel(executor());
}
How can I add transaction support after split while using TransactionSynchronizationFactory. I don't want to make polling transacational. The only solution I can think of is putting transactional on activator but that won't solve my problem. I would like to make it applicable to any service activator uses this channel.
You question is not so clear, but you definitely need to consider to add transaction into the service activator. Although you don't show what is the subscriber for that taskSourceChannel, but you need to think do not have several subscribers on it.
Nevertheless I think your point is to apply TX into the service activator on this taskSourceChannel and everything after that one.
For this purpose Spring Integration provides a TransactionHandleMessageAdvice. See more info the Reference Manual: https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#tx-handle-message-advice.
The TransactionSynchronizationFactory is only used from the AbstractPollingEndpoint implementations. However you can still utilize it in your transactional context relying on the TransactionSynchronizationManager.registerSynchronization().
A fragment from doc
A Message Endpoint represents the "filter" of a pipes-and-filters
architecture. As mentioned earlier, the endpoint’s primary role is to
connect application code to the messaging framework and to do so in a
non-invasive manner. In other words, the application code should
ideally have no awareness of the message objects or the message
channels.
Transformers, Filters, Service Activators are all Message Endpoints.
What I don't understand is why they are called so as an application actually can send messages into input channels and these "endpoints" are used in the middle of the chain. In other words, the app is aware about a channel, and not about endpoints.
Example:
#SpringBootApplication
#IntegrationComponentScan
public class SendToInputChannel {
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from("input")
.handle(System.out::println).get();
}
public static void main(String[] args) throws IOException {
ConfigurableApplicationContext ctx = SpringApplication.run(SendToInputChannel.class, args);
MessageChannel inputChannel = ctx.getBean("input", MessageChannel.class);
for (int i = 0; i < 10; i++) {
inputChannel.send(MessageBuilder.withPayload(i)
.build());
}
ctx.close();
}
}
Ekaterina, It all started from the Enterprise Integration Patterns (EIP) book and Spring Integration (SI) is effectively a Reference Implementation of the EIP. We were trying to stay pretty close to "that script". So as you can see from this excerpt - Message Endpoint is used to connect an application to a messaging channels. In other words SI in itself is a messaging framework since it provides internal channels to which various Filters/Endpoints could be connected.
Keep in mind that MessageChannel is an interface and could represent not only the internal channels implemented in SI. For example one can implement a MessageChannel interface that provides some proxy to a JMS queue or anything else, while Filters/Endpoints will remain the same/unchanged.
I can go on, but I hope that is enough for now, but fee free to follow up.
This question is more of a design question than a real problem. Given following basic flow:
#Bean
public DirectChannel getFileToSftpChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow sftpOutboundFlow() {
return IntegrationFlows.from(getFileToSftpChannel())
.handle(Sftp.outboundAdapter(this.sftpSessionFactory)
.useTemporaryFileName(false)
.remoteDirectory("test")).get();
}
#Bean
public IntegrationFlow filePollingInboundFlow() {
return from(s -> s.file(new File("path")).patternFilter("*.ext"),
e -> e.poller(fixedDelay(60, SECONDS).channel(getFileToSftpChannel()).get();
}
There is an inbound file polling flow which publishes messages via a DirectChannel to an outbound SFTP flow uploading the file.
After the entire flow finishes, I want to execute a "success" action: move the original file (locally) to an archive folder.
Using the DirectChannel, I understand that the upload will happen in the same thread as the file polling.
In other words, the file poller blocks untill the upload completes (or an error message is returned which is then pushed to the error channel).
Knowing this, I want to place the 'success' action (= moving the original file) on the inbound flow. Things I already know about and don't want to use:
Another 'handle' on the sftpOutbound. Reason: moving the file is tied to the inboud flow not the outbound flow. For ex. if I would introduce another, 2nd, producer later on (eg. a JMS inbound flow) publishing to the same channel, there would be no 'file' to be moved.
Adding an interceptor on the DirectChannel and use the 'afterSendCompletion'. Reason: same as above, I want to logic to be tied to the inbound flow
Add transaction semantics on the inbound flow and react on 'commit'. Reason: as all of this is non transactional (file system/SFTP based) I want to avoid using this.
Another thing I tried was adding an 'handle' on the inbound flow. However, I learned as the inbound flow has no real 'reply', the handle is executed before the message is sent, so this doesn't work as the move has to be executed after successful processing of the message.
Question in short: what is the standard way of executing an action supplied by the producer (=inbound flow) after the message was successfully processed by a consumer (=outbound flow) via the DirectChannel?
Well, the standard way to do something similar is transaction and that's why we some time ago introduced the PseudoTransactionManager and the XML sample for similar task looks like:
<int-file:inbound-channel-adapter id="realTx" channel="txInput" auto-startup="false"
directory="${java.io.tmpdir}/si-test2">
<int:poller fixed-rate="500">
<int:transactional synchronization-factory="syncFactory"/>
</int:poller>
</int-file:inbound-channel-adapter>
<bean id="transactionManager" class="org.springframework.integration.transaction.PseudoTransactionManager"/>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.delete()"/>
</int:transaction-synchronization-factory>
As you see we remove the file in the end of transaction which is caused really after your move to SFTP.
I'd say it is the best way to be tied with only the producer.
Another way is to introduce one more channel before getFileToSftpChannel() and apply the ChannelInterceptor.afterSendCompletion which will be invoked in the end too, by the same single-thread reason. With this approach you should just bridge all your producers with their specific DirectChannels to that single getFileToSftpChannel() for the SFTP adapter.
So, it's up to you what to choose. You have good argument from the architectural perspective to divide the logic by the responsibility levels, but as you see there is no so much choice...
Any other ideas are welcome!
You can try something like the following
#Bean
public DirectChannel getFileToSftpChannel() {
DirectChannel directChannel = new DirectChannel();
directChannel.addInterceptor(new ChannelInterceptorAdapter() {
#Override
public void afterSendCompletion(final Message<?> message,
final MessageChannel channel, final boolean sent, final Exception ex) {
if (ex == null) {
new Archiver().archive((File) message.getPayload());
}
}
});
return directChannel;
}