Spring Integration 4 asynchronous request/response - spring-integration

I am trying to write a simple message flow using Spring Integration v4's DSL APIs which would look like this:
-> in.ch -> Processing -> JmsGatewayOut -> JMS_OUT_QUEUE
Gateway
<- out.ch <- Processing <- JmsGatewayIn <- JMS_IN_QUEUE
With the request/response being asynchronous, when I inject a message via the initial Gateway, the message goes all the way to JMS_OUT_QUEUE. Beyond this message flow, a reply message is put back into JMS_IN_QUEUE which it is then picked up by JmsGatewayIn. At this point, the message is Processed and placed into out.ch (I know the response gets to out.ch because I have a logger interceptor there which logs the message being placed there) but, the Gateway never receives the response.
Instead of a response, the system outside of this message flow which picked up the message from JMS_OUT_QUEUE and placed the response in JMS_IN_QUEUE, receives a javax.jms.MessageFormatException: MQJMS1061: Unable to deserialize object on its own JmsOutboundgateway (I think it is failing to deserialize a jms reply object from looking at the logs).
I have clearly not got something configured correctly but I don't know exactly what. Does anyone know what I am missing?
Working with spring-integration-core-4.0.3.RELEASE, spring-integration-jms-4.0.3.RELEASE, spring-integration-java-dsl-1.0.0.M2, spring-jms-4.0.6.RELEASE.
My Gateway is configured as follows:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyChannel = "out.ch",
replyTimeout = 45000)
AResponse process(ARequest request);
}
My Integration flow is configured as follows:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Bean(name = "out.ch")
public DirectChannel outCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.correlationKey("JMSCorrelationID")
.get();
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(Jms.inboundGateway(mqConnectionFactory)
.destination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.channel("out.ch")
.get();
}
}
Thanks for any help on this,
PM.

First of all your configuration is bad:
Since you start the flow from WsGateway#process you really should wait reply there.
The gateway's request/reply capability is based on TemporaryReplyChannel, which is placed to the headers as non-serializable value.
As long as you wait rely on that gateway, actually there is no reason to provide the replyChannel, if you aren't going to do some publish-subscribe logic on the reply.
As you send message to the JMS queue, you should understand that consumer part might be a separete remote application. And the last one might know nothing about your out.ch.
The JMS request/reply capability is really based on JMSCorrelationID, but it isn't enough. The one more thing here is a ReplyTo JMS header. Hence, if you are going to send reply from the consumer you should really just rely on the JmsGatewayIn stuff.
So I'd change your code to this:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyTimeout = 45000)
AResponse process(ARequest request);
}
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.replyDestination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.get();
}
}
Let me know, if it is appropriate for you or try to explian why you use two-way gateways for one one-way cases. Maybe Jms.outboundAdapter() and Jms.inboundAdapter() are more good for you?
UPDATE
How to use <header-channels-to-string> from Java DSL:
.enrichHeaders(e -> e.headerChannelsToString())

Related

Spring Integration aws Kinesis , message aggregator, Release Strategy

this is a follow-up question to Spring Integration AWS RabbitMQ Kinesis
I have the following configuration. I am noticing that when I send a message to the input channel named kinesisSendChannel for the first time, the aggregator and release strategy is getting invoked and messages are sent to Kinesis Streams. I put debug breakpoints at different places and could verify this behavior. But when I again publish messages to the same input channel the release strategy and the outbound processor are not getting invoked and messages are not sent to the Kinesis. I am not sure why the aggregator flow is getting invoked only the first time and not for subsequent messages. For testing purpose , the TimeoutCountSequenceSizeReleaseStrategy is set with count as 1 & time as 60 seconds. There is no specific MessageStore used. Could you help identify the issue?
#Bean(name = "kinesisSendChannel")
public MessageChannel kinesisSendChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "resultChannel")
public MessageChannel resultChannel() {
return MessageChannels.direct().get();
}
#Bean
#ServiceActivator(inputChannel = "kinesisSendChannel")
public MessageHandler aggregator(TestMessageProcessor messageProcessor,
MessageChannel resultChannel,
TimeoutCountSequenceSizeReleaseStrategy timeoutCountSequenceSizeReleaseStrategy) {
AggregatingMessageHandler handler = new AggregatingMessageHandler(messageProcessor);
handler.setCorrelationStrategy(new ExpressionEvaluatingCorrelationStrategy("headers['foo']"));
handler.setReleaseStrategy(timeoutCountSequenceSizeReleaseStrategy);
handler.setOutputProcessor(messageProcessor);
handler.setOutputChannel(resultChannel);
return handler;
}
#Bean
#ServiceActivator(inputChannel = "resultChannel")
public MessageHandler kinesisMessageHandler1(#Qualifier("successChannel") MessageChannel successChannel,
#Qualifier("errorChannel") MessageChannel errorChannel, final AmazonKinesisAsync amazonKinesis) {
KinesisMessageHandler kinesisMessageHandler = new KinesisMessageHandler(amazonKinesis);
kinesisMessageHandler.setSync(true);
kinesisMessageHandler.setOutputChannel(successChannel);
kinesisMessageHandler.setFailureChannel(errorChannel);
return kinesisMessageHandler;
}
public class TestMessageProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected Object aggregatePayloads(MessageGroup group, Map<String, Object> defaultHeaders) {
final PutRecordsRequest putRecordsRequest = new PutRecordsRequest().withStreamName("test-stream");
final List<PutRecordsRequestEntry> putRecordsRequestEntry = group.getMessages().stream()
.map(message -> (PutRecordsRequestEntry) message.getPayload()).collect(Collectors.toList());
putRecordsRequest.withRecords(putRecordsRequestEntry);
return putRecordsRequestEntry;
}
}
I believe the problem is here handler.setCorrelationStrategy(new ExpressionEvaluatingCorrelationStrategy("headers['foo']"));. All your messages come with the same foo header. So, all of them form the same message group. As long as you release group and don’t remove it, all the new messages are going to be discarded.
Please, revise aggregator documentation to make yourself familiar with all the possible behavior : https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator

Retry handler in Spring Java DSL

currently, I have Spring Integration Flow where reading payload from JMS queue, transforming to XML format, then send the XML payload to the core app. at the RecordSenderHandler, there is logic to make call rest API to my core app and store the response to Redis according to the response I received. If my core app is not accessible or something wrong with my backend, I flag as error HTTP 500. But I do want to retry the execution for certain times and limit maximum error I got. below is my code. any suggestions?
#Bean
public IntegrationFlow jmsMessageDrivenFlowWithContainer() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(
Jms.container(this.jmsConnectionFactory, recordDestinationQueue)
.concurrentConsumers(xmlConcurrentConsumers)
.maxConcurrentConsumers(xmlMaxConcurrentConsumers))
.errorChannel("errorChannel"))
.handle(payloadSender(), e->e.advice(circuitBreakerAdvice()))
.get();
}
#Bean
#ServiceActivator(inputChannel = "handleChannel")
public PayloadSender payloadSender() {
return new PayloadSender ();
}
#Bean
public RequestHandlerCircuitBreakerAdvice circuitBreakerAdvice() {
RequestHandlerCircuitBreakerAdvice requestHandlerCircuitBreakerAdvice = new RequestHandlerCircuitBreakerAdvice();
requestHandlerCircuitBreakerAdvice.setThreshold(3);
requestHandlerCircuitBreakerAdvice.setHalfOpenAfter(15000);
return requestHandlerCircuitBreakerAdvice;
}
See Adding Behavior to Endpoints and in particular the RequestHandlerRetryAdvice.
.handle(..., e -> e.advice(retryAdvice()))
...
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
...
}

#Transformer for ObjectToJson Not Working in Spring Integration

A POJO Message.java is to be Converted to JSON(JSON is to be sent to pubsub Topic,using Spring Integration MessageChannels.),using following:
#Bean
#Transformer(inputChannel = "pubsubOutputChannel", outputChannel = "handleOutChannel")
public ObjectToJsonTransformer transformOut() {
return new ObjectToJsonTransformer();
}
#MessagingGateway(defaultRequestChannel = "pubsubOutputChannel")
public interface PubsubOutboundGateway {
void sendToPubsub(Messages msg);
}
#Bean
#ServiceActivator(inputChannel = "handleOutChannel")
public MessageHandler messageSender(PubSubOperations pubsubTemplate) {
return new PubSubMessageHandler(pubsubTemplate, "TestTopic");
}
When i call sendToPubsub() with an instance of Message.java with required properties set,i get an error "Null".
Is serviceActivator not able to receive the required data?
Any suggestions to fix this?.
Yes, it can't do that because you just don't tell it to do that.
Your gateway is configured for this:
#MessagingGateway(defaultRequestChannel = "handleOutChannel")
But that is not an input channel for the ObjectToJsonTransformer. So, whatever you send over that gateway is going directly to the messageSender service activator.
Try to configure your gateway like this:
#MessagingGateway(defaultRequestChannel = "pubsubOutputChannel")

Spring Integration: reuse MessageProducer definition

I have an outbound gateway for soap calls (MarshallingWebServiceOutboundGateway) with elaborate setup. I need to use that gateway definition from multiple flows.
The question spring-integration: MessageProducer may only be referenced once is somewhat similar, but this question is about the proper use of the spring bean scope prototype for spring integration collaborators.
I have a separate config file which sets up the gateway and its dependencies:
#Bean
public MarshallingWebServiceOutboundGateway myServiceGateway() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("blah.*");
MarshallingWebServiceOutboundGateway gateway = new MarshallingWebServiceOutboundGateway(
serviceEndpoint, marshaller, messageFactory);
gateway.setMessageSender(messageSender);
gateway.setRequestCallback(messageCallback);
return gateway;
}
This is how I initially tried to wire up the outbound gateway from two different flows in two different config files.
In one config file:
#Bean
public IntegrationFlow flow1() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res1")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.get();
}
In a separate config file:
#Bean
public IntegrationFlow flow2() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res2")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.handle(servicePojo)
.get();
}
This is a problem because - as I understand it - myServiceGateway cannot be the same instance, since that instance has only one outbound channel and cannot belong to two different flows.
In the related question spring-integration: MessageProducer may only be referenced once, #artem-bilan advised not to create the outbound gateway in an #Bean method, rather to use a plain method which creates new instances for every call.
That works, but it is inconvenient in my case. I need to reuse the outbound gateway from several flows in different config files and I would have to copy the code to create the gateway into each config file. Also, the gateway dependencies inflate my Configuration file constructors, making Sonar bail.
Since the error message coming out of IntegrationFlowDefinition.checkReuse() says A reply MessageProducer may only be referenced once (myServiceGateway) - use #Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) on #Bean definition. I wanted to give the scope prototype another try.
So I try to make spring integration look up a prototype gateway from the context by name, hoping to get a different gateway instance in flow1 and flow2:
.handle(context.getBean("myServiceGateway",
MarshallingWebServiceOutboundGateway.class))
And I annotated the outbound gateway #Bean definition with
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
But I can see that the myServiceGateway() method is only invoked once, despite the prototype scope, and application startup still fails with the error message which advises to use the prototype scope - quite confusing, actually ;-)
Based on Mystery around Spring Integration and prototype scope I also tried:
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE, proxyMode = ScopedProxyMode.TARGET_CLASS)
The application starts, but the responses never reach the step after the gateway, the widgetTransformer. (Even more strange, exactly the widgetTransformer is skipped: in flow1 the outcome is the untransformed gateway response and in flow2 the untransformed messages hit the step after the widgetTransformer, i.e. the servicePojo). Making a proxy out of a message producer seems not to be a good idea.
I really want to get to the bottom of this. Is the exception message wrong which asks to use the prototype scope or am I just getting it wrong? How can I avoid to repeat the bean definition for message producers if I need several such producers which are all set up the same way?
Using spring-integration 5.0.9.
I am not entirely sure why the #Scope is not working, but here is a work-around...
#SpringBootApplication
public class So52453934Application {
public static void main(String[] args) {
SpringApplication.run(So52453934Application.class, args);
}
#Autowired
private HandlerConfig config;
#Bean
public IntegrationFlow flow1() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public IntegrationFlow flow2() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public ApplicationRunner runner() {
return args -> {
context.getBean("flow1.input", MessageChannel.class).send(new GenericMessage<>("foo"));
context.getBean("flow2.input", MessageChannel.class).send(new GenericMessage<>("bar"));
};
}
}
#Configuration
class HandlerConfig {
public AbstractReplyProducingMessageHandler myHandler() {
return new AbstractReplyProducingMessageHandler() {
#Override
protected Object handleRequestMessage(Message<?> requestMessage) {
return ((String) requestMessage.getPayload()).toUpperCase();
}
};
}
}
i.e. do as #artem suggested, but inject the bean with the factory method.

create sftp reply-channel to reply with error or messages that was unsuccessfully sent

I am using java dsl to configure sfp outbound flow.
Gateway:
#MessagingGateway
public interface SftpGateway {
#Gateway(requestChannel = "sftp-channel")
void sendFiles(List<Message> messages);
}
Config:
#Bean
public IntegrationFlow sftpFlow(DefaultSftpSessionFactory sftpSessionFactory) {
return IntegrationFlows
.from("sftp-channel")
.split()
.handle(Sftp.outboundAdapter(sftpSessionFactory, FileExistsMode.REPLACE)
.useTemporaryFileName(false)
.remoteDirectory(REMOTE_DIR_TO_CREATE).autoCreateDirectory(true)).get();
}
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
...
}
How can i configure flow to make my gateway reply with Messages that were failed?
In other words i want my gateway to be able to return list of messages which were failed, not void.
I marked gateway with
#MessagingGateway(errorChannel = "errorChannel")
and wrote error channel
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from("errorChannel").handle(new GenericHandler<MessagingException>() {
public Message handle(MessagingException payload, Map headers) {
System.out.println(payload.getFailedMessage().getHeaders());
return payload.getFailedMessage();
}
})
.get();
}
#Bean
public MessageChannel errorChannel() {
return MessageChannels.direct().get();
}
and in case of some errors(i.e. no connection to SFTP) i get only one error (payload of first message in list).
Where should i put Advice to aggregate all messages?
This is not the question to Spring Integration Java DSL.
This is mostly a design and architecture task.
Currently you don't have any choice because you use Sftp.outboundAdapter() which is one-way, therefore without any reply. And your SftpGateway is ready for that behavior with the void return type.
If you have a downstream errorr, you can only throw them or catch and send to some error-channel.
According to your request of:
i want my gateway to be able to return list of messages which were failed, not void.
I'd say it depends. Actually it is just return from your gateway. So, if you return an empty list into gateway that may mean that there is no errors.
Since Java doesn't provide multi-return capabilities we don't have choice unless do something in our stream which builds that single message to return. As we decided list of failed messages.
Since you have there .split(), you should look into .aggregate() to build a single reply.
Aggregator correlates with the Splitter enough easy, via default applySequence = true.
To send to aggregator I'd suggest to take a look into ExpressionEvaluatingRequestHandlerAdvice on the Sftp.outboundAdapter() endpoint (second param of the .handle()). With that you should send both good and bad messages to the same .aggregate() flow. Than you can iterate a result list to clean up it from the good result. The result after that can be send to the SftpGateway using replyChannel header.
I understand that it sounds a bit complicated, but what you want doesn't exist out-of-the-box. Need to think and play yourself to figure out what can be reached.

Resources