I want to consume messages from a Queue channel using java-dsl, but Integrations.from doesn't have a signature allowing me to specify a Poller.
How can I achieve this?
Ex.:
#Bean
IntegrationFlow flow() {
return IntegrationFlows.from(this.channel())
.handle(...)
.get();
}
#Bean
MessageChannel channel() {
return MessageChannels.queue().get();
}
Well, actually it is an endpoint responsibility to provide poller properties.
If you are familiar with an XML configuration you should remember that to poll from <queue> we should configure <poller> sub-element for <service-activator> etc.
The same approach is applied in Java DSL as well. The next endpoint definition should be with desired poller:
IntegrationFlows.from(this.channel())
.handle(..., e -> e.poller(Pollers...))
.get();
I've had trouble for some reason setting a poller on the endpoint definition as Artem described - for some reason it gets ignored. You can always set a default poller. This has worked for me:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(500).get();
}
Related
How can I setup a reactive flow using DSL for the following steps:
Receive an SQS Message using SqsMessageDrivenChannelAdapter
Validate the Json message [JsonSchemaValidator class with validate method]
Transform the json to objects
Pass the objects to a service activator (BusinessService : business logic, state machine)
Persist the Objects R2DBC outbound adapter
I was looking at this : https://github.com/spring-projects/spring-integration/blob/master/spring-integration-core/src/test/java/org/springframework/integration/dsl/reactivestreams/ReactiveStreamsTests.java
In the above example, there are dedicated flows created that return a Publisher and in the tests the Publishers are subscribed. However, my flow will be triggered when SqsMessageDrivenChannelAdapter brings in a message into a channel.
How to achieve a reactive flow configuration, for the scenario above steps 1 to 5?
Update : Sample code added
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
Update 2 : Moved JPA to a diff thread using executor channel
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.channel(persistChannel())
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
#Bean
public MessageChannel persistChannel() {
return MessageChannels.executor(Executors.newCachedThreadPool()).get();
}
You probably need to make yourself more familiar with what we have so far for Reactive Streams in Spring Integration: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams
The sample you show with that test class is fully not relevant to your use case. In that test we try to cover some API we expose in Spring Integration, kinda unit tests. It has nothing to do with the whole flow.
Your use-case is really just a full black box flow starting with SQS listener and ending in the R2DBC. Therefore there is no point in your flow to try to convert part of it into the Publisher and then bring it back to another part of the flow: you are not going to track some how and subscribe to that Publisher yourself.
You may consider to place a FluxMessageChannel in between endpoints in your flow, but it still does not make sense for your use-case. It won't be fully reactive as you expect just because a org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer is not blocking on the consumer thread to be ready for a back-pressure from downstream.
The only really reactive part of your flow is that R2DBC outbound channel adapter, but probably it does not bring you too much value because the source of data is not reactive.
As I said: you can try to place a channel(channels -> channels.flux()) just after an SqsMessageDrivenChannelAdapter definition to start a reactive flow from that point. At the same time you should try to set a maxNumberOfMessages to 1 to try to make it waiting for a free space in before pulling the next mesasge from SQS.
Given I have application which uses Spring Integration and I define gateway:
#Component
#MessagingGateway
public interface SmsGateway {
#Gateway(requestChannel = CHANNEL_SEND_SMS)
void sendSms(SendSmsRequest request);
}
public interface IntegrationChannels {
String CHANNEL_SEND_SMS = "channelSendSms";
}
I also attach IntegrationFlow to CHANNEL_SEND_SMS channel:
#Bean
public IntegrationFlow sendSmsFlow() {
return IntegrationFlows.from(CHANNEL_SEND_SMS)
.transform(...)
.handle(...)
.get();
}
Whenever I call sendSms gateway method from business code, sendSmsFlow is executed as expected.
When I want to attach another IntegrationFlow to the same CHANNEL_SEND_SMS channel, e.g.
#Bean
public IntegrationFlow differentFlow() {
return IntegrationFlows.from(CHANNEL_SEND_SMS)
.transform(...)
.handle(...)
.get();
}
then this differentFlow is not executed.
Why does it behave this way?
Is there any solution to make it work for both flows?
The default channel type is DirectChannel and messages are distributed to multiple subscribed channels in a round robin fashion by default.
Declare CHANNEL_SEND_SMS as a PublishSubscribeChannel if you want each flow to get every message.
This will only work with a void gateway method; if there is a return type, you will get the first (or random if there is any async downstream processing) and the others will be discarded.
I have this direct channel:
#Bean
public DirectChannel emailingChannel() {
return MessageChannels
.direct( "emailingChannel")
.get();
}
Can I define multiple flows for the same channel like this:
#Bean
public IntegrationFlow flow1FromEmailingChannel() {
return IntegrationFlows.from( "emailingChannel" )
.handle( "myService" , "handler1")
.get();
}
#Bean
public IntegrationFlow flow2FromEmailingChannel() {
return IntegrationFlows.from( "emailingChannel" )
.handle( "myService" , "handler2" )
.get();
}
EDIT
#Service
public class MyService {
public void handler1(Message<String> message){
....
}
public void handler2(Message<List<String>> message){
....
}
}
Each flow's handle(...) method manipulates different payload data types but the goal is the same, i-e reading data from the channel and call the relevant handler. I would like to avoid many if...else to check the data type in one handler.
Additional question: What happens when multiple threads call the same channel (no matter its type: Direct, PubSub or Queue) at the same time (as per default a #Bean has singleton scope)?
Thanks a lot
With a direct channel messages will be round-robin distributed to the consumers.
With a queue channel only one consumer will get each message; the distribution will be based on their respective pollers.
With a pub/sub channel both consumers will get each message.
You need to provide more information but it sounds like you need to add a payload type router to your flow to direct the messages to the right consumer.
EDIT
When the handler methods are in the same class you don't need two flows; the framework will examine the methods and, as long as there is no ambiguity) will call the method that matches the payload type.
.handle(myServiceBean())
I have an outbound gateway for soap calls (MarshallingWebServiceOutboundGateway) with elaborate setup. I need to use that gateway definition from multiple flows.
The question spring-integration: MessageProducer may only be referenced once is somewhat similar, but this question is about the proper use of the spring bean scope prototype for spring integration collaborators.
I have a separate config file which sets up the gateway and its dependencies:
#Bean
public MarshallingWebServiceOutboundGateway myServiceGateway() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan("blah.*");
MarshallingWebServiceOutboundGateway gateway = new MarshallingWebServiceOutboundGateway(
serviceEndpoint, marshaller, messageFactory);
gateway.setMessageSender(messageSender);
gateway.setRequestCallback(messageCallback);
return gateway;
}
This is how I initially tried to wire up the outbound gateway from two different flows in two different config files.
In one config file:
#Bean
public IntegrationFlow flow1() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res1")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.get();
}
In a separate config file:
#Bean
public IntegrationFlow flow2() {
MarshallingWebServiceOutboundGateway myServiceGateway = context.getBean("myServiceGateway", MarshallingWebServiceOutboundGateway.class);
return IntegrationFlows
.from(Http.inboundGateway("/res2")
.requestMapping(r -> r.methods(HttpMethod.GET))
.transform(soapRequestTransformer)
.handle(myServiceGateway) // wrong: cannot be same bean
.transform(widgetTransformer)
.handle(servicePojo)
.get();
}
This is a problem because - as I understand it - myServiceGateway cannot be the same instance, since that instance has only one outbound channel and cannot belong to two different flows.
In the related question spring-integration: MessageProducer may only be referenced once, #artem-bilan advised not to create the outbound gateway in an #Bean method, rather to use a plain method which creates new instances for every call.
That works, but it is inconvenient in my case. I need to reuse the outbound gateway from several flows in different config files and I would have to copy the code to create the gateway into each config file. Also, the gateway dependencies inflate my Configuration file constructors, making Sonar bail.
Since the error message coming out of IntegrationFlowDefinition.checkReuse() says A reply MessageProducer may only be referenced once (myServiceGateway) - use #Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) on #Bean definition. I wanted to give the scope prototype another try.
So I try to make spring integration look up a prototype gateway from the context by name, hoping to get a different gateway instance in flow1 and flow2:
.handle(context.getBean("myServiceGateway",
MarshallingWebServiceOutboundGateway.class))
And I annotated the outbound gateway #Bean definition with
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
But I can see that the myServiceGateway() method is only invoked once, despite the prototype scope, and application startup still fails with the error message which advises to use the prototype scope - quite confusing, actually ;-)
Based on Mystery around Spring Integration and prototype scope I also tried:
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE, proxyMode = ScopedProxyMode.TARGET_CLASS)
The application starts, but the responses never reach the step after the gateway, the widgetTransformer. (Even more strange, exactly the widgetTransformer is skipped: in flow1 the outcome is the untransformed gateway response and in flow2 the untransformed messages hit the step after the widgetTransformer, i.e. the servicePojo). Making a proxy out of a message producer seems not to be a good idea.
I really want to get to the bottom of this. Is the exception message wrong which asks to use the prototype scope or am I just getting it wrong? How can I avoid to repeat the bean definition for message producers if I need several such producers which are all set up the same way?
Using spring-integration 5.0.9.
I am not entirely sure why the #Scope is not working, but here is a work-around...
#SpringBootApplication
public class So52453934Application {
public static void main(String[] args) {
SpringApplication.run(So52453934Application.class, args);
}
#Autowired
private HandlerConfig config;
#Bean
public IntegrationFlow flow1() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public IntegrationFlow flow2() {
return f -> f.handle(this.config.myHandler())
.handle(System.out::println);
}
#Bean
public ApplicationRunner runner() {
return args -> {
context.getBean("flow1.input", MessageChannel.class).send(new GenericMessage<>("foo"));
context.getBean("flow2.input", MessageChannel.class).send(new GenericMessage<>("bar"));
};
}
}
#Configuration
class HandlerConfig {
public AbstractReplyProducingMessageHandler myHandler() {
return new AbstractReplyProducingMessageHandler() {
#Override
protected Object handleRequestMessage(Message<?> requestMessage) {
return ((String) requestMessage.getPayload()).toUpperCase();
}
};
}
}
i.e. do as #artem suggested, but inject the bean with the factory method.
I am trying to write a simple message flow using Spring Integration v4's DSL APIs which would look like this:
-> in.ch -> Processing -> JmsGatewayOut -> JMS_OUT_QUEUE
Gateway
<- out.ch <- Processing <- JmsGatewayIn <- JMS_IN_QUEUE
With the request/response being asynchronous, when I inject a message via the initial Gateway, the message goes all the way to JMS_OUT_QUEUE. Beyond this message flow, a reply message is put back into JMS_IN_QUEUE which it is then picked up by JmsGatewayIn. At this point, the message is Processed and placed into out.ch (I know the response gets to out.ch because I have a logger interceptor there which logs the message being placed there) but, the Gateway never receives the response.
Instead of a response, the system outside of this message flow which picked up the message from JMS_OUT_QUEUE and placed the response in JMS_IN_QUEUE, receives a javax.jms.MessageFormatException: MQJMS1061: Unable to deserialize object on its own JmsOutboundgateway (I think it is failing to deserialize a jms reply object from looking at the logs).
I have clearly not got something configured correctly but I don't know exactly what. Does anyone know what I am missing?
Working with spring-integration-core-4.0.3.RELEASE, spring-integration-jms-4.0.3.RELEASE, spring-integration-java-dsl-1.0.0.M2, spring-jms-4.0.6.RELEASE.
My Gateway is configured as follows:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyChannel = "out.ch",
replyTimeout = 45000)
AResponse process(ARequest request);
}
My Integration flow is configured as follows:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Bean(name = "out.ch")
public DirectChannel outCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.correlationKey("JMSCorrelationID")
.get();
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(Jms.inboundGateway(mqConnectionFactory)
.destination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.channel("out.ch")
.get();
}
}
Thanks for any help on this,
PM.
First of all your configuration is bad:
Since you start the flow from WsGateway#process you really should wait reply there.
The gateway's request/reply capability is based on TemporaryReplyChannel, which is placed to the headers as non-serializable value.
As long as you wait rely on that gateway, actually there is no reason to provide the replyChannel, if you aren't going to do some publish-subscribe logic on the reply.
As you send message to the JMS queue, you should understand that consumer part might be a separete remote application. And the last one might know nothing about your out.ch.
The JMS request/reply capability is really based on JMSCorrelationID, but it isn't enough. The one more thing here is a ReplyTo JMS header. Hence, if you are going to send reply from the consumer you should really just rely on the JmsGatewayIn stuff.
So I'd change your code to this:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyTimeout = 45000)
AResponse process(ARequest request);
}
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.replyDestination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.get();
}
}
Let me know, if it is appropriate for you or try to explian why you use two-way gateways for one one-way cases. Maybe Jms.outboundAdapter() and Jms.inboundAdapter() are more good for you?
UPDATE
How to use <header-channels-to-string> from Java DSL:
.enrichHeaders(e -> e.headerChannelsToString())