Send messages from a BlockingQueue through a dynamically created web socket using Spring Integration - spring-integration

I have a SpringBoot application that also uses SpringIntegration to dynamically create web sockets upon requests that come from an Angular web application and send messages on that web sockets that are taken from a BlockingQueue.
I want those messages to be pushed on the web socket only when a message is available on the BlockingQueue, so without polling.
At the moment, my code looks like below, but it is very inefficient:
#Service
public class WebSocketPublisherService {
#Autowired
private IntegrationFlowContext integrationFlowContext;
#Bean
public HandshakeHandler handshakeHandler() {
return new DefaultHandshakeHandler(new TomcatRequestUpgradeStrategy());
}
public void createWebSocket() {
ServerWebSocketContainer serverWebSocketContainer = new ServerWebSocketContainer("/test")
.setHandshakeHandler(handshakeHandler())
.setAllowedOrigins("http://localhost:4200");
serverWebSocketContainer.setMessageListener(...); // I want messages comming from the web application to be processed
WebSocketOutboundMessageHandler webSocketOutboundMessageHandler = new WebSocketOutboundMessageHandler(serverWebSocketContainer);
MethodInvokingMessageSource methodInvokingMessageSource = new MethodInvokingMessageSource();
methodInvokingMessageSource.setObject(...);
methodInvokingMessageSource.setMethodName(...); // this method returns the element from the BlockingQueue, if it exists
StandardIntegrationFlow standardIntegrationFlow = IntegrationFlow
.from(methodInvokingMessageSource, polling -> polling.poller(pollerFactory -> pollerFactory.fixedRate(10)))
.split(new DecorateMessageWithSessionId(serverWebSocketContainer))
.handle(webSocketOutboundMessageHandler)
.get();
IntegrationFlowContext.IntegrationFlowRegistration flowRegistration = integrationFlowContext
.registration(standardIntegrationFlow)
.addBean(serverWebSocketContainer)
.register();
flowRegistration.start();
}
}
Is there a smarter way of creating an IntegrationFlow using the content of a BlockingQueue (based on subscription and not polling)?

Related

Spring Integration Flow DSL with SQS and Reactive

How can I setup a reactive flow using DSL for the following steps:
Receive an SQS Message using SqsMessageDrivenChannelAdapter
Validate the Json message [JsonSchemaValidator class with validate method]
Transform the json to objects
Pass the objects to a service activator (BusinessService : business logic, state machine)
Persist the Objects R2DBC outbound adapter
I was looking at this : https://github.com/spring-projects/spring-integration/blob/master/spring-integration-core/src/test/java/org/springframework/integration/dsl/reactivestreams/ReactiveStreamsTests.java
In the above example, there are dedicated flows created that return a Publisher and in the tests the Publishers are subscribed. However, my flow will be triggered when SqsMessageDrivenChannelAdapter brings in a message into a channel.
How to achieve a reactive flow configuration, for the scenario above steps 1 to 5?
Update : Sample code added
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
Update 2 : Moved JPA to a diff thread using executor channel
#Bean
public IntegrationFlow importFlow() {
IntegrationFlows.from(sqsInboundChannel())
.handle((payload, messageHeaders) -> jsonSchemaValidator.validate(payload.toString()))
.transform(Transformers.fromJson(Entity.class))
.handle((payload, messageHeaders) ->businessService.process((Entity) payload))
.channel(persistChannel())
.handle(
Jpa.outboundAdapter(this.entityManagerFactory)
.entityClass(Entity)
.persistMode(PersistMode.PERSIST),
ConsumerEndpointSpec::transactional)
.get();
}
#Bean
public MessageProducer sqsMessageDrivenChannelAdapter() {
SqsMessageDrivenChannelAdapter sqsMessageDrivenChannelAdapter =
new SqsMessageDrivenChannelAdapter(asyncSqsClient, queueName);
sqsMessageDrivenChannelAdapter.setAutoStartup(true);
sqsMessageDrivenChannelAdapter.setOutputChannel(sqsInboundChannel());
return sqsMessageDrivenChannelAdapter;
}
#Bean
public MessageChannel sqsInboundChannel() {
return MessageChannels.flux().get();
}
#Bean
public MessageChannel persistChannel() {
return MessageChannels.executor(Executors.newCachedThreadPool()).get();
}
You probably need to make yourself more familiar with what we have so far for Reactive Streams in Spring Integration: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams
The sample you show with that test class is fully not relevant to your use case. In that test we try to cover some API we expose in Spring Integration, kinda unit tests. It has nothing to do with the whole flow.
Your use-case is really just a full black box flow starting with SQS listener and ending in the R2DBC. Therefore there is no point in your flow to try to convert part of it into the Publisher and then bring it back to another part of the flow: you are not going to track some how and subscribe to that Publisher yourself.
You may consider to place a FluxMessageChannel in between endpoints in your flow, but it still does not make sense for your use-case. It won't be fully reactive as you expect just because a org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer is not blocking on the consumer thread to be ready for a back-pressure from downstream.
The only really reactive part of your flow is that R2DBC outbound channel adapter, but probably it does not bring you too much value because the source of data is not reactive.
As I said: you can try to place a channel(channels -> channels.flux()) just after an SqsMessageDrivenChannelAdapter definition to start a reactive flow from that point. At the same time you should try to set a maxNumberOfMessages to 1 to try to make it waiting for a free space in before pulling the next mesasge from SQS.

Abstracting Spring Cloud Stream Producer and Consumer code

I have a Service that is Producing and Consuming messages from different Spring Cloud Stream Channels (bound to EventHub/Kafka topics). There are several such Services which are setup similarly.
The configuration looks like below
public interface MessageStreams {
String WORKSPACE = "workspace";
String UPLOADNOTIFICATION = "uploadnotification";
String BLOBNOTIFICATION = "blobnotification";
String INGESTIONSTATUS = "ingestionstatusproducer";
#Input(WORKSPACE)
SubscribableChannel workspaceChannel();
#Output(UPLOADNOTIFICATION)
MessageChannel uploadNotificationChannel();
#Input(BLOBNOTIFICATION)
SubscribableChannel blobNotificationChannel();
#Output(INGESTIONSTATUS)
MessageChannel ingestionStatusChannel();
}
#EnableBinding(MessageStreams.class)
public class EventHubStreamsConfiguration {
}
The Producer/Publisher code looks like below
#Service
#Slf4j
public class IngestionStatusEventPublisher {
private final MessageStreams messageStreams;
public IngestionStatusEventPublisher(MessageStreams messageStreams) {
this.messageStreams = messageStreams;
}
public void sendIngestionStatusEvent() {
log.info("Sending ingestion status event");
System.out.println("Sending ingestion status event");
MessageChannel messageChannel = messageStreams.ingestionStatusChannel();
boolean messageSent = messageChannel.send(MessageBuilder
.withPayload(IngestionStatusMessage.builder()
.correlationId("some-correlation-id")
.status("done")
.source("some-source")
.eventTime(OffsetDateTime.now())
.build())
.setHeader("tenant-id", "some-tenant")
.build());
log.info("Ingestion status event sent successfully {}", messageSent);
}
}
Similarly I have multiple other Publishers which publish to different Event Hubs/Topics. Notice that there is a tenant-id header being set for each published message. This is something specific to my multi-tenant application to track the tenant context. Also notice that I am getting the channel to be published to while sending the message.
My Consumer code looks like below
#Component
#Slf4j
public class IngestionStatusEventHandler {
private AtomicInteger eventCount = new AtomicInteger();
#StreamListener(TestMessageStreams.INGESTIONSTATUS)
public void handleEvent(#Payload IngestionStatusMessage message, #Header(name = "tenant-id") String tenantId) throws Exception {
log.info("New ingestion status event received: {} in Consumer: {}", message, Thread.currentThread().getName());
// set the tenant context as thread local from the header.
}
Again I have several such consumers and also there is a tenant context that is set in each consumer based on the incoming tenant-id header that is sent by the Publisher.
My questions is
How do I get rid of the boiler plate code of setting the tenant-id header in Publisher and setting the tenant context in the Consumer by abstracting it into a library which could be included in all the different Services that I have.
Also, is there a way of dynamically identifying the Channel based on the Type of the Message being published. for ex IngestionStatusMessage.class in the given scenario
To set and tenant-id header in the common code and to avoid its copy/pasting in every microservice you can use a ChannelInterceptor and make it as global one with a #GlobalChannelInterceptor and its patterns option.
See more info in Spring Integration: https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/core.html#channel-interceptors
https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/overview.html#configuration-enable-integration
You can't make a channel selection by the payload type because the payload type is really determined from the #StreamListener method signature.
You can try to have a general #Router with a Message<?> expectation and then return a particular channel name to route according that request message context.
See https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/message-routing.html#messaging-routing-chapter

Retry handler in Spring Java DSL

currently, I have Spring Integration Flow where reading payload from JMS queue, transforming to XML format, then send the XML payload to the core app. at the RecordSenderHandler, there is logic to make call rest API to my core app and store the response to Redis according to the response I received. If my core app is not accessible or something wrong with my backend, I flag as error HTTP 500. But I do want to retry the execution for certain times and limit maximum error I got. below is my code. any suggestions?
#Bean
public IntegrationFlow jmsMessageDrivenFlowWithContainer() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(
Jms.container(this.jmsConnectionFactory, recordDestinationQueue)
.concurrentConsumers(xmlConcurrentConsumers)
.maxConcurrentConsumers(xmlMaxConcurrentConsumers))
.errorChannel("errorChannel"))
.handle(payloadSender(), e->e.advice(circuitBreakerAdvice()))
.get();
}
#Bean
#ServiceActivator(inputChannel = "handleChannel")
public PayloadSender payloadSender() {
return new PayloadSender ();
}
#Bean
public RequestHandlerCircuitBreakerAdvice circuitBreakerAdvice() {
RequestHandlerCircuitBreakerAdvice requestHandlerCircuitBreakerAdvice = new RequestHandlerCircuitBreakerAdvice();
requestHandlerCircuitBreakerAdvice.setThreshold(3);
requestHandlerCircuitBreakerAdvice.setHalfOpenAfter(15000);
return requestHandlerCircuitBreakerAdvice;
}
See Adding Behavior to Endpoints and in particular the RequestHandlerRetryAdvice.
.handle(..., e -> e.advice(retryAdvice()))
...
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
...
}

spring-integration-mqtt With multiple Mqtt Servers for subscription

I am using Spring's spring-integration-mqtt and i can connect to a single Mqtt server and can receive messages on subscribed topics , and now i want to make application which can connect to multiple Mqtt Servers and can receive data from every connection and i want to manage it as dynamic where i can add more Mqtt servers from database or text file.
a simple bean for single Mqtt connection for subscription is as follow
#Bean
public MessageProducer inbound() {
MqttPahoMessageDrivenChannelAdapter adapter2 =
new MqttPahoMessageDrivenChannelAdapter("tcp://192.168.100.1:1883","mqtt_virtual_received_sus_2",
"DATA/#", "LD/#","CONF/#","CONFIG/#");
adapter2.setCompletionTimeout(0);
adapter2.setConverter(new DefaultPahoMessageConverter());
adapter2.setQos(2);
adapter2.setOutputChannel(mqttInputChannel());
return adapter2;
}
above code creates a connection for the mqtt server and can receive messages and if i copy paste the same code twice for second server with different Mqtt ip address i can connect to both Mqtt Server as follows
#Bean
public MessageProducer inbound() {
MqttPahoMessageDrivenChannelAdapter adapter2 =
new MqttPahoMessageDrivenChannelAdapter("tcp://192.168.100.1:1883","mqtt_virtual_received_sus_2",
"DATA/#", "LD/#","CONF/#","CONFIG/#");
adapter2.setCompletionTimeout(0);
adapter2.setConverter(new DefaultPahoMessageConverter());
adapter2.setQos(2);
adapter2.setOutputChannel(mqttInputChannel());
return adapter2;
}
#Bean
public MessageProducer inbound2() {
MqttPahoMessageDrivenChannelAdapter adapter2 =
new MqttPahoMessageDrivenChannelAdapter("tcp://192.168.100.14:1883","mqtt_virtual_received_sus_1",
"DATA/#", "LD/#","CONF/#","CONFIG/#");
adapter2.setCompletionTimeout(0);
adapter2.setConverter(new DefaultPahoMessageConverter());
adapter2.setQos(2);
adapter2.setOutputChannel(mqttInputChannel());
return adapter2;
}
above code also works fine and i can receive message from both Mqtt Servers, but is there any way i can manage it dynamically like as follows, i change the bean's return type to list, but didn't worked:
#Bean
public List<MqttPahoMessageDrivenChannelAdapter> getAdapter () {
List<MqttPahoMessageDrivenChannelAdapter > logConfList=new ArrayList<MqttPahoMessageDrivenChannelAdapter>();
MqttPahoMessageDrivenChannelAdapter adapter2 =
new MqttPahoMessageDrivenChannelAdapter("tcp://192.168.100.1:1883","mqtt_virtual_received_sus_2",
"DATA/#", "LD/#","CONF/#","CONFIG/#");
adapter2.setCompletionTimeout(0);
adapter2.setConverter(new DefaultPahoMessageConverter());
adapter2.setQos(2);
adapter2.setOutputChannel(mqttInputChannel() );
MqttPahoMessageDrivenChannelAdapter adapter =
new MqttPahoMessageDrivenChannelAdapter("tcp://192.168.100.14:1883","mqtt_virtual_received_sus_1",
"DATA/#", "LD/#","CONF/#","CONFIG/#");
adapter.setCompletionTimeout(0);
adapter.setConverter(new DefaultPahoMessageConverter());
adapter.setQos(2);
adapter.setOutputChannel(mqttInputChannel() );
logConfList.add(adapter);
logConfList.add(adapter2);
return logConfList;
}
is there any way i can manage these beans dynamically, where i can fetch mqtt server details from text file and in a for loop or something i can manage multiple connections.
See Dynamic and runtime Integration Flows.
#Autowired
private IntegrationFlowContext flowContext;
private IntegrationFlowRegistration addAnAdapter(String uri, String clientId, MessageChannel channel,
String... topics) {
MqttPahoMessageDrivenChannelAdapter adapter = new MqttPahoMessageDrivenChannelAdapter(uri, clientId, topics);
// more adapter configuration
IntegrationFlow flow = IntegrationFlows.from(adapter)
.channel(channel)
.get();
return this.flowContext.registration(flow).register();
}
private void removeAdapter(IntegrationFlowRegistration flowReg) {
this.flowContext.remove(flowReg.getId());
}

Spring Integration 4 asynchronous request/response

I am trying to write a simple message flow using Spring Integration v4's DSL APIs which would look like this:
-> in.ch -> Processing -> JmsGatewayOut -> JMS_OUT_QUEUE
Gateway
<- out.ch <- Processing <- JmsGatewayIn <- JMS_IN_QUEUE
With the request/response being asynchronous, when I inject a message via the initial Gateway, the message goes all the way to JMS_OUT_QUEUE. Beyond this message flow, a reply message is put back into JMS_IN_QUEUE which it is then picked up by JmsGatewayIn. At this point, the message is Processed and placed into out.ch (I know the response gets to out.ch because I have a logger interceptor there which logs the message being placed there) but, the Gateway never receives the response.
Instead of a response, the system outside of this message flow which picked up the message from JMS_OUT_QUEUE and placed the response in JMS_IN_QUEUE, receives a javax.jms.MessageFormatException: MQJMS1061: Unable to deserialize object on its own JmsOutboundgateway (I think it is failing to deserialize a jms reply object from looking at the logs).
I have clearly not got something configured correctly but I don't know exactly what. Does anyone know what I am missing?
Working with spring-integration-core-4.0.3.RELEASE, spring-integration-jms-4.0.3.RELEASE, spring-integration-java-dsl-1.0.0.M2, spring-jms-4.0.6.RELEASE.
My Gateway is configured as follows:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyChannel = "out.ch",
replyTimeout = 45000)
AResponse process(ARequest request);
}
My Integration flow is configured as follows:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Bean(name = "out.ch")
public DirectChannel outCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.correlationKey("JMSCorrelationID")
.get();
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(Jms.inboundGateway(mqConnectionFactory)
.destination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.channel("out.ch")
.get();
}
}
Thanks for any help on this,
PM.
First of all your configuration is bad:
Since you start the flow from WsGateway#process you really should wait reply there.
The gateway's request/reply capability is based on TemporaryReplyChannel, which is placed to the headers as non-serializable value.
As long as you wait rely on that gateway, actually there is no reason to provide the replyChannel, if you aren't going to do some publish-subscribe logic on the reply.
As you send message to the JMS queue, you should understand that consumer part might be a separete remote application. And the last one might know nothing about your out.ch.
The JMS request/reply capability is really based on JMSCorrelationID, but it isn't enough. The one more thing here is a ReplyTo JMS header. Hence, if you are going to send reply from the consumer you should really just rely on the JmsGatewayIn stuff.
So I'd change your code to this:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyTimeout = 45000)
AResponse process(ARequest request);
}
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.replyDestination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.get();
}
}
Let me know, if it is appropriate for you or try to explian why you use two-way gateways for one one-way cases. Maybe Jms.outboundAdapter() and Jms.inboundAdapter() are more good for you?
UPDATE
How to use <header-channels-to-string> from Java DSL:
.enrichHeaders(e -> e.headerChannelsToString())

Resources