Abstracting Spring Cloud Stream Producer and Consumer code - spring-integration

I have a Service that is Producing and Consuming messages from different Spring Cloud Stream Channels (bound to EventHub/Kafka topics). There are several such Services which are setup similarly.
The configuration looks like below
public interface MessageStreams {
String WORKSPACE = "workspace";
String UPLOADNOTIFICATION = "uploadnotification";
String BLOBNOTIFICATION = "blobnotification";
String INGESTIONSTATUS = "ingestionstatusproducer";
#Input(WORKSPACE)
SubscribableChannel workspaceChannel();
#Output(UPLOADNOTIFICATION)
MessageChannel uploadNotificationChannel();
#Input(BLOBNOTIFICATION)
SubscribableChannel blobNotificationChannel();
#Output(INGESTIONSTATUS)
MessageChannel ingestionStatusChannel();
}
#EnableBinding(MessageStreams.class)
public class EventHubStreamsConfiguration {
}
The Producer/Publisher code looks like below
#Service
#Slf4j
public class IngestionStatusEventPublisher {
private final MessageStreams messageStreams;
public IngestionStatusEventPublisher(MessageStreams messageStreams) {
this.messageStreams = messageStreams;
}
public void sendIngestionStatusEvent() {
log.info("Sending ingestion status event");
System.out.println("Sending ingestion status event");
MessageChannel messageChannel = messageStreams.ingestionStatusChannel();
boolean messageSent = messageChannel.send(MessageBuilder
.withPayload(IngestionStatusMessage.builder()
.correlationId("some-correlation-id")
.status("done")
.source("some-source")
.eventTime(OffsetDateTime.now())
.build())
.setHeader("tenant-id", "some-tenant")
.build());
log.info("Ingestion status event sent successfully {}", messageSent);
}
}
Similarly I have multiple other Publishers which publish to different Event Hubs/Topics. Notice that there is a tenant-id header being set for each published message. This is something specific to my multi-tenant application to track the tenant context. Also notice that I am getting the channel to be published to while sending the message.
My Consumer code looks like below
#Component
#Slf4j
public class IngestionStatusEventHandler {
private AtomicInteger eventCount = new AtomicInteger();
#StreamListener(TestMessageStreams.INGESTIONSTATUS)
public void handleEvent(#Payload IngestionStatusMessage message, #Header(name = "tenant-id") String tenantId) throws Exception {
log.info("New ingestion status event received: {} in Consumer: {}", message, Thread.currentThread().getName());
// set the tenant context as thread local from the header.
}
Again I have several such consumers and also there is a tenant context that is set in each consumer based on the incoming tenant-id header that is sent by the Publisher.
My questions is
How do I get rid of the boiler plate code of setting the tenant-id header in Publisher and setting the tenant context in the Consumer by abstracting it into a library which could be included in all the different Services that I have.
Also, is there a way of dynamically identifying the Channel based on the Type of the Message being published. for ex IngestionStatusMessage.class in the given scenario

To set and tenant-id header in the common code and to avoid its copy/pasting in every microservice you can use a ChannelInterceptor and make it as global one with a #GlobalChannelInterceptor and its patterns option.
See more info in Spring Integration: https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/core.html#channel-interceptors
https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/overview.html#configuration-enable-integration
You can't make a channel selection by the payload type because the payload type is really determined from the #StreamListener method signature.
You can try to have a general #Router with a Message<?> expectation and then return a particular channel name to route according that request message context.
See https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/message-routing.html#messaging-routing-chapter

Related

Send messages from a BlockingQueue through a dynamically created web socket using Spring Integration

I have a SpringBoot application that also uses SpringIntegration to dynamically create web sockets upon requests that come from an Angular web application and send messages on that web sockets that are taken from a BlockingQueue.
I want those messages to be pushed on the web socket only when a message is available on the BlockingQueue, so without polling.
At the moment, my code looks like below, but it is very inefficient:
#Service
public class WebSocketPublisherService {
#Autowired
private IntegrationFlowContext integrationFlowContext;
#Bean
public HandshakeHandler handshakeHandler() {
return new DefaultHandshakeHandler(new TomcatRequestUpgradeStrategy());
}
public void createWebSocket() {
ServerWebSocketContainer serverWebSocketContainer = new ServerWebSocketContainer("/test")
.setHandshakeHandler(handshakeHandler())
.setAllowedOrigins("http://localhost:4200");
serverWebSocketContainer.setMessageListener(...); // I want messages comming from the web application to be processed
WebSocketOutboundMessageHandler webSocketOutboundMessageHandler = new WebSocketOutboundMessageHandler(serverWebSocketContainer);
MethodInvokingMessageSource methodInvokingMessageSource = new MethodInvokingMessageSource();
methodInvokingMessageSource.setObject(...);
methodInvokingMessageSource.setMethodName(...); // this method returns the element from the BlockingQueue, if it exists
StandardIntegrationFlow standardIntegrationFlow = IntegrationFlow
.from(methodInvokingMessageSource, polling -> polling.poller(pollerFactory -> pollerFactory.fixedRate(10)))
.split(new DecorateMessageWithSessionId(serverWebSocketContainer))
.handle(webSocketOutboundMessageHandler)
.get();
IntegrationFlowContext.IntegrationFlowRegistration flowRegistration = integrationFlowContext
.registration(standardIntegrationFlow)
.addBean(serverWebSocketContainer)
.register();
flowRegistration.start();
}
}
Is there a smarter way of creating an IntegrationFlow using the content of a BlockingQueue (based on subscription and not polling)?

Spring Integration aws Kinesis , message aggregator, Release Strategy

this is a follow-up question to Spring Integration AWS RabbitMQ Kinesis
I have the following configuration. I am noticing that when I send a message to the input channel named kinesisSendChannel for the first time, the aggregator and release strategy is getting invoked and messages are sent to Kinesis Streams. I put debug breakpoints at different places and could verify this behavior. But when I again publish messages to the same input channel the release strategy and the outbound processor are not getting invoked and messages are not sent to the Kinesis. I am not sure why the aggregator flow is getting invoked only the first time and not for subsequent messages. For testing purpose , the TimeoutCountSequenceSizeReleaseStrategy is set with count as 1 & time as 60 seconds. There is no specific MessageStore used. Could you help identify the issue?
#Bean(name = "kinesisSendChannel")
public MessageChannel kinesisSendChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "resultChannel")
public MessageChannel resultChannel() {
return MessageChannels.direct().get();
}
#Bean
#ServiceActivator(inputChannel = "kinesisSendChannel")
public MessageHandler aggregator(TestMessageProcessor messageProcessor,
MessageChannel resultChannel,
TimeoutCountSequenceSizeReleaseStrategy timeoutCountSequenceSizeReleaseStrategy) {
AggregatingMessageHandler handler = new AggregatingMessageHandler(messageProcessor);
handler.setCorrelationStrategy(new ExpressionEvaluatingCorrelationStrategy("headers['foo']"));
handler.setReleaseStrategy(timeoutCountSequenceSizeReleaseStrategy);
handler.setOutputProcessor(messageProcessor);
handler.setOutputChannel(resultChannel);
return handler;
}
#Bean
#ServiceActivator(inputChannel = "resultChannel")
public MessageHandler kinesisMessageHandler1(#Qualifier("successChannel") MessageChannel successChannel,
#Qualifier("errorChannel") MessageChannel errorChannel, final AmazonKinesisAsync amazonKinesis) {
KinesisMessageHandler kinesisMessageHandler = new KinesisMessageHandler(amazonKinesis);
kinesisMessageHandler.setSync(true);
kinesisMessageHandler.setOutputChannel(successChannel);
kinesisMessageHandler.setFailureChannel(errorChannel);
return kinesisMessageHandler;
}
public class TestMessageProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected Object aggregatePayloads(MessageGroup group, Map<String, Object> defaultHeaders) {
final PutRecordsRequest putRecordsRequest = new PutRecordsRequest().withStreamName("test-stream");
final List<PutRecordsRequestEntry> putRecordsRequestEntry = group.getMessages().stream()
.map(message -> (PutRecordsRequestEntry) message.getPayload()).collect(Collectors.toList());
putRecordsRequest.withRecords(putRecordsRequestEntry);
return putRecordsRequestEntry;
}
}
I believe the problem is here handler.setCorrelationStrategy(new ExpressionEvaluatingCorrelationStrategy("headers['foo']"));. All your messages come with the same foo header. So, all of them form the same message group. As long as you release group and don’t remove it, all the new messages are going to be discarded.
Please, revise aggregator documentation to make yourself familiar with all the possible behavior : https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator

Reply Channel for Messaging Gateway using Java DSL

I have a REST API which receives a POST request from a client application.
#Autowired
private EdiTranslationEvents.TransformToString transformToString;
#PostMapping("/testPost")
#ResponseStatus(HttpStatus.OK)
public String testPostLogging(#RequestBody Student student) {
log.info("In controller....");
System.out.println(this.transformToString.objectToInputGateway(student));
return "testPost logging";
}
As you can see, in the controller, I have an autowired messaging gateway and I am using it to send data to a channel.
#Configuration
#EnableIntegration
public class EdiTranslationEvents {
#Component
#MessagingGateway
public interface TransformToString {
#Gateway(requestChannel = "inputObjectChannel")
String objectToInputGateway(Student student);
}
#Bean
public IntegrationFlow inputObjectString() {
return IntegrationFlows.from(inputObjectChannel())
.transform(Transformers.objectToString())
.log(LoggingHandler.Level.DEBUG, "com.dash.Logger")
.get();
}
}
When I send data to the REST API, the API just hangs. The gateway does not return anything. I have the return type specified, and I was assuming that the gateway creates a temporary reply channel and sends the response to that channel.
However, I am not doing anything in the DSL configuration for creating or managing a reply. So, how do I send a reply back to the reply channel from the DSL flow?
Your current flow does not return a value, you are simply logging the message.
A terminating .log() ends the flow.
Delete the .log() element so the result of the transform will automatically be routed back to the gateway.
Or add a .bridge() (a bridge to nowhere) after the log and it will bridge the output to the reply channel.

Spring Integration one channel for multiple producers and consumers

I have this direct channel:
#Bean
public DirectChannel emailingChannel() {
return MessageChannels
.direct( "emailingChannel")
.get();
}
Can I define multiple flows for the same channel like this:
#Bean
public IntegrationFlow flow1FromEmailingChannel() {
return IntegrationFlows.from( "emailingChannel" )
.handle( "myService" , "handler1")
.get();
}
#Bean
public IntegrationFlow flow2FromEmailingChannel() {
return IntegrationFlows.from( "emailingChannel" )
.handle( "myService" , "handler2" )
.get();
}
EDIT
#Service
public class MyService {
public void handler1(Message<String> message){
....
}
public void handler2(Message<List<String>> message){
....
}
}
Each flow's handle(...) method manipulates different payload data types but the goal is the same, i-e reading data from the channel and call the relevant handler. I would like to avoid many if...else to check the data type in one handler.
Additional question: What happens when multiple threads call the same channel (no matter its type: Direct, PubSub or Queue) at the same time (as per default a #Bean has singleton scope)?
Thanks a lot
With a direct channel messages will be round-robin distributed to the consumers.
With a queue channel only one consumer will get each message; the distribution will be based on their respective pollers.
With a pub/sub channel both consumers will get each message.
You need to provide more information but it sounds like you need to add a payload type router to your flow to direct the messages to the right consumer.
EDIT
When the handler methods are in the same class you don't need two flows; the framework will examine the methods and, as long as there is no ambiguity) will call the method that matches the payload type.
.handle(myServiceBean())

Spring Integration 4 asynchronous request/response

I am trying to write a simple message flow using Spring Integration v4's DSL APIs which would look like this:
-> in.ch -> Processing -> JmsGatewayOut -> JMS_OUT_QUEUE
Gateway
<- out.ch <- Processing <- JmsGatewayIn <- JMS_IN_QUEUE
With the request/response being asynchronous, when I inject a message via the initial Gateway, the message goes all the way to JMS_OUT_QUEUE. Beyond this message flow, a reply message is put back into JMS_IN_QUEUE which it is then picked up by JmsGatewayIn. At this point, the message is Processed and placed into out.ch (I know the response gets to out.ch because I have a logger interceptor there which logs the message being placed there) but, the Gateway never receives the response.
Instead of a response, the system outside of this message flow which picked up the message from JMS_OUT_QUEUE and placed the response in JMS_IN_QUEUE, receives a javax.jms.MessageFormatException: MQJMS1061: Unable to deserialize object on its own JmsOutboundgateway (I think it is failing to deserialize a jms reply object from looking at the logs).
I have clearly not got something configured correctly but I don't know exactly what. Does anyone know what I am missing?
Working with spring-integration-core-4.0.3.RELEASE, spring-integration-jms-4.0.3.RELEASE, spring-integration-java-dsl-1.0.0.M2, spring-jms-4.0.6.RELEASE.
My Gateway is configured as follows:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyChannel = "out.ch",
replyTimeout = 45000)
AResponse process(ARequest request);
}
My Integration flow is configured as follows:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Bean(name = "out.ch")
public DirectChannel outCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.correlationKey("JMSCorrelationID")
.get();
}
#Bean
public IntegrationFlow responseFlow() {
return IntegrationFlows.from(Jms.inboundGateway(mqConnectionFactory)
.destination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.channel("out.ch")
.get();
}
}
Thanks for any help on this,
PM.
First of all your configuration is bad:
Since you start the flow from WsGateway#process you really should wait reply there.
The gateway's request/reply capability is based on TemporaryReplyChannel, which is placed to the headers as non-serializable value.
As long as you wait rely on that gateway, actually there is no reason to provide the replyChannel, if you aren't going to do some publish-subscribe logic on the reply.
As you send message to the JMS queue, you should understand that consumer part might be a separete remote application. And the last one might know nothing about your out.ch.
The JMS request/reply capability is really based on JMSCorrelationID, but it isn't enough. The one more thing here is a ReplyTo JMS header. Hence, if you are going to send reply from the consumer you should really just rely on the JmsGatewayIn stuff.
So I'd change your code to this:
#MessagingGateway
public interface WsGateway {
#Gateway(requestChannel = "in.ch", replyTimeout = 45000)
AResponse process(ARequest request);
}
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class IntegrationConfig {
#Bean(name = "in.ch")
public DirectChannel inCh() {
return new DirectChannel();
}
#Autowired
private MQQueueConnectionFactory mqConnectionFactory;
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows.from("in.ch")
.handle("processor", "processARequest")
.handle(Jms.outboundGateway(mqConnectionFactory)
.requestDestination("JMS_OUT_QUEUE")
.replyDestination("JMS_IN_QUEUE"))
.handle("processor", "processAResponse")
.get();
}
}
Let me know, if it is appropriate for you or try to explian why you use two-way gateways for one one-way cases. Maybe Jms.outboundAdapter() and Jms.inboundAdapter() are more good for you?
UPDATE
How to use <header-channels-to-string> from Java DSL:
.enrichHeaders(e -> e.headerChannelsToString())

Resources