Spring Integration Multiplexing correlation - spring-integration

I am new to SI. I am using the code from the SI TCP Multiplexing example as a starting point for an app server I am writing. The caller of the service already exists and will be sending the payload prefixed by a byte length header. I am having a bit of trouble with the correlation of the response. As you can see below, I changed the Multiplexing example to first add a correlation id header to the incoming request before pushing on to the publish-subscribe-channel. The rest of the code is pretty much the same as the example.
So, the problem. The correlation id header is not available on the call to MessageController from the TcpSendingMessageHandler which serializes the message payload and sends it. Should I enrich the payload to include the correlation id (no correlation header) or is there a simpler way of doing all of this? Any guidance would be greatly appreciated.
<gateway id="gw"
service-interface="is.point.tokens.server.MessageGateway"
default-request-channel="input">
</gateway>
<ip:tcp-connection-factory id="client"
type="client"
host="${tcpClientServer.address}"
port="${tcpClientServer.port}"
single-use="false"
serializer="bigEndianFormatSerializer"
deserializer="bigEndianFormatSerializer"
so-timeout="10000"/>
<channel id="input" datatype="java.lang.String"/>
<header-enricher input-channel="input" output-channel="enriched.input">
<correlation-id expression="headers['id']"/>
</header-enricher>
<publish-subscribe-channel id="enriched.input"/>
<ip:tcp-outbound-channel-adapter id="outAdapter.client"
order="2"
channel="enriched.input"
connection-factory="client"/>
<!-- Collaborator -->
<!-- Also send a copy to the custom aggregator for correlation and
so this message's replyChannel will be transferred to the
aggregated message. The order ensures this gets to the aggregator first -->
<bridge input-channel="enriched.input" output-channel="toAggregator.client" order="1"/>
<!-- Asynchronously receive reply. -->
<ip:tcp-inbound-channel-adapter id="inAdapter.client"
channel="toAggregator.client"
connection-factory="client"/>
<!-- Collaborator -->
<channel id="toAggregator.client" datatype="java.lang.String"/>
<aggregator input-channel="toAggregator.client"
output-channel="toTransformer.client"
correlation-strategy-expression="headers.get('correlationId')"
release-strategy-expression="size() == 2">
</aggregator>
<!-- The response is always second -->
<transformer input-channel="toTransformer.client" expression="payload.get(1)"/>
<!-- Server side -->
<ip:tcp-connection-factory id="server"
type="server"
port="${tcpClientServer.port}"
using-nio="true"
serializer="bigEndianFormatSerializer"
deserializer="bigEndianFormatSerializer"/>
<ip:tcp-inbound-channel-adapter id="inAdapter.server"
channel="toSA"
connection-factory="server" />
<channel id="toSA" datatype="java.lang.String"/>
<service-activator input-channel="toSA"
output-channel="toObAdapter"
ref="messageController"
method="handleMessage"/>
<beans:bean id="messageController"
class="example.server.MessageController"/>
<channel id="toObAdapter"/>
<ip:tcp-outbound-channel-adapter id="outAdapter.server"
channel="toObAdapter"
connection-factory="server"/>

Yes, you need something in the data so the reply can be correlated.
But I am confused. If you are the server side for an existing application, then you can simply use an inbound gateway.
If you really are providing the client and server side, we did add a mechanism in 3.0 to selectively add headers (to the tcp message), for example, using JSON.
EDIT:
From your comments, you only need the server side. All you need is this...
<int-ip:tcp-connection-factory id="server"
type="server"
serializer="serializer"
deserializer="serializer"
port="${port}"/>
<int-ip:tcp-inbound-gateway id="gateway"
connection-factory="crLfServer"
request-channel="toSA"
error-channel="errorChannel"/>
<int:channel id="toSA" />
<int:service-activator input-channel="toSA"
ref="messageController"
method="handleMessage"/>
<bean id="serializer" class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer" />
The fact that the service-activator has no output-channel means the framework will send the reply to the gateway automatically.
This assumes the service can handle a byte[]; if you need a String, you'll need a transformer like in the sample. Either way, the payload will not include the length header; it is stripped off (inbound) and added (outbound).
This also assumes the length header is 4 bytes (default); the serializer takes a size in a constructor...
<bean id="serializer" class="org.springframework.integration.ip.tcp.serializer.ByteArrayLengthHeaderSerializer">
<constructor-arg value="2" />
</bean>

Related

Spring Integration Aggregator Continuing to output channel without release-strategy-expression being True

I asked a question a bit ago about a specific use case when multi-threading in spring integration, basically I have to have one thread of execution that stays on the initial and then a second that spawns a thread. I have implemented this like so
<int:channel id="newThread" >
<int:dispatcher task-executor="workerThreadPoolAdapter"/>
</int:channel>
<!--This bean is used to split incoming messages -->
<bean id="splitterBean" class="orchestration.MessageSplitter">
<property name="channels" ref="splitterChannelsList" />
</bean>
<util:list id="splitterChannelsList" value-type="java.lang.String">
<value>newThread</value>
<value>mainThread</value>
</util:list>
<!-- This bean is used to aggregate incoming messages -->
<bean id="aggregator" class="orchestration.MessageAggregator">
<property name="wrapperNode" value="container" />
</bean>
<!-- Channel for aggregator output and that will be input for response transformer -->
<int:publish-subscribe-channel id="gatherChannel" apply-sequence="true"/>
<!-- This splitter splits request and send -->
<!-- will add a header called channelHeader which is the channel the message should be routed to using the recipient-list-router -->
<int:splitter input-channel="splitter"
output-channel="recipientListRouter"
apply-sequence="true"
ref="splitterBean" method="split" />
<!-- Aggregator that aggregates responses received from the calls -->
<int:aggregator input-channel="gatherChannel"
output-channel="transformResponse"
ref="aggregator"
method="aggregateMessages"
send-partial-result-on-expiry="false"
expire-groups-upon-completion="true"
message-store="removeMessageFromStore"
release-strategy-expression="size() == 2"/>
<int:recipient-list-router input-channel="recipientListRouter">
<int:recipient channel="mainThread"
selector-expression="headers.get('channelHeader') == 'mainThread'"/>
<int:recipient channel="newThread"
selector-expression="headers.get('channelHeader') == 'newThread'"/>
</int:recipient-list-router>
<!-- only route to call if boolean is populated -->
<int:header-value-router header-name="shouldMakeExtraOutboundCall"
input-channel="newThread"
default-output-channel="gatherChannel" >
<int:mapping value="true" channel="outboundCall" />
</int:header-value-router>
<int:chain input-channel="outboundCall" output-channel="gatherChannel">
<!-- make an outbound call -->
</int:chain>
<int:chain input-channel="mainThread" output-channel="gatherChannel">
<!-- make a bunch of outbound calls -->
</int:chain>
<int:chain input-channel="transformResponse" output-channel="backToClient">
<!-- do some stuff and respond back to client -->
</int:chain>
I've had the output channel of the aggregator as both a publish-subscribe and a direct channel and had the issue for both.
When I look at the logs I can see that one of the threads has a preSend to the 'gatherChannel' then AggregatingMessageHandler saying it received the message, then another log from AggregatingMessageHandler saying Handling message with correlationKey [f2b16b6a-3605-778f-a628-870ed8ce3f5e] then a postSend (sent=True) on channel 'gatherChannel'.
I thought that it would not send to the transformResponse channel until both messages that got split out from the splitter got to it. I even added the size() == 2 as the release strategy expression as an extra layer but that doesn't seem to be causing the aggregator to wait either.
I'm a little perplexed why this is happening, it's happening when both the main thread or the spawned thread gets to the aggregator, I'm trying to figure out how to get that aggregator to wait to send to the output channel until BOTH messages that were split from the splitter are received.
According to your current configuration it really does not happen.
Do you really have some evidences that first message in the group is really sent to that transformResponse unconditionally?
You probably is a bit confused with those preSend and postSend since an aggregator is really non-blocking until release. It does accept the message and store it into the MessageStore for grouping if release condition is false. Just after this it returns immediately to the caller, which is that gatherChannel and therefore your see postSend on a first message.

AWS Async Response routing to success-channel for sqs-outbound-channel-adapter

I have registered an AsyncHandler and also added a success-channel to an SQS outbound flow. The success-channel has a int:logging-channel-adapter endpoint. However I am not able to see any logs from this adapter. The AsyncHandler is able to receive the call-backs but nothing on the success-channel.
In the SqsMessageHandler I see that we are setting an output channel in the obtainAsyncHandler method, but I did not see the success-channel set anywhere. Am I missing something?
I would prefer using the success and failure channels and not AsyncHandler call-back Impl to avoid having AWS specific code in my classes.
Also my <int-aws:sqs-outbound-channel-adapter> is inside a <int:chain> which has no output channel, since the flow ends when the message is sent.
EDIT - Added Config
This is the only way I can get it to log the callback.
<int:channel id="chainChannel" />
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" output-channel="chainChannel">
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="chainChannel" />
Here I can just use the same channel in both chain (outbound channel) and sqs-outbound (success-channel)
Unable to get it to work like below:
<int:channel id="successChannel" />
<bean class="ServiceTransformer" id="serviceTransformer" />
<int:chain input-channel="serviceChannel" id="sendToServiceSqsChain" >
<int:transformer ref="serviceTransformer" method="transform" />
<int:header-filter header-names="config" />
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS" queue="some-queue" async-handler="sqsPublishCallbackHandler" success-channel="successChannel"/>
</int:chain>
<int:logging-channel-adapter log-full-message="true" channel="successChannel" />
The <int-aws:sqs-outbound-channel-adapter> component is one-way, therefore there is no outputChannel option expose. However the target class is AbstractMessageProducingHandler. To avoid code duplication we reuse an existing outputChannel internally for that AsyncHandler.
In the XML parser we just remap one to another:
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(builder, element, "success-channel", "outputChannel");
You probably don't see anything in logs because you need to adjust logging config respectively for the appropriate category and level.
UPDATE
According my testing this is definitely not possible to configure such a component with XML DSL within the <chain>. That <int-aws:sqs-outbound-channel-adapter> has to be presented outside of the <chain>.
Consider to more your configuration to Java DSL instead: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/dsl.html#java-dsl.

spring integration - splitter and aggregator

Currently am working with spring integration for new application and started poc to know how to handle the failure cases.
In my application spring integration will receive message from IBM mq and validate the header information and route to different queue depends on the message type. incoming message could be bulk message, so i've used splitter and aggregator from spring integration and am having good progress and control over the technical workflow.
Currently am facing few issues, we have IBM mq and also webservice as our gateway. Both gateway receive the message and send to splitter channel where splitter splits the message and send to outbound channel( executor channel). so message will be send to destination in parallel and status update service activator will receive the message with same channel with order=2 and send to aggregator. so for its good with implementation.
Problem:
if jms outbound gateway throws the execption i've added advise as exception handler which wil send to another service activator to update failure status to DTO object and will have same aggregator channel as output but am not receiving the message in aggregator channel in this case and aggregator receive only in happy flow.
I want to aggregate the outbound successful message and failure message(other service activator update the status) and then the complete status needs to posted to response queue as another outbound or as response in webservice.
i tried to have ordered succesful service activator and failure error handler service activator to have same channel which is input channel for aggregator and its not working.
Appreciated for your guidance to proceed with this workflow
using Spring Integration 2.2.2
<channel id="inbound"/>
<channel id="splitterInChannel"/>
<channel id="splitterOutChannel">
<dispatcher task-executor="splitterExecutor"/>
</channel>
<beans:bean id="splitterExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<beans:property name="corePoolSize" value="5" />
<beans:property name="maxPoolSize" value="10" />
<beans:property name="queueCapacity" value="25" />
</beans:bean>
<channel id="ValidatorChannel"/>
<channel id="outBoundjmsChannel"/>
<channel id="outBoundErrorChannel"/>
<channel id="finalOutputChannel"></channel>
<channel id="aggregatorChannel"/>
<jms:inbound-channel-adapter connection-factory="AMQConnectionFactory"
destination="AMQueue" channel="inbound" auto-startup="true"
extract-payload="false" acknowledge="transacted"></jms:inbound-channel-adapter>
<service-activator ref="InBoundProcessor" input-channel="inbound" output-channel="splitterInChannel"></service-activator>
<!-- splitter -->
<splitter ref="Splitter" method="splitInput" input-channel="splitterInChannel" output-channel="splitterOutChannel"/>
<!-- validator -->
<service-activator ref="Validator" method="validate" input-channel="splitterOutChannel" output-channel="ValidatorChannel"/>
<!-- need to add enricher -->
<service-activator ref="Enricher" method="enrich" input-channel="ValidatorChannel" output-channel="outBoundjmsChannel"/>
<!-- outbound gateway -->
<jms:outbound-channel-adapter channel="outBoundjmsChannel" connection-factory="AMQConnectionFactory" destination-name="outputQueue"
message-converter="customMessageConvertor" order="1" >
<jms:request-handler-advice-chain>
<beans:bean class="org.springframework.integration.handler.advice.RequestHandlerRetryAdvice">
<beans:property name="retryTemplate" >
<beans:bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
<beans:property name="retryPolicy">
<beans:bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<beans:property name="maxAttempts" value="2" />
</beans:bean>
</beans:property>
<beans:property name="backOffPolicy">
<beans:bean class="org.springframework.retry.backoff.FixedBackOffPolicy">
<beans:property name="backOffPeriod" value="1000" />
</beans:bean>
</beans:property>
</beans:bean>
</beans:property>
<beans:property name="recoveryCallback">
<beans:bean class="org.springframework.integration.handler.advice.ErrorMessageSendingRecoverer">
<beans:constructor-arg ref="outBoundErrorChannel" />
</beans:bean>
</beans:property>
</beans:bean>
</jms:request-handler-advice-chain>
</jms:outbound-channel-adapter>
<!-- outBound error processor -->
<service-activator ref="ErrorProcessor" method="errorHandling" input-channel="outBoundErrorChannel" output-channel="aggregatorChannel" />
<!-- Post send processor -->
<service-activator ref="PostProcessor" method="Postprocessing" input-channel="outBoundjmsChannel" output-channel="aggregatorChannel" order="2"/>
<!-- aggregator -->
<aggregator ref="Aggregator" correlation-strategy-method="aggregateStrategy" input-channel="aggregatorChannel" output-channel="finalOutputChannel"
release-strategy-method="isRelease" method="aggregate" expire-groups-upon-completion="true"/>
<!-- final processor or responder -->
<service-activator ref="FinalProcessor" method="endProcessing" input-channel="finalOutputChannel"/>
</beans:beans>
In the above configuration as of now i've given the release strategy as false and correlation method as empty string if this works, i will generate UUID for the batch and will attach the UUID in splitter to corrlate.
when debugging the above configuration i noticed the outbound error channel receive whenever it attempts to send to the outbound adapter(in my case its send twice ). I don't want to make an reattempt in one of the application and in another application it needs to attempt for reposting the message.
In both case i want to send the message to outbound error channel after the final attempt to aggregate, if fails i will update the status in ErrorProcessor as failed to send.
Two issues.
1. am receiving duplicate message to the channel and difficult to identify the last failure or success.
2.Couldn't make logic for release strategy and difficult to identify which is the duplicate and whether its successful or not.
In the above case i couldn't find a generic way to compare objects because equals method doesn't have proper attributes to compare and it will not be
correct way to compare with boolean field.
please help me out to resolve this issue to proceed my workflow design and completion.
Much appreciated for guiding me to proceed.
Thanks,
Krish S
currently
public Object errorHandling(Object object){
OutBoundMessage outBoundMessage = null;
if(object instanceof MessagingException){
outBoundMessage =((MessagingException) object).getFailedMessage();
}else{
//TODO: log the message
}
return outBoundMessage;
}
public String aggregateStrategy(OutBoundMessage outBoundMessage){
//TODO: get the UUID from outbound message and return
return "";
}
public List<OutBoundMessage> splitter(InBoundMessage inBoundMessage){
String[] message = inBoundMessage.getRawMessage().split(",");
long uuid = java.util.UUID.randomUUID().getLeastSignificantBits();
List<OutBoundMessage> outBoundMessagelist = new ArrayList<OutBoundMessage>();
for (String string : message) {
OutBoundMessage outBoundMessage = new outBoundMessage();
outBoundMessage.setCorrelationID(uuid);
outBoundMessagelist.add(outBoundMessage);
}
}
Added as default false in the following method to validate
public boolean isRelease(List<OutBoundMessage> outBoundMessage){
//TODO: need to define condition for closing the list aggregation
return false;
}
Please, share your ErrorProcessor source code. And correlation-strategy-method="aggregateStrategy" as well.
I would like to know how you deal with ErrorMessage there and how you restore correlationKey from the message after your ErrorProcessor.
Not sure how you build your own correlationKey, but the <splitter> provide applySequence = true by default. So, the Correlation Details are available in each splitted message to be able to aggregate afterwards.
For your ErrorMessage from the ErrorMessageSendingRecoverer I can recommend to pay attention to the Exception payload there. It looks like (from the ErrorMessageSendingRecoverer source code):
else if (!(lastThrowable instanceof MessagingException)) {
lastThrowable = new MessagingException((Message<?>) context.getAttribute("message"),
lastThrowable.getMessage(), lastThrowable);
}
....
messagingTemplate.send(new ErrorMessage(lastThrowable));
So, that MessagingException, has a "guilty" message for the Exception and exactly that message has an appropriate Correlation Details headers for aggregator. Therefore you should rely on them if you'd like to aggregate errors to the same message group.
Finally I understood how it works,
I've a boolean set to true in message convertor and in Errorhandle I set it to false and return null so the recovery is that message is received as failed message to aggregator and understood what happens when I return the object
Thanks #ArtemBilan, your code block gave me an insight of what's happening and what should I do

Extract XML payload from http inbound gateway and supply to xslt transformer spring integration

I have requirement wherein I need to expose a restful service with XML payload. After that I need to get hold of xml payload and transform it to different xml using xslt transformer.
I am struggling how to get hold of xml payload that can act as a input to the xslt transformer.
I want to avoid marshelling and unmarshlling overhead.
Can someone please help me on same.
Regards
Lalit
When asking questions like this, it's better if you show what you have tried and what didn't work.
In this case, all you need is an HTTP Inbound Gateway.
If the incoming content-type contains text; the payload will be String, by default. If not (e.g. application/xml) then you'll need to configure the gateway with the type you want...
request-payload-type="java.lang.String"
otherwise, the payload will be a byte[].
Based on Gary's suggestion... used following code and it worked...
<int:channel id="inputChannel"></int:channel>
<int-http:inbound-gateway request-channel="inputChannel" path="/test" supported-methods="POST">
<int-http:request-mapping consumes="application/xml" produces="application/xml" />
</int-http:inbound-gateway>
<int:chain input-channel="inputChannel">
<int:service-activator ref="sa1"></int:service-activator>
<int-xml:xslt-transformer
xsl-resource="classpath:/testTransformer.xsl"/>
<int:service-activator ref="sa2"></int:service-activator>
</int:chain>
<!-- input type is byte[] -->
<bean id="sa1" class="com.fidintl.integration.ServiceActivator1">
</bean>
<!-- input type is String -->
<bean id="sa2" class="com.fidintl.integration.ServiceActivator2"></bean>
It looks like in post (as Gary mentioned) output is byte[] which I converted to string in ServiceActivator1
Regards,
Lalit Kumar

Generalize http:outbound-gateway reply-channel

Given a gateway that handles service calls to a ws. My goal is to supply the http:outbound-gateway's reply-channel using header-enricher since I'll be adding multiple methods to gateway and I would like to make use of only 1 http:outbound-gateway
I can currently receive the response up to groovy script (2) BUT it doesn't seem to want to return the results to the actual method that calls the service
Any help would be appreciated. Thanks!
<gateway id="registryService" service-interface="RegistryService">
<method name="create" request-channel="create-request-channel"
reply-channel="create-reply-channel" />
</gateway>
<chain input-channel="create-request-channel" output-channel="create-request-fulfillment-channel">
<transformer>
// groovy script that contains the method to be called in the ws (1)
</transformer>
<object-to-json-transformer/>
<header-enricher>
<reply-channel overwrite="true" ref="create-reply-fulfillment-channel" />
</header-enricher>
</chain>
<http:outbound-gateway request-channel="create-request-fulfillment-channel"
extract-request-payload="true"
expected-response-type="java.lang.String"
url="http://localhost:4567" http-method="POST" />
<chain input-channel="create-reply-fulfillment-channel"
output-channel="create-reply-channel">
<json-to-object-transformer type="JsonRpcResponse"/>
<transformer>
//groovy script to manipulate response (2)
</transformer>
</chain>
Do the following:
Each method of your gateway should enrich message with some unique 'routing' header value:
<gateway id="registryService" service-interface="RegistryService">
<method name="create" request-channel="http-request-channel"
reply-channel="registryService-create-responseChannel">
<header name="routingHeader" value="registryService-create" />
</method>
</gateway>
And then send message straight forward to outbound gateway:
<http:outbound-gateway request-channel="http-request-channel"
response-channel="http-response-channel"
extract-request-payload="true"
expected-response-type="java.lang.String"
url="http://localhost:4567" http-method="POST" />
Http outbound gateway sends request to the remote server and then forward response to http-response-channel. To this channel is attached header value router, which basis on the value of routing header, sends (routes) message to the appropriate channel:
<header-value-router input-channel="http-response-channel" header-name="routingHeader">
<mapping value="registryService-create" channel="registryService-create-responseChannel" />
<mapping value="someOtherService-otherMethod" channel="someOtherService-otherMethod-responseChannel" />
</header-value-router>
Of course you don't need to send it back directly to the gateway - you can add some additional processing between those components, and all the time you can route message basis on the header value.
It's simpler than hacks with groovy and I use it myself - proven that works ;)

Resources