Overloading MessageHistory object for auditing purpouses - spring-integration

I need to audit every message that pass throught an spring-integration flow, very basic one.
The application reads from an int-redis:queue-inbound-channel-adapter and writes in an int-http:outbound-gateway.
The output gateway has a reply-channel in order to receive the response of the http method, is in this exact moment when I want to register de audit information, including the initial message and http response.
If the http method fails, the message goes to errorHandler and I can obtain the history object with timestamps and payload, what's perfect for audit.
But if the http method works (Code 200,201 ...), I get a message including the ResponseEntity object with the status code, but I've no the initial payload from redis.
So, is there any way to corelate the two messages (before sending http outbound and after http response) in order to get the full info for auditing?
Thanks in advance!
Here the integration flow:
<int:message-history/>
<int:channel id="responseChannel"/>
<int:channel id="responseError"/>
<int:logging-channel-adapter channel="responseChannel" level="INFO"
expression="#messageHandlerSpel.handle( #this )" id="logger"/>
<int:logging-channel-adapter channel="responseError"
level="ERROR" expression="#errorHandlerSpel.handle( #this )"
id="LogingERROR"/>
<int-redis:queue-inbound-channel-adapter id="inputRedis" connection-factory="redisCF"
channel="redisInput" queue="redis-input" serializer="" error-channel="responseError"/>
<int:channel id="redisInput" />
<int-http:outbound-gateway id="HttpOutbound" request-channel="redisInput"
url="http://{server}:{port}/{index}/{type}/{id}"
http-method="PUT" extract-request-payload="true" charset="UTF-8"
reply-channel="responseChannel" request-factory="requestFactory"/>
And the handler class:
#Component
public class MessageHandlerSpel {
public String handle(Message<byte[]> message) {
MessageHeaders msgH = message.getHeaders();
MessageHistory msgHistory =
message.getHeaders().get(MessageHistory.HEADER_NAME, MessageHistory.class);
... // Do whatever
return "";
}

One trick is to store the request payload in the custom header before performing the HTTP request.
After receiving response, the request headers will be merged with that response payload. Therefore in the downstream you will have both: the request in the header, and reply in the payload.
The MessageHistory remains the same and there is no reason to hack there somehow.
Another approach is based on the <aggregator> to correlate request and reply. But this one is for more complex or distributed scenario.
The simple copy/paste to header via:
<header-enricher>
<header name="requestPayload" expressio="payload"/>
</header-enricher>
should be enough for you.

Related

Spring Integration Outbound Gate way for Synchronous REST Calls

Previously i was able to develop a little framework using the Spring Integration Where developers can specify the URLs, HTTP methods and Request Body and invoke any external REST API.
This is the configuration for my Spring Integration
<int:channel id='reply.channel'>
<int:queue capacity='10' />
</int:channel>
<int:channel id='request.channel'/>
<int:channel id='outbound.Channel'/>
<int:gateway id="outboundGateway"
service-interface="com.bst.pm.PostGateway"
default-request-channel="outbound.Channel">
</int:gateway>
<int:object-to-json-transformer input-channel="outbound.Channel" output-channel="request.channel"/>
<int-http:outbound-gateway id="outbound.gateway"
request-channel="request.channel" url-expression="headers.bstUrl"
http-method-expression="headers.bstHttpMethod" expected-response-type-expression="headers.bstExpectedResponseType"
charset="UTF-8" reply-timeout="5000" reply-channel="reply.channel"
mapped-request-headers="bst*, HTTP_REQUEST_HEADERS">
</int-http:outbound-gateway>
Then Developers can invoke external rest API calls using the above infrastruture as below
#Autowired #Qualifier("reply.channel") PollableChannel receivedChannel;
#Autowired #Qualifier("request.channel") MessageChannel getRequestChannel;
#Autowired #Qualifier("outbound.Channel") MessageChannel httpOutboundGateway;
Post post = new Post();
post.setTitle("Spring INtegration Test");
post.setBody("This is a sample request body to test Spring Integration HTTP Outbound gateway");
post.setUserId(Long.valueOf(1));
Message<?> message = MessageBuilder.withPayload(post)
.setHeader("bstUrl", "https://jsonplaceholder.typicode.com/posts")
.setHeader("bstHttpMethod", "POST")
.setHeader("bstExpectedResponseType", "com.bst.pages.crm.web.Post")
.build();
httpOutboundGateway.send(message);
Message<?> receivedMsg = receivedChannel.receive();
Post post = (Post) receivedMsg.getPayload();
System.out.println("############## ServerMsg ##############");
System.out.println(o);
System.out.println("############## Done! ##############");
In here i want to make all the REST calls via this integration infrastructure to be Synchronous. However i have used a QUEUE channel as the reply-channel for the http:outbound-gateway. Therefore as per my understanding the reply can be received by a wrong sender as any one can pool the channel for the messages.
How can we make sure the the correct sender will always receive the correct response?
Thanks,
Keth
Your is correct. Really having a global reply channel and several concurrent processes you may end up with work stealing situation.
To fix your problem you need to get rid of the reply-channel on your HTTP Gateway and just rely on your the replyChannel header populated by the Messaging Gateway. However your Gateway method should really be as a request-reply signature: it has to return Object.
See more info about replyChannel header in the Reference Manual: https://docs.spring.io/spring-integration/docs/5.0.6.RELEASE/reference/html/messaging-endpoints-chapter.html#gateway

How to config post parameters in http:outbound-gateway

We have an case that retrieve orders from our customers system by using http post call, post request include username and password, and we can get order contents response then.
We want to call customer's service every 10 minutes, and then process the response in our service activator, however, I don't know how to config username and password as post parameters in spring integration configuration, anyone can help?
my current configuration
<int:inbound-channel-adapter channel="inChannel" expression="''">
<int:poller fixed-delay="60000"></int:poller>
</int:inbound-channel-adapter>
<int:channel id="inChannel"/>
<int:channel id="outChannel"/>
<http:outbound-gateway
url="http://****/vendorServer/order"
request-channel="inChannel" reply-channel="outChannel" http-method="GET" expected-response-type="java.lang.String">
</http:outbound-gateway>
<int:service-activator input-channel="outChannel" ref="orderService"/>
how to config username and password as post parameters in above configuration?
The HTTP POST method implies the body. The organic way to provide the body in Spring Integration is message payload. Therefore you should consider to build a Map payload with required properties before sending to the http:outbound-Gateway

Using QueueChannel in Spring Integration

I am working on the below case study.
Create a Rest Service to accept Reference id.
Use the Reference Id to get data (CLOB) from Database.
Put the data(CLOB) in a channel(queue) for further processing.
Reply to the rest client with response data in JSON format {"status": true,"message": "RECEIVED"}
I have the Rest Service created and using the Ref Id am getting the data from database but I am unable to send the response back to the rest client after putting the message in a channel(queue).
The output received by Rest Client is : No reply received within timeout
Basically I want the request thread to be returned immediately after the data (CLOB) is pushed to the channel(queue).
Below is the configuration.
<int:channel id="responseChannel"/>
<int:channel id="initCalculation">
<int:queue/>
</int:channel>
<!-- GET -->
<int-http:inbound-gateway
request-channel="httpGetChannel"
reply-channel="responseChannel"
supported-methods="GET"
path="/init/{refId}"
payload-expression="#pathVariables.refId">
<int-http:request-mapping produces="application/json"/>
</int-http:inbound-gateway>
<int:chain input-channel="httpGetChannel" output-channel="initCalculation">
<int-jdbc:stored-proc-outbound-gateway
id="outbound-gateway-storedproc-get-forma" data-source="dataSource"
is-function="false"
stored-procedure-name="XX_EMPROC.GET_FRMA"
ignore-column-meta-data="true"
expect-single-result="true">
<int-jdbc:sql-parameter-definition name="V_REF_ID" direction="IN" />
<int-jdbc:sql-parameter-definition name="V_FRMA" direction="OUT" type="#{T(oracle.jdbc.OracleTypes).CLOB}"/>
<int-jdbc:parameter name="V_REF_ID" expression="payload" />
</int-jdbc:stored-proc-outbound-gateway>
<!- Convert to JSON Format -->
<int:service-activator ref="brInitGateway" method="getResponse"/>
</int:chain>
<int:outbound-channel-adapter channel="initCalculation" ref="brInitGateway" method="process"/>
Kindly advise on the corrections needed in the above.
Thanks
Look, there is no body who sends message as a reply to the <int-http:inbound-gateway>. You have declared responseChannel, but who is going to use it as an output-channel?
I'd suggest you to do this:
<publish-subscribe-channel id="responseChannel"/>
<int:chain input-channel="httpGetChannel" output-channel="responseChannel">
<int:bridge input-channel="responseChannel" output-channel="initCalculation"/>
So, what happens here:
The publish-subscribe-channel for the reply-channel makes an internal bridge to the replyChannel header as one of the subscribers.
You send a result from the <chain> to that channel. Therefore the <int-http:inbound-gateway> gets its reply.
Having that <int:bridge> from response to the initCalculation you have a second subscriber and, therefore, send a message to the required queue.
If you are not interested in the brInitGateway.getResponse() as a reply for the HTTP request, you should consider do not have that reply-channel="responseChannel" at all, but still use some <publish-subscribe-channel> to send to the queue and some transformer to prepare a reply, e.g.:
<transformer input-channel="prepareProcess" expression="' {"status": true,"message": "RECEIVED"}'"/>
This transformer is without output-channel because it is going to send its result into the replyChannel header, therefore to the <int-http:inbound-gateway> initiator.

error-channel and reply-channel vanishes during header aggregation

I have the following workflow.
inbound-channel
splitter
task executor for split channels - all the threads execute the same workflow.
3.a. construct the request
3.b. service activator wrapper for a gateway message endpoint.
3.c. gateway wrapper over the http-outbound-gateway with the configuration of error-channel (to handle exceptions while invoking http-outbound-gateway)
3.d. http-outbound-gateway
aggregator
response out of spring integration workflow.
If an exception occurs in 3.d, the control goes to the service-activator configured for the gateway error channel.
I copy just the following from the failed message to the new header to the header passed to the error channel.
a. correlationId
b. sequenceNumber
c. sequenceSize
But while aggregating the splitter response, the DefaultAggregatingMessageGroupProcessor.java removes the conflicting headers and by that it removes the error-channel and reply-channel before providing the control to aggregator.
So once the aggregator completes it's operation it is unable to find the reply or error channel and it results in an exception.
I'm using spring-integration-core version 2.2.1 and I'm not sure why the reply-channel and error-channel is being removed during header aggregation.
Any input on resolving this issue will be of great help.
Thank You :)
EDIT 1:
Thank You very much Gary for helping me out with this scenario. I'm sharing my current configuration
<!-- SPLITTER -->
<int:splitter id="dentalSplitter" ref="dentalServiceSplitter"
method="getDentalServiceList" input-channel="dentalServiceSplitterChannel"
output-channel="dentalSplitterTaskChannel" />
<int:channel id="dentalSplitterTaskChannel">
<int:dispatcher task-executor="dentalTaskExecutor" />
</int:channel>
<int:chain input-channel="dentalSplitterTaskChannel" output-channel="dentalGatewayChannel">
<int:header-enricher>
<int:header name="CHAIN_START_TIME" expression="T(System).currentTimeMillis()" overwrite="true" />
<int:object-to-json-transformer content-type="application/json"/>
</int:chain>
<int:service-activator input-channel="dentalGatewayChannel" ref="dentalGatewayWrapper" output-channel="dentalReplyChannel" />
<int:gateway id="dentalGatewayWrapper" default-request-channel="dentalCostEstimateRequestChannel" error-channel="dentalErrorChannel"/>
<int-http:outbound-gateway id="dentalGateway"
url-expression="#urlBuilder.build('${service.endpoint}')"
http-method="POST" request-factory="clientHttpRequestFactory"
request-channel="dentalCostEstimateRequestChannel" extract-request-payload="true"
expected-response-type="com.dental.test.DentalResponse">
<int-http:request-handler-advice-chain>
<ref bean="logChainTimeInterceptor" />
</int-http:request-handler-advice-chain>
</int-http:outbound-gateway>
<!-- EXCEPTION -->
<int:chain input-channel="dentalErrorChannel" output-channel="dentalAggregatorChannel">
<int:transformer ref="commonErrorTransformer" method="dentalGracefulReturn"/>
</int:chain>
<!-- SUCCESS -->
<int:chain input-channel="dentalReplyChannel" output-channel="dentalAggregatorChannel">
<int:filter discard-channel="dentalErrorChannel"
expression="T(com.dental.util.InvocationOutcomeHelper).isOutcomeSuccess(payload?.metadata?.outcome?.code,payload?.metadata?.outcome?.message)" />
</int:chain>
<!-- AGGREGATION -->
<int:chain input-channel="dentalAggregatorChannel" output-channel="wsDentalServiceOutputChannel" >
<int:aggregator ref="dentalServiceAggregator" />
<int:service-activator ref="dentalResponseServiceActivator" />
</int:chain>
What I noticed was this, every split channel when passing through the gateway creates a new temporary channel for error and reply and after getting the response back from the gateway, it retains the preserved (original inbound) error and reply channel header. And as you had mentioned, after the control gets to the error transformer that flow of retaining the preserved headers gets broken and the aggregating message group processor receives three different instances of temporary channel and hence removes them.
I was planning to have a custom message group processor and modify the conflict resolution strategy for aggregating the header and came up with this config.
<bean id="channelPreservingAggregatingMessageHandler" class="org.springframework.integration.aggregator.AggregatingMessageHandler">
<constructor-arg name="processor" ref="channelPreservingMessageGroupProcessor"/>
</bean>
I'm yet to test this out though. But based on this discussion, this does not look like a viable solution.
And looks like my configuration for error handling in gateway is incorrect.
However, I'm confused on this statement of yours "Instead of forwarding the message directly, simply handle the error on your error flow and return the result normally to the gateway "wrapper"". If I remove the error channel how will I get the control back when an exception occurs? May be I'm missing to understand something here. Can you elaborate more on this please?
When asking questions about scenarios such as this, you generally need to show your configuration. However, I suspect you are forwarding the message from the error flow directly to the aggregator.
This is like doing a GOTO in code and breaks the scoping.
It won't work because the replyChannel header in the error message is for the gateway "wrapper", not the original upstream inbound gateway. When the aggregator gets conflicting headers, it has no choice but to drop the headers (you will see a DEBUG log message to that effect).
Instead of forwarding the message directly, simply handle the error on your error flow and return the result normally to the gateway "wrapper" (simply omit the error channel on the last element on the error flow).
The gateway will then fix up the reply so it is consistent with other messages (good and bad) and forward it to the aggregator.
You don't need to mess with headers in your error flow, just return the value you want to be aggregated along with the good results.
You should really update to a current release, or at least the latest in the 2.2.x line (2.2.6).

How do I get the request URL for message headers

I have an inbound gateway to process JSON payloads. Since we utilize the same backend service as our SOAP endpoints, I need to have an inbound gateway that maps to all paths. I can see the default HTTP headers come across the into the message, but I specifically need the request URL so I can parse it and route to the appropriate serve as needed.
My inbound-gateway:
<int-http:inbound-gateway id="JSONGateway"
path="*" request-channel="JSONRequestChannel" supported-methods="POST"
reply-timeout="5000"
header-mapper="headerMapper"
request-payload-type="java.lang.String" >
</int-http:inbound-gateway>
I tried adding a header-enricher like I do for SOAP, but the TransportContextListener is null.
<int:header-enricher input-channelJSONequestChannel"
output-channel="JSONRequestChannelWithHeaders">
<int:header name="service"
expression="T(org.springframework.ws.transport.context.TransportContextHolder).transportContext.connection.uri.toString().substring(T(org.springframework.ws.transport.context.TransportContextHolder).transportContext.connection.uri.toString().lastIndexOf('/')+1)" />
</int:header-enricher>
I need a way to get the URL of the request so I can parse out the service into the message headers for my downstream router.
/json/ContactService = "ContactService"
/json/ContactService/insert = "ContactService"
/json/ContactService/get/234 = "ContactService"
I also tried adding a header-mapper class, but still run into the same problem. How can I get a handle to the HTTPServletRequest in code? Once I get that I can get all the headers I need.
Well, you didn't check the message headers.
After converting ServletServerHttpRequest to the Message it gets these headers by default:
Message<?> message = messageBuilder
.setHeader(org.springframework.integration.http.HttpHeaders.REQUEST_URL,
request.getURI().toString())
.setHeader(org.springframework.integration.http.HttpHeaders.REQUEST_METHOD,
request.getMethod().toString())
.setHeader(org.springframework.integration.http.HttpHeaders.USER_PRINCIPAL,
servletRequest.getUserPrincipal())
.build();
So, http_requestUrl is for you to go ahead and parse it appropriately.

Resources