I am building a Kafka consumer with spring. My configuration seems to be pretty straightforward. Messages are being consumed and saved in files. However, the payload is cryptic and I can't get the data (short "hello world" messages).
This is what I get when I access the payload (e.g. when I set up a transformer bean btween in inbound kafka and the outboud file:
{test-topic={0=[[B#713c9d72, [B#7d656f90, [B#26bb8c83, [B#4b959d83 [B#5ed74e8e]}}
My question is: How do I access the actual payload (the "hellow world" string")?
My configuration is :
<int:channel id="inputFromKafka">
<int:queue />
</int:channel>
<int:poller
max-messages-per-poll="5" default = "true" fixed-delay="10" time-unit="MILLISECONDS"/>
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="true" channel="inputFromKafka">
<int-kafka:consumer-context id="consumerContext"
consumer-timeout="40000" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration
group-id="group12" max-messages="5">
<int-kafka:topic id="test-topic" streams="1" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int-kafka:zookeeper-connect id="zookeeperConnect"
zk-connect="localhost:2181" zk-connection-timeout="6000"
zk-session-timeout="6000" zk-sync-time="2000" />
<file:outbound-channel-adapter id="filesOut"
directory="/tmp/fromKafka">
</file:outbound-channel-adapter>
You are seeing raw byte[].
Add...
<bean id="decoder"
class="org.springframework.integration.kafka.serializer.common.StringDecoder" />
and
<int-kafka:consumer-configuration
value-decoder="decoder"
...
Related
I have a flow that uses a pub/sub channel based gateway with 2 subscribers to send message to 2 SQS queues. In between I have chain to do message transformation. An aggregator is used to summarize the report of every gateway invocation. Happy path works fine, but when my transformer throws error I get this message Reply message received but the receiving thread has already received a reply and the aggregator gets invoked but future.get in the Runner never returns. Sample Config and code to test is as follows:
Config
<!-- Gateway to Publish Data to SQS -->
<task:executor id="dataExecutor" pool-size="10"/>
<int:publish-subscribe-channel id="dataChannel" task-executor="dataExecutor" apply-sequence="true"/>
<int:publish-subscribe-channel id="sqsResultChannel"/>
<int:publish-subscribe-channel id="gatewayErrorChannel"/>
<int:publish-subscribe-channel id="gatewayReplyChannel"/>
<int:gateway service-interface="com.abc.DataPublishGateway" id="dataPublishGateway"
error-channel="gatewayErrorChannel">
<int:method name="publishToDataService"
payload-expression="#args[0]"
request-channel="dataChannel"
reply-channel="gatewayReplyChannel">
</int:method>
</int:gateway>
<int:chain input-channel="gatewayErrorChannel" output-channel="sqsResultChannel">
<int:header-enricher>
<int:correlation-id expression="payload.failedMessage.headers.correlationId" />
<int:header name="sequenceSize" expression="payload.failedMessage.headers.sequenceSize" />
<int:header name="sequenceNumber" expression="payload.failedMessage.headers.sequenceNumber" />
</int:header-enricher>
</int:chain>
<!-- Route to system-a SQS -->
<int:channel id="sqsSystemAPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemAPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-a" />
</int:header-enricher>
<int:transformer expression="payload/2"/> <!-- Simulate transformer error -->
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="a-queue"
channel="sqsSystemAPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<!-- Route to system-b SQS -->
<int:channel id="sqsSystemBPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemBPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-b" />
</int:header-enricher>
<int:transformer expression="payload.toLowerCase() "/>
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="b-queue"
channel="sqsSystemBPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<bean class="com.abc.DataResponseAggregator" id="responseAggregator" />
<int:aggregator input-channel="sqsResultChannel" output-channel="gatewayReplyChannel" ref="responseAggregator"/>
<!-- Generic Error Channel Logger -->
<int:logging-channel-adapter log-full-message="true"
logger-name="errorLogger"
level="ERROR"
channel="errorChannel"
id="globalErrorLoggingAdapter"/>
Aggregator
public class DataResponseAggregator {
public Map<String, String> aggregate(List<Message> responses) {
Map<String, String> resultMap = new HashMap<>();
responses.forEach(message -> {
if (message instanceof ErrorMessage) {
String exceptionMessage = ((ErrorMessage) message).getPayload().getCause().getMessage();
String targetSystem = ((MessagingException) message.getPayload()).getFailedMessage().getHeaders()
.get("targetSystem").toString();
resultMap.put(targetSystem, exceptionMessage);
}
else {
String targetSystem = message.getHeaders().get("targetSystem").toString();
resultMap.put(targetSystem, "Ack -> " + message.getHeaders().get("aws_messageId").toString());
}
});
return resultMap;
}
}
Gateway
public interface DataPublishGateway {
Future<Map<String, String>> publishToDataService(String message);
}
Runner
#Bean
CommandLineRunner runner(DataPublishGateway dataPublishGateway) {
return args -> {
String[] messages = new String[]{"Message 1", "Message 2"};
List<Future<Map<String, String>>> futureList = new ArrayList<>();
Arrays.stream(messages).forEach(s -> {
futureList.add(dataPublishGateway.publishToDataService(s));
});
System.out.println("Processing Futures and Printing Results...");
futureList.forEach(mapFuture -> {
try {
mapFuture.get().entrySet().forEach(entry -> {
System.out.println(entry.getKey() + " - " + entry.getValue());
});
} catch (Exception e) {
e.printStackTrace();
}
});
};
}
Logs
14:47:21.650 INFO 91449 --- [pool-1-thread-1] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one.
2020-09-07 14:47:21.651 INFO 91449 --- [pool-1-thread-2] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one.
2020-09-07 14:47:21.655 WARN 91449 --- [pool-1-thread-2] cMessagingTemplate$TemporaryReplyChannel : Reply message received but the receiving thread has already received a reply: GenericMessage [payload={system-a=Expression evaluation failed: payload/2; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1030E: The operator 'DIVIDE' is not supported between objects of type 'java.lang.String' and 'java.lang.Integer', system-b=Ack -> 90b62dff-f3c3-4288-b5e3-8178e410f60d}, headers={aws_messageId=90b62dff-f3c3-4288-b5e3-8178e410f60d, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1b519afe, sequenceNumber=2, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1b519afe, sequenceSize=2, correlationId=dc914588-f9a9-537f-0623-94938f84cec4, aws_serviceResult={MD5OfMessageBody: 83b2330607fe8f817ce6d24249dea373,MD5OfMessageAttributes: 5f1f442c363809afbd334ff00232c834,MessageId: 90b62dff-f3c3-4288-b5e3-8178e410f60d,}, id=9a6ad209-5291-6216-100b-502b4c37eb01, targetSystem=system-b, timestamp=1599482841654}]
2020-09-07 14:47:21.655 WARN 91449 --- [pool-1-thread-1] cMessagingTemplate$TemporaryReplyChannel : Reply message received but the receiving thread has already received a reply: GenericMessage [payload={system-a=Expression evaluation failed: payload/2; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1030E: The operator 'DIVIDE' is not supported between objects of type 'java.lang.String' and 'java.lang.Integer', system-b=Ack -> 3c8a4479-22e1-48d8-aca6-a86c252e90c1}, headers={aws_messageId=3c8a4479-22e1-48d8-aca6-a86c252e90c1, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#f5a99c3, sequenceNumber=2, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#f5a99c3, sequenceSize=2, correlationId=74ba49e0-fef8-02e6-bc4b-b2ccdd7cd13b, aws_serviceResult={MD5OfMessageBody: 1db65a6a0a818fd39655b95e33ada11d,MD5OfMessageAttributes: cd4eae624b7c115f948034aafaba01bc,MessageId: 3c8a4479-22e1-48d8-aca6-a86c252e90c1,}, id=3e3de3f3-e8b0-470e-e62f-31815e662857, targetSystem=system-b, timestamp=1599482841654}]
What am I missing? I would expect the error from the transformer to follow the flow : gatewayErrorChannel -> sqsResultChannel -> aggregator -> gatewayReplyChannel which it does since the aggregator output appears in the WARN message. But why does future.get never return and also it seems like gatewayReplyChannel receives aggregator output twice?
With some logging and most importantly the understanding of error and reply channels set on Gateway messages from this post, I am able to solve the issue.
What was the issue here?
The errorChannel was not set on the messages coming into the chain.
In case when both chains throw exceptions, the error messages coming to the aggregator did not have a reply channel set on them to return the aggregated message to the gateway's reply channel. The final config looks as below:
<!-- Gateway to Publish Data to SQS -->
<task:executor id="dataExecutor" pool-size="10"/>
<int:publish-subscribe-channel id="dataChannel" task-executor="dataExecutor" apply-sequence="true"/>
<int:publish-subscribe-channel id="sqsResultChannel"/>
<int:publish-subscribe-channel id="gatewayErrorChannel"/>
<int:publish-subscribe-channel id="gatewayReplyChannel"/>
<int:gateway service-interface="com.abc.DataPublishGateway" id="dataPublishGateway"
error-channel="gatewayErrorChannel">
<int:method name="publishToDataService"
payload-expression="#args[0]"
request-channel="dataChannel"
reply-channel="gatewayReplyChannel">
</int:method>
</int:gateway>
<int:chain input-channel="gatewayErrorChannel" output-channel="sqsResultChannel">
<int:header-enricher>
<int:correlation-id expression="payload.failedMessage.headers.correlationId" />
<int:header name="sequenceSize" expression="payload.failedMessage.headers.sequenceSize" />
<int:header name="sequenceNumber" expression="payload.failedMessage.headers.sequenceNumber" />
<int:reply-channel expression="payload.failedMessage.headers.replyChannel" />
</int:header-enricher>
</int:chain>
<!-- Route to system-a SQS -->
<int:channel id="sqsSystemAPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemAPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-a" />
<int:error-channel ref="gatewayErrorChannel" overwrite="true" />
</int:header-enricher>
<int:transformer expression="payload/2"/> <!-- Simulate transformer error -->
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="a-queue"
channel="sqsSystemAPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<!-- Route to system-b SQS -->
<int:channel id="sqsSystemBPublishChannel" />
<int:chain input-channel="dataChannel" output-channel="sqsSystemBPublishChannel">
<int:header-enricher>
<int:header name="targetSystem" value="system-b" />
<int:error-channel ref="gatewayErrorChannel" overwrite="true" />
</int:header-enricher>
<int:transformer expression="payload.toLowerCase() "/>
</int:chain>
<int-aws:sqs-outbound-channel-adapter sqs="amazonSQS"
queue="b-queue"
channel="sqsSystemBPublishChannel"
success-channel="sqsResultChannel"
failure-channel="gatewayErrorChannel"/>
<bean class="com.abc.DataResponseAggregator" id="responseAggregator" />
<int:aggregator input-channel="sqsResultChannel" output-channel="gatewayReplyChannel" ref="responseAggregator"/>
<!-- Generic Error Channel Logger -->
<int:logging-channel-adapter log-full-message="true"
logger-name="errorLogger"
level="ERROR"
channel="errorChannel"
id="globalErrorLoggingAdapter"/>
In current model, we have a REST endpoint, which gets requestbody, based on which a jms text message is created and sent to JMS queue,
TextMessage outMessage = session.createTextMessage(messagePayloadText);
..
outMessage.setStringProperty("clientType", clientType);
outMessage.setStringProperty("DYNAMIC", dynaHeader);
In above code DYNAMIC is required to help me in creating our url
<int:chain input-channel="gCStatusInChannel" output-channel="headerFilterChannel">
<int:header-enricher>
<int:header name="Api-Key" value="B8872853E8B"></int:header>
<int:header name="Accept" value="application/json" />
<int:header name="Content-Type" value="application/json" />
</int:header-enricher>
<int-http:outbound-gateway
url="https://i-zaie.sr13.tst.bst/ia-zaaie/rest/search/v2/cReference/{cref}"
http-method="PUT"
header-mapper="headerMapper"
expected-response-type="java.lang.String"
encode-uri="false"
request-factory="sslFactory">
<int-http:uri-variable name="cref" expression="headers['DYNAMIC']" />
</int-http:outbound-gateway>
<int:object-to-string-transformer></int:object-to-string-transformer>
</int:chain>
Everything works in this model. Now I want to use gateway instead of JMS
New code:
<int:gateway id="gService"
service-interface="n.d.lr.eai.gw.GGateway"
default-reply-channel="dest-channel"
default-request-timeout="5000" default-reply-timeout="5000">
<int:method name="vCreateSignal" request-channel="vCreateSignalInChannel"/> ...
Question:
can i have method in gateway as below?
public String vCreateSignal(String caseDat, String dynamic);
what should I do to enable
<int:chain input-channel="gCStatusInChannel"...
..>
to get headers['DYNAMIC'] value and continue.
Yes, you can do that. What you just need is to add a #Header("DYNAMIC") into that dynamic parameter:
public String vCreateSignal(String caseDat, #Header("DYNAMIC") String dynamic);
And when you call this gateway's method you just specify an argument and it will be mapped to an appropriate header and that all: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints-chapter.html#gateway-mapping
configuration of my application is like :
<int:inbound-channel-adapter channel="quakeinfotrigger.channel"
expression="''">
<int:poller fixed-delay="60000"></int:poller>
</int:inbound-channel-adapter>
<int:channel id="quakeinfo.channel">
<int:queue capacity="10" />
</int:channel>
<int:channel id="quakeinfotrigger.channel"></int:channel>
<int-http:outbound-gateway id="quakerHttpGateway"
request-channel="quakeinfotrigger.channel"
url="http://fff.com/rest/objects"
http-method="POST"
expected-response-type="java.lang.String"
charset="UTF-8"
reply-timeout="5000"
reply-channel="quakeinfo.channel">
</int-http:outbound-gateway>
I need to handle errors according to http_statusCode on the response header. Can you tell me to intercept it in my code.
Thanks
It depends on exactly what you want to do.
There are lots of ways to use the header, for example:
You can subscribe a <router/> to quakeinfo.channel and route based on the header value.
You can use a service activator...
<service-activator input-channel="quakeinfo.channel" ref="foo" method="handle" />
and
public class Foo {
public void handle(#Header("http_statusCode") HttpStatus statusCode) {
...
}
}
I have to analyze a spring integration piece of code given below :
<int:channel id="errorChannel" />
<int:exception-type-router input-channel="errorChannel"
default-output-channel="otherError">
<int:mapping
exception-type="MessageRejectedException"
channel="mreError" />
</int:exception-type-router>
<int:channel id="otherError" />
<int:transformer input-channel="otherError"
ref="otherExceptionTransformer" output-channel="errors" />
<bean id="otherExceptionTransformer"
class="OtherExceptionTransformer">
</bean>
<int:channel id="mreError" />
<int:transformer input-channel="mreError"
ref="mreExceptionTransformer" output-channel="errors" />
<bean id="mreExceptionTransformer"
class="MessageRejectedExceptionTransformer">
</bean>
<int:channel id="errors"/>
<int-jms:outbound-channel-adapter channel="errors"
connection-factory="connectionFactory" destination-
name="${myQueue.inbound.comp2}"/>
MessageRejectedExceptionTransformer takes as input as MessagingException object , however OtherExceptionTransformer takes ErrorMessage object as input.
Till now what I have understood is MessagingException contains the failedMessage and the cause of the exception,
whereas The ErrorMessage should contain the Message type.
Now , my main point of worry is , I am not understanding in which scenario , I won't be receiving MessagingException object .
I have tried to throw nullPointerException and IllegalArgumentException from my code , and spring processed both as Messaging Exception Only. So I am wondering when that otherError channel would receive a message.
Does anyone have a view on this?
The ErrorMessage has a payload of MessagingException which has two properties: failedMessage and cause (the original exception).
If a consumer (e.g. transformer) has an argument of MessagingException, the framework unwraps the payload from the message and invokes it. If the consumer takes a message (or ErrorMessage) it is invoked with the raw message.
Your router specifically routes on a MessageRejectedException, which is a subclass of MessagingException.
If any other MessagingException is thrown, the other route will be taken.
I would like to keep the original payload of the original requests and ise it in a xslt-transformer or in other operation. I lose it because I use an xslt-transformer and I need just some of the elements in the transformation. So my scenario is:
1.inbound-gateway (incoming WS req) -> 2.xslt-transformer (mapping for calling an external WS) -> 3.outbound-gateway (calling the external WS) -> 4.xslt-transformer (creating response from the resp. of the external WS and the original req)
At the 4th step I don't have the original req but I'd need it as I have to put values from it to the response. How could I implement it?
Thanks,
V.
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xmlns:int-ws="http://www.springframework.org/schema/integration/ws" xmlns:int-xml="http://www.springframework.org/schema/integration/xml" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="http://www.springframework.org/schema/integration/ws http://www.springframework.org/schema/integration/ws/spring-integration-ws.xsd http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/integration/xml http://www.springframework.org/schema/integration/xml/spring-integration-xml.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="authenticator" class="uk.co.virginmedia.test.Authenticator"/>
<bean id="webserviceDestinationProvider" class="uk.co.virginmedia.test.WebserviceDestinationProvider"/>
<bean id="resultToDocumentTransformer" class="org.springframework.integration.xml.transformer.ResultToDocumentTransformer"/>
<util:map id="orderNamespaceMap">
<entry key="res" value="http://schema/ReserveAppointment/2/0" />
</util:map>
<int-ws:inbound-gateway id="ws-gateway-for-rbta" request-channel="incoming-req-channel" reply-channel=""/>
<int:channel id="incoming-req-channel"/>
<int:service-activator id="authentication" input-channel="incoming-req-channel" ref="authenticator" method="authenticate" output-channel="authenticated-channel" />
<int:channel id="authenticated-channel"/>
<int-xml:xpath-router id="servicetype-router" input-channel="authenticated-channel" evaluate-as-string="true">
<int-xml:xpath-expression expression="//res:ReserveAppointmentRequest/res:serviceType/text()" ns-prefix="res" ns-uri="http://schema/ReserveAppointment/2/0"/>
<int-xml:mapping value="Broadband" channel="broadband-channel"/>
<int-xml:mapping value="FTTC+WholesaleLineRental" channel="fttc-wlr-channel"/>
</int-xml:xpath-router>
<int:channel id="broadband-channel"/>
<int-xml:xslt-transformer id="req_for_bt_xslt_transformer" input-channel="broadband-channel" output-channel="domresult_for_bt_channel" xsl-resource="classpath:/xsl/ToBTReq.xsl" result-type="StringResult"/>
<int:channel id="domresult_for_bt_channel"/>
<int:transformer input-channel="domresult_for_bt_channel" output-channel="document_for_bt_channel" expression="payload.toString()"/>
<int:channel id="document_for_bt_channel"/>
<int-ws:outbound-gateway request-channel="document_for_bt_channel" reply-channel="resp_from_bt_channel" destination-provider="webserviceDestinationProvider" id="call_bt-outbound_gateway" />
<int:channel id="resp_from_bt_channel"/>
<int-xml:xslt-transformer id="resp_for_rbta_xslt_transformer" input-channel="resp_from_bt_channel" output-channel="resp_for_rbta_channel" xsl-resource="classpath:/xsl/ToBTReq.xsl" result-type="StringResult"/>
Since your original message is just text you could copy it to a header field. This should work as long as you don't do anything special in between when you store and afterwards retrieve it.
So what I would try is:
<int:header-enricher input-channel="authenticated-channel" output-channel="pre-routing-channel">
<int:header name="original-payload" expression="payload.toString()" />
</int:header-enricher>
<!-- changed input channel of router -->
<int-xml:xpath-router id="servicetype-router" input-channel="pre-routing-channel" evaluate-as-string="true">
If this is not working for you (maybe because you have to do something more special in between or the payload is too big), you still have the option to use a ClaimCheck. Which is actually exactly what you are asking for. For this you'll need a MessageStore and then just store the message payload before modifying it. So instead of the header-enricher you will call
<int:claim-check-in input-channel="authenticated-channel" output-channel="pre-routing-channel" message-store="payloadstore" />
<!-- MessageStore definition storing payload using in memory map -->
<bean id="simpleMessageStore"
class="org.springframework.integration.store.SimpleMessageStore"/>