we are using spring-kafka, and for non-spring apps that communicate with us that don't set the spring_json_header_types header, we are specifying that certain headers should be mapped in as Strings by adding them as rawMappedHeaders. We do this for all of our binders by setting the config:
spring.cloud.stream.kafka.binder.headerMapperBeanName as the defaultKafkaHeaderMapper which we've set rawMappedHeaders for. This applies to both the producer and consumer bindings we have.
However, when we send data outbound to some of our spring applications, we actually don't want to leave out the spring_json_header_types values for those rawMappedHeader fields, since the downstream apps have not adjusted to this non-spring app communication yet.
Is there a way through the config or code that I can apply a headerMapperBeanName to only all of the producers or only to all of the consumers so that I can have them map headers differently for outbound vs inbound? Or is there a better way of doing this that the DefaultKafkaHeaderMapper itself can handle?
There is only one header mapper at the binder level.
It is probably easiest to create a custom KafkaHeaderMapper implementation that delegates to a different DefaultKafkaHeaderMapper for inbound and outbound mappings.
public interface KafkaHeaderMapper {
/**
* Map from the given {#link MessageHeaders} to the specified target headers.
* #param headers the abstracted MessageHeaders.
* #param target the native target headers.
*/
void fromHeaders(MessageHeaders headers, Headers target);
/**
* Map from the given native headers to a map of headers for the eventual
* {#link MessageHeaders}.
* #param source the native headers.
* #param target the target headers.
*/
void toHeaders(Headers source, Map<String, Object> target);
}
Related
I am upgrading one of our services from spring boot 1.5 to 2, and now I am seeing an exception related to kafka producer related to writing messageHistory to the header. Here is the exception recevied:
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is java.lang.IllegalArgumentException: Incorrect type specified for header 'history'. Expected [class org.springframework.integration.history.MessageHistory] but actual type is [class org.springframework.kafka.support.DefaultKafkaHeaderMapper$NonTrustedHeaderType].
By digging into the code, it looks like the exception was thrown in DefaultKafkaHeaderrMapper when converting the MessageHistory header https://github.com/spring-projects/spring-kafka/blob/14157742d7fa51ce8a22dfbdccc2e3c5b43c1c6f/spring-kafka/src/main/java/org/springframework/kafka/support/DefaultKafkaHeaderMapper.java#L253-L275 so anyone can help me understand:
is MessageHistory supposed to be written to the message before
sending out?
if the answer to 1) is yes, what's the right way to make MessageHistory to the trusted packages?
if 1) is no, what might have been done incorrectly to prevent the history to be written to the header?
thanks in advance!
You can add that org.springframework.integration.history package to white list to restore previous behavior though: DefaultKafkaHeaderMapper.addTrustedPackages():
/**
* Add packages to the trusted packages list (default {#code java.util, java.lang}) used
* when constructing objects from JSON.
* If any of the supplied packages is {#code "*"}, all packages are trusted.
* If a class for a non-trusted package is encountered, the header is returned to the
* application with value of type {#link NonTrustedHeaderType}.
* #param packagesToTrust the packages to trust.
*/
public void addTrustedPackages(String... packagesToTrust) {
https://docs.spring.io/spring-kafka/docs/current/reference/html/#headers
Or you can exclude such a header from mapping altogether.
Another way to add a custom serializer to store that MessageHistory as a List.
When the Spring Integration does the Http.outboundGateway call with the HttpMethod.PATCH operation there comes the exception:
Caused by: java.net.ProtocolException: Invalid HTTP method: PATCH
at java.base/java.net.HttpURLConnection.setRequestMethod(HttpURLConnection.java:487)
at java.base/sun.net.www.protocol.http.HttpURLConnection.setRequestMethod(HttpURLConnection.java:569)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.setRequestMethod(HttpsURLConnectionImpl.java:365)
at org.springframework.http.client.SimpleClientHttpRequestFactory.prepareConnection(SimpleClientHttpRequestFactory.java:226)
at org.springframework.http.client.SimpleClientHttpRequestFactory.createRequest(SimpleClientHttpRequestFactory.java:146)
at org.springframework.http.client.support.HttpAccessor.createRequest(HttpAccessor.java:87)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:719)
This is the problem in the Java class HttpURLConnection, because it doesn't support the PATCH operation.
I must use the PATCH method. What are the best ways to handle this situation?
The RestTemplate by default uses a SimpleClientHttpRequestFactory based on the Java HttpURLConnection. Consider to use some other ClientHttpRequestFactory, e.g. HttpComponentsClientHttpRequestFactory. When you declare such a bean you can inject it into the Http.outboundGateway().requestFactory():
/**
* Set the {#link ClientHttpRequestFactory} for the underlying {#link RestTemplate}.
* #param requestFactory The request factory.
* #return the spec
*/
public HttpMessageHandlerSpec requestFactory(ClientHttpRequestFactory requestFactory) {
I have been working on a "paved road" for setting up asynchronous messaging between two micro services using AMQP. We want to promote the use of separate domain objects for each service, which means that each service must define their own copy of any objects passed across the queue.
We are using Jackson2JsonMessageConverter on both the producer and the consumer side and we are using the Java DSL to wire the flows to/from the queues.
I am sure there is a way to do this, but it is escaping me: I want the consumer side to ignore the __TypeID__ header that is passed from the producer, as the consumer may have a different representation of that event (and it will likely be in in a different java package).
It appears there was work done such that if using the annotation #RabbitListener, an inferredArgumentTypeargument is derived and will override the header information. This is exactly what I would like to do, but I would like to use the Java DSL to do it. I have not yet found a clean way in which to do this and maybe I am just missing something obvious. It seems it would be fairly straight forward to derive the type when using the following DSL:
return IntegrationFlows
.from(
Amqp.inboundAdapter(factory, queueRemoteTaskStatus())
.concurrentConsumers(10)
.errorHandler(errorHandler)
.messageConverter(messageConverter)
)
.channel(channelRemoteTaskStatusIn())
.handle(listener, "handleRemoteTaskStatus")
.get();
However, this results in a ClassNotFound exception. The only way I have found to get around this, so far, is to set a custom message converter, which requires explicit definition of the type.
public class ForcedTypeJsonMessageConverter extends Jackson2JsonMessageConverter {
ForcedTypeJsonMessageConverter(final Class<?> forcedType) {
setClassMapper(new ClassMapper() {
#Override
public void fromClass(Class<?> clazz, MessageProperties properties) {
//this class is only used for inbound marshalling.
}
#Override
public Class<?> toClass(MessageProperties properties) {
return forcedType;
}
});
}
}
I would really like this to be derived, so the developer does not have to really deal with this.
Is there an easier way to do this?
The simplest way is to configure the Jackson converter's DefaultJackson2JavaTypeMapper with TypeIdMapping (setIdClassMapping()).
On the sending system, map foo:com.one.Foo and on the receiving system map foo:com.two.Foo.
Then, the __TypeId__ header gets foo and the receiving system will map it to its representation of a Foo.
EDIT
Another option would be to add an afterReceiveMessagePostProcessor to the inbound channel adapter's listener container - it could change the __TypeId__ header.
I've reviewed the documentation I can find and haven't worked out an answer so was hoping to get some enlightenment here. Is there any difference between these two calls?
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey("my-queue"))
.channel(Amqp.channel(connectionFactory).queueName("my-queue"))
First of all let's come back to the Reference Manual any way:
Prior to version 4.3, AMQP-backed channels only supported messages with Serializable payloads and headers. The entire message was converted (serialized) and sent to RabbitMQ. Now, you can set the extract-payload attribute (or setExtractPayload() when using Java configuration) to true. When this flag is true, the message payload is converted and the headers mapped, in a similar manner to when using channel adapters. This allows AMQP-backed channels to be used with non-serializable payloads (perhaps with another message converter such as the Jackson2JsonMessageConverter). The default mapped headers are discussed in Section 11.12, “AMQP Message Headers”. You can modify the mapping by providing custom mappers using the outbound-header-mapper and inbound-header-mapper attributes. You can now also specify a default-delivery-mode, used to set the delivery mode when there is no amqp_deliveryMode header. By default, Spring AMQP MessageProperties uses PERSISTENT delivery mode.
So, typically (by default) Amqp.channel(connectionFactory) sends the whole Message<?> to the target queue. Meanwhile the Amqp.outboundAdapter(amqpTemplate) does exactly opposite - map the payload to the body and headers:
if (this.amqpTemplate instanceof RabbitTemplate) {
MessageConverter converter = ((RabbitTemplate) this.amqpTemplate).getMessageConverter();
org.springframework.amqp.core.Message amqpMessage = MappingUtils.mapMessage(requestMessage, converter,
getHeaderMapper(), getDefaultDeliveryMode());
addDelayProperty(requestMessage, amqpMessage);
((RabbitTemplate) this.amqpTemplate).send(exchangeName, routingKey, amqpMessage, correlationData);
}
The AMQP-backed channels are aimed for the persistence and should not be used in the logic which relies on the messages content.
I need to invoke a REST API via Spring Integration's HTTP outbound gateway. How do I substitute the path variable with a value from payload. Payload has just one String value. The following code snippet is sending the place holder as such. Any help is appreciated.
#Bean
public MessageHandler httpGateway(#Value("http://localhost:8080/api/test-resource/v1/{parameter1}/codes") URI uri) {
HttpRequestExecutingMessageHandler httpHandler = new HttpRequestExecutingMessageHandler(uri);
httpHandler.setExpectedResponseType(Map.class);
httpHandler.setHttpMethod(HttpMethod.GET);
Map<String, Expression> uriVariableExp = new HashMap();
SpelExpressionParser parser = new SpelExpressionParser();
uriVariableExp.put("parameter1", parser.parseExpression("payload.Message"));
httpHandler.setUriVariableExpressions(uriVariableExp);
return httpHandler;
}
Let take a look what #Value is for first of all!
* Annotation at the field or method/constructor parameter level
* that indicates a default value expression for the affected argument.
*
* <p>Typically used for expression-driven dependency injection. Also supported
* for dynamic resolution of handler method parameters, e.g. in Spring MVC.
*
* <p>A common use case is to assign default field values using
* "#{systemProperties.myProp}" style expressions.
Looking to your sample there is nothing to resolve as a dependency injection value.
You can just use:
new HttpRequestExecutingMessageHandler("http://localhost:8080/api/test-resource/v1/{parameter1}/codes");
Although it doesn't matter in this case...
You code looks good, unless we don't know what your payload is.
That payload.Message expression looks odd. From big height it may mean like MyClass.getMessage(), but it can't invoke the getter because you use a property name as capitalized. If you really have there such a getter, so use it like payload.message. Otherwise, please, elaborate more. Some logs, StackTrace, the info about payload etc... It's fully unclear what is the problem.