How to get Azure Service Bus message id when sending a message to a topic using Spring Integration - spring-integration

After I send a message to a topic on Azure Service Bus using Spring Integration I would like to get the message id Azure generates. I can do this using JMS. Is there a way to do this using Spring Integration? The code I'm working with:
#Service
public class ServiceBusDemo {
private static final String OUTPUT_CHANNEL = "topic.output";
private static final String TOPIC_NAME = "my_topic";
#Autowired
TopicOutboundGateway messagingGateway;
public String send(String message) {
// How can I get the Azure message id after sending here?
this.messagingGateway.send(message);
return message;
}
#Bean
#ServiceActivator(inputChannel = OUTPUT_CHANNEL)
public MessageHandler topicMessageSender(ServiceBusTopicOperation topicOperation) {
DefaultMessageHandler handler = new DefaultMessageHandler(TOPIC_NAME, topicOperation);
handler.setSendCallback(new ListenableFutureCallback<>() {
#Override
public void onSuccess(Void result) {
System.out.println("Message was sent successfully to service bus.");
}
#Override
public void onFailure(Throwable ex) {
System.out.println("There was an error sending the message to service bus.");
}
});
return handler;
}
#MessagingGateway(defaultRequestChannel = OUTPUT_CHANNEL)
public interface TopicOutboundGateway {
void send(String text);
}
}

You could use ChannelInterceptor to get message headers:
public class CustomChannelInterceptor implements ChannelInterceptor {
#Override
public Message<?> preSend(Message<?> message, MessageChannel channel) {
//key of the message-id header is not stable, you should add logic here to check which header key should be used here.
//ref: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/spring/azure-spring-cloud-starter-servicebus#support-for-service-bus-message-headers-and-properties
String messageId = message.getHeaders().get("message-id-header-key").toString();
return ChannelInterceptor.super.preSend(message, channel);
}
}
Then in the configuration, set this interceptor to your channel
#Bean(name = OUTPUT_CHANNEL)
public BroadcastCapableChannel pubSubChannel() {
PublishSubscribeChannel channel = new PublishSubscribeChannel();
channel.setInterceptors(Arrays.asList(new CustomChannelInterceptor()));
return channel;
}

Related

Spring integration migration to Redis from RabbitMQ to share application events

We are migrating to Redis from RabbitMQ in our microservice applications.
Here is our service activator
#ServiceActivator(inputChannel = ApplicationEventChannelNames.REMOTE_CHANNEL)
public void handleApplicationEvent(#Header(value = ApplicationEventHeaders.APPLICATION_EVENT) final ApplicationEvent event,
#Payload Object message) {
...
}
Initially we had a problem where we were losing application event in the SimpleMessageConverter. We solved it by implementing a CustomRedisMessageConverter and putting application event into the payload in the fromMessage method and retreiving it from payload and create a new message headers with application event in the toMessage method.
#Override
public Object fromMessage(Message<?> message, Class<?> targetClass) {
if (message.getHeaders().get(ApplicationEventHeaders.APPLICATION_EVENT) != null) {
Map<String, Object> map = new HashMap<>();
map.put("headers", ((ApplicationEvent) message.getHeaders().get(ApplicationEventHeaders.APPLICATION_EVENT)).getName());
map.put("payload", message.getPayload());
GenericMessage<Map<String, Object>> msg = new GenericMessage<>(map, message.getHeaders());
return super.fromMessage(msg, targetClass);
}
return super.fromMessage(message, targetClass);
}
#Override
public Message<?> toMessage(Object payload, MessageHeaders headers) {
try {
final Map<String, ?> message = new ObjectMapper().readValue((String) payload, new TypeReference<Map<String, ?>>() {});
if (message.get("headers") != null) {
final Map<String, Object> messageHeaders = new HashMap<>(headers);
messageHeaders.put(ApplicationEventHeaders.APPLICATION_EVENT, new ApplicationEvent((String) message.get("headers")));
return super.toMessage(message.get("payload"), new MessageHeaders(messageHeaders));
}
} catch (JsonProcessingException exception) {
/* Intentionally left blank */
}
return super.toMessage(payload, headers);
}
We are wondering if there is a better approach for doing this?
Lastly, payload in the service activator come as a LinkedHashMap but we want it to be an object. With RabbitMQ this was handled.
Is there any way to do the same in Redis? Or do we use headers to keep track of the type of a payload and manually convert them into an object?
UPDATE - REDIS Configuration
#Bean
public RedisInboundChannelAdapter applicationEventInboundChannelAdapter(#Value(value = "${com.xxx.xxx.xxx.integration.spring.topic}") String topic,
MessageChannel applicationEventRemoteChannel,
RedisConnectionFactory connectionFactory) {
final RedisInboundChannelAdapter inboundChannelAdapter = new RedisInboundChannelAdapter(connectionFactory);
inboundChannelAdapter.setTopics(topic);
inboundChannelAdapter.setOutputChannel(applicationEventRemoteChannel);
inboundChannelAdapter.setErrorChannel(errorChannel());
inboundChannelAdapter.setMessageConverter(new CustomRedisMessageConverter());
return inboundChannelAdapter;
}
#ServiceActivator(inputChannel = "errorChannel")
public void processError(MessageHandlingException exception) {
try {
logger.error(
"Could not process {}, got exception: {}",
exception.getFailedMessage().getPayload(),
exception.getMessage());
logger.error(
ExceptionUtils.readStackTrace(exception));
} catch (Throwable throwable) {
logger.error(
"Got {} during processing with message: {} ",
MessageHandlingException.class.getSimpleName(),
exception);
}
}
#Bean
#ServiceActivator(inputChannel = ApplicationEventChannelNames.LOCAL_CHANNEL)
public RedisPublishingMessageHandler redisPublishingMessageHandler(#Value(value = "${com.xxx.xxx.xxx.integration.spring.topic}") String topic,
RedisConnectionFactory redisConnectionFactory) {
final RedisPublishingMessageHandler redisPublishingMessageHandler = new RedisPublishingMessageHandler(redisConnectionFactory);
redisPublishingMessageHandler.setTopic(topic);
redisPublishingMessageHandler.setSerializer(new Jackson2JsonRedisSerializer<>(String.class));
redisPublishingMessageHandler.setMessageConverter(new CusomRedisMessageConverter());
return redisPublishingMessageHandler;
}
/*
* MessageChannel
*/
#Bean
public MessageChannel errorChannel() {
return new DirectChannel();
}
Redis does not support headers, so you have to embed them into a body. See EmbeddedJsonHeadersMessageMapper which could be supplied into that org.springframework.integration.support.converter.SimpleMessageConverter on both side.

Spring Integration Wiretap with Annotations (4.3 Spring Integration)

I have a Spring Integration (4.3) application and i have a JMS receiver which sends the received message to a ServiceActivator.
I am trying to Wiretap a channel, and send the JMS message to Logger (which in this case is Service Activator as well)
I get a strange behavior with the code below.
Every odd message sent (first, third) - it only goes to the logger.
Every even message sent (second, fourth, etc) - only goes to the ServiceActivator that is supposed to do some work and in the future sent the message downstream.
Here are logs
2018-07-23 16:14:43.278 INFO 16532 --- [ container-1] zzz : MSG1
Received via barChannel: MSG2
How can i change my code so all messages go both to the logger and the ServiceActivator that is supposed to do the work?
Thanks a lot in advance
#Bean
public ConnectionFactory jmsConnectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:61616");
}
#Bean
public JmsMessageDrivenEndpoint inbound() {
JmsMessageDrivenEndpoint jmsMessageDrivenEndpoint = new JmsMessageDrivenEndpoint(container(), listener());
jmsMessageDrivenEndpoint.setSessionAcknowledgeMode("transacted");
jmsMessageDrivenEndpoint.setAutoStartup(true);
return jmsMessageDrivenEndpoint;
}
#Bean
public DefaultMessageListenerContainer container() {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
container.setConnectionFactory(jmsConnectionFactory());
container.setDestinationName("foo.bar");
return container;
}
#Bean
public ChannelPublishingJmsMessageListener listener() {
ChannelPublishingJmsMessageListener listener = new ChannelPublishingJmsMessageListener();
listener.setRequestChannelName("myLogChannel");
return listener;
}
#ServiceActivator(inputChannel = "myLogChannel")
public void bar(String in) {
System.out.println("Received via barChannel: " + in);
}
#Bean
public MessageChannel TappingChannel(MessageChannel myLogChannel) {
DirectChannel d = new DirectChannel();
d.addInterceptor(new WireTap("myLogChannel"));
return d;
}
#Bean
#ServiceActivator(inputChannel = "myLogChannel")
public MessageHandler logger() {
LoggingHandler loggingHandler = new LoggingHandler(LoggingHandler.Level.INFO.name());
loggingHandler.setLoggerName("zzz");
return loggingHandler;
}
#Bean
public MessageChannel myLogChannel() {
return new DirectChannel();
}
EDIT: Here is the original code that works
#Bean
public MessageChannel toRouter() {
return new DirectChannel();
}
#Bean
public ConnectionFactory jmsConnectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:61616");
}
#Bean
public JmsMessageDrivenEndpoint inbound(ConnectionFactory jmsConnectionFactory) {
return new JmsMessageDrivenEndpoint(container(jmsConnectionFactory), listener());
}
#Bean
public DefaultMessageListenerContainer container(ConnectionFactory jmsConnectionFactory) {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
container.setConnectionFactory(jmsConnectionFactory);
container.setDestinationName("foo.bar");
return container;
}
#Bean
public ChannelPublishingJmsMessageListener listener() {
ChannelPublishingJmsMessageListener listener = new ChannelPublishingJmsMessageListener();
listener.setRequestChannelName("toRouter");
return listener;
}
#ServiceActivator(inputChannel = "toRouter")
public void bar(String in) {
System.out.println("Received via barChannel: " + in);
}
I want to do the Wiretap logging on this code
You have a mess in your config.
First of all you have two #ServiceActivator for the same myLogChannel DirectChannel.
By default it uses a round-robin dispatching strategy. That's why you see that odd/even behavior between your two consumers on the same channel.
You separate TappingChannel doesn't bring any value and it is out of use at all. No one sends message to this channel: you send to the listener.setRequestChannelName("myLogChannel"); directly though... Therefore a WireTap isn't performed at all as well.
I'm not sure how to help you since it isn't clear what you are going to use as a main channel and what should be used from the wire-tap.
UPDATE
So, you need to configure that new WireTap("myLogChannel") on this toRouter channel.

ServiceActivator does not receive message from ImapIdleChannelAdapter

ServiceActivator does not receive messages from ImapIdleChannelAdapter...
JavaMail logs successful FETCH, but MIME messages do not get delivered to SA endpoint... I want to understand what is wrong in my code.
A7 FETCH 1:35 (ENVELOPE INTERNALDATE RFC822.SIZE FLAGS BODYSTRUCTURE)
* 1 FETCH (ENVELOPE ("Fri....
Code snippet below:
`
#Autowired
EmailConfig emailCfg;
#Bean
public SubscribableChannel mailChannel() {
return MessageChannels.direct().get();
}
#Bean
public ImapIdleChannelAdapter getMailAdapter() {
ImapMailReceiver mailReceiver = new ImapMailReceiver(emailCfg.getImapUrl());
mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
ImapIdleChannelAdapter imapIdleChannelAdapter = new ImapIdleChannelAdapter(mailReceiver);
imapIdleChannelAdapter.setOutputChannel(mailChannel());
imapIdleChannelAdapter.setAutoStartup(true);
imapIdleChannelAdapter.afterPropertiesSet();
return imapIdleChannelAdapter;
}
#ServiceActivator(inputChannel = "mailChannel")
public void receive(String mail) {
log.warn(mail);
}
private Properties javaMailProperties() {
Properties javaMailProperties = new Properties();
javaMailProperties.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
javaMailProperties.setProperty("mail.imap.socketFactory.fallback", "false");
javaMailProperties.setProperty("mail.store.protocol", "imaps");
javaMailProperties.setProperty("mail.debug", "true");
javaMailProperties.setProperty("mail.imap.ssl", "true");
return javaMailProperties;
}
`
The problem was due to wrong bean initialization. Full version that works OK:
#Slf4j
#Configuration
#EnableIntegration
public class MyMailAdapter {
#Autowired
EmailConfig emailCfg;
#Bean
public SubscribableChannel mailChannel() {
log.info("Channel ready");
return MessageChannels.direct().get();
}
#Bean
public ImapMailReceiver receiver() {
ImapMailReceiver mailReceiver = new ImapMailReceiver(emailCfg.getImapUrl());
mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
return mailReceiver;
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter imapIdleChannelAdapter = new ImapIdleChannelAdapter(receiver());
imapIdleChannelAdapter.setOutputChannel(mailChannel());
imapIdleChannelAdapter.afterPropertiesSet();
return imapIdleChannelAdapter;
}
#ServiceActivator(inputChannel = "mailChannel")
public void receive(Message<MimeMessage> mail) throws MessagingException {
log.info(mail.getPayload().toString());
}
private Properties javaMailProperties() {
Properties javaMailProperties = new Properties();
javaMailProperties.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
javaMailProperties.setProperty("mail.imap.socketFactory.fallback", "false");
javaMailProperties.setProperty("mail.store.protocol", "imaps");
javaMailProperties.setProperty("mail.debug", "true");
javaMailProperties.setProperty("mail.imap.ssl", "true");
return javaMailProperties;
}
}
I don't know what's exactly wrong with your code but I will suggest you few approches that could help you.
Firstly I suggest you to use java DSL in java based configuration. It will provide you nice way to directly specific flow of your integration application (and avoid simply mistakes). For example for spliiter and service activator:
#Bean
public IntegrationFlow yourFlow(AbstractMessageSplitter splitter,
MessageHandler handler) {
return
IntegrationFlows
.from(CHANNEL)
.split(splitter)
.handle(handler).get();
}
Secondly it's generally bad idea to directly specify message type to String. Try something like this (why String?):
#ServiceActivator(inputChannel = "mailChannel")
public void receive(Message<?> message) {
/* (String) message.getPayload() */
}
Maybe it's not a case but let's check it.

Spring Integration Kafka Manual Acknowledgment

I'm having issues using manual acknowledgements with the KafkaTopicOffsetManager. When acknowledge() is called, the topic begins to get spammed repeatedly. Kafka has log.cleaner.enable set to true and the topic is using cleanup.policy=compact. Thanks for any help.
Config:
#Bean
public ZookeeperConfiguration zookeeperConfiguration() {
ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration(kafkaConfig.getZookeeperAddress());
zookeeperConfiguration.setClientId("clientId");
return zookeeperConfiguration;
}
#Bean
public ConnectionFactory connectionFactory() {
return new DefaultConnectionFactory(zookeeperConfiguration());
}
#Bean
public TestMessageHandler messageListener() {
return new TestMessageHandler();
}
#Bean
public OffsetManager offsetManager() {
ZookeeperConnect zookeeperConnect = new ZookeeperConnect(kafkaConfig.getZookeeperAddress());
OffsetManager offsetManager = new KafkaTopicOffsetManager(zookeeperConnect, kafkaConfig.getTopic() + "_OFFSET");
return offsetManager;
}
#Bean
public KafkaMessageListenerContainer kafkaMessageListenerContainer() {
KafkaMessageListenerContainer kafkaMessageListenerContainer = new KafkaMessageListenerContainer(connectionFactory(), kafkaConfig.getTopic());
kafkaMessageListenerContainer.setMessageListener(messageListener());
kafkaMessageListenerContainer.setOffsetManager(offsetManager());
return kafkaMessageListenerContainer;
}
Listener:
public class TestMessageHandler implements AcknowledgingMessageListener {
private static final Logger logger = LoggerFactory.getLogger(TestMessageHandler.class);
#Override
public void onMessage(KafkaMessage message, Acknowledgment acknowledgment) {
logger.info(message.toString());
acknowledgment.acknowledge();
}
}
The KafkaTopicOffsetManager needs its own topic to maintain the offset of the actual topic being consumed.
If you don't want to deal with decoding the message payload yourself (its painful in my opinion), extend listener from abstract class AbstractDecodingAcknowledgingMessageListener and provide org.springframework.integration.kafka.serializer.common.StringDecoder as the decoder.
public class TestMessageHandlerDecoding extends AbstractDecodingAcknowledgingMessageListener {
public TestMessageHandlerDecoding(Decoder keyDecoder, Decoder payloadDecoder) {
super(keyDecoder, payloadDecoder);
}
#Override
public void doOnMessage(Object key, Object payload, KafkaMessageMetadata metadata, Acknowledgment acknowledgment) {
LOGGER.info("payload={}",payload);
}

Logging in Azure web jobs

I am working with Azure web jobs. Also I am aware that the TextWriter is used to write logs in case of web jobs (VS 2013). However, The logs are created under the Output logs folder under the blob container. THese are not user friendly. I have to open each file to read the message written to it.
Is there any way to change the logging to table, which is user friendly to read?
Thanks in advance.
I'm not sure if there's a "native" way to do this, but you can add Azure Storage Client through nuget and write your own "Log To Azure Tables".
You can use the Semantic Logging Application Block for Windows Azure.
It allows you to log into an Azure Table Storage.
Define your Eventsource:
// A simple interface to log what you need ...
public interface ILog
{
void Debug(string message);
void Info(string message);
void Warn(string message);
void Error(string message);
void Error(string message, Exception exception);
}
Implement the interface :
And the implementation ( implementation of your interface must be decorated with the NonEventAttribute see this post) :
[EventSource(Name = "MyLogEventsource")]
public class Log : EventSource, ILog
{
public Log()
{
EventSourceAnalyzer.InspectAll(this);
}
[NonEvent]
public void Debug(string message)
{
DebugInternal(message);
}
[Event(1)]
private void DebugInternal(string message)
{
WriteEvent(1, message);
}
[NonEvent]
public void Info(string message)
{
InfoInternal(message);
}
[Event(2)]
private void InfoInternal(string message)
{
WriteEvent(2, message);
}
[NonEvent]
public void Warn(string message)
{
WarnInternal(message);
}
[Event(3)]
private void WarnInternal(string message)
{
WriteEvent(3, message);
}
[NonEvent]
public void Error(string message)
{
ErrorInternal(message, "", "");
}
[NonEvent]
public void Error(string message, Exception exception)
{
ErrorInternal(message, exception.Message, exception.ToString());
}
[Event(4)]
private void ErrorInternal(string message, string exceptionMessage, string exceptionDetails)
{
WriteEvent(4, message, exceptionMessage, exceptionDetails);
}
}
Now you can register your event source like that :
var log = new Log();
var eventListeners = new List<ObservableEventListener>();
// Log to Azure Table
var azureListener = new ObservableEventListener();
azureListener.EnableEvents(log , EventLevel.LogAlways, Keywords.All);
azureListener.LogToWindowsAzureTable(
instanceName: Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID") ?? "DevelopmentInstance",
connectionString: CloudConfigurationManager.GetSetting("MyStorageConnectionString")
tableAddress: "MyLogTable");
eventListeners .Add(azureListener);

Resources