I am trying to create a integration flow for JMS MessageDriverChannelAdapter through which I need to send message to the Kafka server. But I really
stuck when I am trying to convert the the xml tag to dsl specific code, not able to convert the xml to required DSL. Can any one please provide
any pointer to it as I am not able to proceed over here.
I have created a MessageListenerContainer like this........
String brokerUrl = "tcp://101.11.102.125:31316";
String topic = "sometpoic";
String kafkaBrokerUrl = "101.11.102.125:1012";
String kafkaTopic = "kafka_Topic";
#Bean
public DefaultMessageListenerContainer listenerContainer() {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
ActiveMQConnectionFactory conFactory = new ActiveMQConnectionFactory();
ActiveMQTopic mqTopic = new ActiveMQTopic(topic);
conFactory.setBrokerURL(brokerUrl);
container.setConnectionFactory(conFactory);
container.setDestination(mqTopic);
container.setSessionTransacted(true);
return container;
}
These are my input and output channels........
#Bean
public MessageChannel jmsInChannel() {
return MessageChannels.publishSubscribe().get();
}
#Bean
public MessageChannel jmsOutChannel() {
return MessageChannels.publishSubscribe().get();
}
And this is my JMS adapter flow............
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDriverChannelAdapter(listenerContainer())
.autoStartup(true))
.channel(jmsInChannel())
.get();
}
Now I need to create a header-enricher like this but not able to covert this into DSL.
<int:header-enricher input-channel="jmsInChannel" output-channel="jmsOutChannel">
<int:header name="kafkaBrokerUrl" value="${kafka.url}"></int:header>
<int:header name="kafkaTopic" value="${kafka.topic}"></int:header>
and I need to create a service-activator and call a kafka producer method form a different class like this in xml....
<int:service-activator input-channel="jmsOutChannel" ref="KafkaProducer" method="produceToJmsKafka"/>
<bean id="KafkaProducer" class="com.david.jms.JmsKafkaProducer"/>
So how to convert these above xml code to similar DSL specific code.
After getting the compilation error I have tried like this...
#SuppressWarnings("unchecked")
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDriverChannelAdapter(listenerContainer())
.autoStartup(true))
.channel(jmsInChannel())
.enrichHeaders(new MapBuilder()
.put("brokerid", brokerid)
.put("topic", topic)
.put("source", source)
.put("fileType", fileType))
.handle("KafkaProducer", "produceToJmsKafka")
.get();
}
#Bean
public JmsProducer KafkaProducer() {
return new JmsProducer();
}
That may be like this:
#Value("${kafka.url}")
private String kafkaBrokerUrl;
#Value("${kafka.topic}")
private String kafkaTopic;
....
#Bean
public IntegrationFlow jmsMessageDrivenFlow() {
return IntegrationFlows
.from(Jms.messageDriverChannelAdapter(listenerContainer())
.autoStartup(true))
.channel(jmsInChannel())
.enrichHeaders(new StringStringMapBuilder()
.put("kafkaBrokerUrl", kafkaBrokerUrl)
.put("kafkaTopic", kafkaTopic))
.handle("KafkaProducer", "produceToJmsKafka")
.get();
}
From here I don't see reason to have those MessageChannel beans, especially like publishSubscribe().
From other side since DSL 1.1 we provide the implementation for Spring Integration Kafka Adapters.
Related
using Boot 2.2.2 and Integration 5.2.2 - when an XML message is sourced from a File and fails unmarshalling (i.e. it is not XML) the message proceeds as expect to errorChannel. However, when the message comes from JMS, through the same route of channels and fails unmarshalling, it is not routed to errorChannel and the message is rolled-back to JMS. After which I am stuck in an endless loop of SAXParseException for the same message.
I had following this example from Proper ultimate way to migrate JMS event listening to Spring Integration with Spring Boot
. Is there some implied transaction control that I am not considering? How do I have Spring Integration forward the message to errorChannel and commit the 'get' from the incoming queue?
Synopsis of code is as follows;
#Bean
public IntegrationFlow fileReader() {
return IntegrationFlows
.from(
Files
.inboundAdapter( ... )
...
.get(), e -> e.poller(Pollers.fixedDelay(1000))
)
.transform(new FileToStringTransformer())
.channel("backUpChannel")
.get();
}
#Bean
public IntegrationFlow getMessageFromJms(ConnectionFactory connectionFactory, #Value("${queues.myQueue}") String myQueue) {
return IntegrationFlows.from(
Jms
.messageDrivenChannelAdapter(connectionFactory)
.destination(myQueue)
)
.channel("backUpChannel")
.get();
}
#Bean
public IntegrationFlow doBackUp() {
return IntegrationFlows
.from("backUpChannel")
.<String>handle((payload, headers) -> {
String uuid = headers.get(MessageHeaders.ID).toString();
File backUpFile = new File("c:/backup/" + uuid + ".txt");
byte[] payloadContent = payload.getBytes();
try {
java.nio.file.Files.write(backUpFile.toPath(), payloadContent);
} catch (IOException e) {
e.printStackTrace();
}
return payload;
})
.channel("XXX")
.get();
}
#Bean
public Jaxb2Marshaller unmarshaller() {
Jaxb2Marshaller unmarshaller = new Jaxb2Marshaller();
unmarshaller.setClassesToBeBound(MyClass.class);
return unmarshaller;
}
#Bean
public IntegrationFlow handleParseXml() {
return IntegrationFlows
.from("XXX")
.transform(new UnmarshallingTransformer(unmarshaller()))
.channel("YYY")
.get();
}
You need to add .errorChannel(...) to the message-driven channel adapter.
i am creating a webapp in spring Maven with PAHO mqtt. and there are two ways to implement mqtt, one is Spring Mqtt Integration and second is a general way to create a connection object and connect/disconnect but what is the difference between them how much convenient and Reliable they are.
right now i am implementing subscribe with Spring Mqtt Integration but can't understand it can anyone please suggest some good easy tutorial.
Thank you!
See the Spring Integration MQTT Sample Spring Boot App.
Reads from stdIn sends the data over MQTT, receives the data and logs it.
// publisher
#Bean
public IntegrationFlow mqttOutFlow() {
return IntegrationFlows.from(CharacterStreamReadingMessageSource.stdin(),
e -> e.poller(Pollers.fixedDelay(1000)))
.transform(p -> p + " sent to MQTT")
.handle(mqttOutbound())
.get();
}
#Bean
public MessageHandler mqttOutbound() {
MqttPahoMessageHandler messageHandler = new MqttPahoMessageHandler("siSamplePublisher", mqttClientFactory());
messageHandler.setAsync(true);
messageHandler.setDefaultTopic("siSampleTopic");
return messageHandler;
}
// consumer
#Bean
public IntegrationFlow mqttInFlow() {
return IntegrationFlows.from(mqttInbound())
.transform(p -> p + ", received from MQTT")
.handle(logger())
.get();
}
private LoggingHandler logger() {
LoggingHandler loggingHandler = new LoggingHandler("INFO");
loggingHandler.setLoggerName("siSample");
return loggingHandler;
}
#Bean
public MessageProducerSupport mqttInbound() {
MqttPahoMessageDrivenChannelAdapter adapter = new MqttPahoMessageDrivenChannelAdapter("siSampleConsumer",
mqttClientFactory(), "siSampleTopic");
adapter.setCompletionTimeout(5000);
adapter.setConverter(new DefaultPahoMessageConverter());
adapter.setQos(1);
return adapter;
}
I'm creating a series of processes that use Spring Integration explicitly using the Java DSL. Each of these processes do different things but they have some of the same processing logic
Example:
get
process
deduplicate
emit
I would like to essentially create a chain of post processing integration flows that can be enabled/disabled via configuration/profiles.
Example:
get
preprocess flow 1 (if enabled)
...
preprocess flow n (if enabled)
process
postprocess flow 1 (if enabled)
...
postprocess flow n (if enabled)
emit
I'm pretty sure this doesn't exist yet in SI but thought I'd ask. The only thing i could think of would be to create a bean that created direct message channels on the fly and that, during configuration, i could give to each of the integration flows to use to get their "from" and "channel" message channels.
Example:
#Configuration
public class BaseIntegrationConfiguration {
#Bean
public MessageChannel preProcessMessageChannel() {
return MessageChannels.direct().get();
}
#Bean
public MessageChannel processMessageChannel() {
return MessageChannels.direct().get();
}
#Bean
public MessageChannel postProcessMessageChannel() {
return MessageChannels.direct().get();
}
#Bean
public MessageChannel emitMessageChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow getDataFlow(MessageChannel preProcessMessageChannel) {
return IntegrationFlows
.from(/* some inbound channel adapter */)
// do other flow stuff
.channel(preProcessMessageChannel)
.get();
}
#Bean
public IntegrationFlowChainMessageChannelGenerator preProcessFlowGenerator(
MessageChannel preProcessMessageChannel,
MessageChannel processMessageChannel) {
IntegrationFlowChainMessageChannelGenerator generator = new IntegrationFlowChainMessageChannelGenerator ();
generator.startWith(preProcessMessageChannel);
generator.endWith(processMessageChannel);
return generator;
}
#Bean
public IntegrationFlow processFlow(
MessageChannel processMessageChannel,
MessageChannel postProcessMessageChannel) {
return IntegrationFlows
.from(processMessageChannel)
// do other flow stuff
.channel(postProcessMessageChannel)
.get();
}
#Bean
public IntegrationFlowChainMessageChannelGenerator postProcessFlowGenerator(
MessageChannel postProcessMessageChannel,
MessageChannel emitMessageChannel) {
IntegrationFlowChainMessageChannelGenerator generator = new IntegrationFlowChainMessageChannelGenerator ();
generator.startWith(postProcessMessageChannel);
generator.endWith(emitMessageChannel);
return generator;
}
}
#Configuration
#Order(1)
#Profile("PreProcessFlowOne")
public class PreProcessOneIntegrationConfiguration {
#Bean
public IntegrationFlow preProcessFlowOne(IntegrationFlowChainMessageChannelGenerator preProcessFlowGenerator) {
return IntegrationFlows
.from(preProcessFlowGenerator.getSourceChannel())
// flow specific behavior here
.channel(preProcessFlowGenerator.getDestinationChannel())
.get();
}
}
#Configuration
#Order(2)
#Profile("PreProcessFlowTwo")
public class PreProcessTwoIntegrationConfiguration {
#Bean
public IntegrationFlow preProcessFlowTwo(IntegrationFlowChainMessageChannelGenerator preProcessFlowGenerator) {
return IntegrationFlows
.from(preProcessFlowGenerator.getSourceChannel())
// flow specific behavior here
.channel(preProcessFlowGenerator.getDestinationChannel())
.get();
}
}
#Configuration
#Order(1)
#Profile("PostProcessFlowOne")
public class PostProcessOneIntegrationConfiguration {
#Bean
public IntegrationFlow postProcessFlowOne(IntegrationFlowChainMessageChannelGenerator postProcessFlowGenerator) {
return IntegrationFlows
.from(postProcessFlowGenerator.getSourceChannel())
// flow specific behavior here
.channel(postProcessFlowGenerator.getDestinationChannel())
.get();
}
}
#Configuration
#Order(2)
#Profile("PostProcessFlowTwo")
public class PostProcessTwoIntegrationConfiguration {
#Bean
public IntegrationFlow postProcessFlowTwo(IntegrationFlowChainMessageChannelGenerator postProcessFlowGenerator) {
return IntegrationFlows
.from(postProcessFlowGenerator.getSourceChannel())
// flow specific behavior here
.channel(postProcessFlowGenerator.getDestinationChannel())
.get();
}
}
The idea here being that the invokations of "getDestinationChannel" would create a new channel every time and bridge the output of the last generated channel to the configured "endWith" and every invokation to "getSourceChannel" returns the last created destination channel or, if there are none, the "startWith" channel.
As I write and think about this, I'm starting to think there is probably a better way but thought that I would put this out there for some input.
Thank you.
It's not currently supported directly in the DSL, but the routing slip might satisfy your needs.
If your get, dedup etc are individual flows, you can initialize the routing slip at the start of the initial flow to either include, or not, input channels for the preprocessing step(s) in the list in between the channels for the main flows.
Although there is not yet first class support in the DSL, you can use a header enricher to set up the routing slip. The header name is IntegrationMessageHeaderAccessor.ROUTING_SLIP.
EDIT
Actually, don't maintain the header yourself; scroll down the reference manual chapter about routing slip to see how to configure the HeaderEnricher using Java.
I am trying to implement the following using Spring Integration with DSL and lambda:
Given a message, send it to N consumers (via publish-subscribe). Wait for limited time and return all results that have arrived form consumers (<= N) during that interval.
Here is an example configuration I have so far:
#Configuration
#EnableIntegration
#IntegrationComponentScan
#ComponentScan
public class ExampleConfiguration {
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedRate(1000).maxMessagesPerPoll(1).get();
}
#Bean
public MessageChannel publishSubscribeChannel() {
return MessageChannels.publishSubscribe(splitterExecutorService()).applySequence(true).get();
}
#Bean
public ThreadPoolTaskExecutor splitterExecutorService() {
final ThreadPoolTaskExecutor executorService = new ThreadPoolTaskExecutor();
executorService.setCorePoolSize(3);
executorService.setMaxPoolSize(10);
return executorService;
}
#Bean
public DirectChannel errorChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel requestChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel channel1() {
return new DirectChannel();
}
#Bean
public DirectChannel channel2() {
return new DirectChannel();
}
#Bean
public DirectChannel collectorChannel() {
return new DirectChannel();
}
#Bean
public TransformerChannel1 transformerChannel1() {
return new TransformerChannel1();
}
#Bean
public TransformerChannel2 transformerChannel2() {
return new TransformerChannel2();
}
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from(errorChannel())
.handle(m -> System.err.println("[" + Thread.currentThread().getName() + "] " + m.getPayload()))
.get();
}
#Bean
public IntegrationFlow channel1Flow() {
return IntegrationFlows.from(publishSubscribeChannel())
.transform("1: "::concat)
.transform(transformerChannel1())
.channel(collectorChannel())
.get();
}
#Bean
public IntegrationFlow channel2Flow() {
return IntegrationFlows.from(publishSubscribeChannel())
.transform("2: "::concat)
.transform(transformerChannel2())
.channel(collectorChannel())
.get();
}
#Bean
public IntegrationFlow splitterFlow() {
return IntegrationFlows.from(requestChannel())
.channel(publishSubscribeChannel())
.get();
}
#Bean
public IntegrationFlow collectorFlow() {
return IntegrationFlows.from(collectorChannel())
.resequence(r -> r.releasePartialSequences(true),
null)
.aggregate(a ->
a.sendPartialResultOnExpiry(true)
.groupTimeout(500)
, null)
.get();
}
}
TransformerChannel1 and TransformerChannel2 are sample consumers and have been implemented with just a sleep to emulate delay.
The message flow is:
splitterFlow -> channel1Flow \
-> channel2Flow / -> collectorFlow
Everything seem to work as expected, but I see warnings like:
Reply message received but the receiving thread has already received a reply
which is to be expected, given that partial result was returned.
Questions:
Overall, is this a good approach?
What is the right way to gracefully service or discard those delayed messages?
How to deal with exceptions? Ideally I'd like to send them to errorChannel, but am not sure where to specify this.
Yes, the solution looks good. I guess it fits for the Scatter-Gather pattern. The implementation is provided since version 4.1.
From other side there is on more option for the aggregator since that version, too - expire-groups-upon-timeout, which is true for the aggregator by default. With this option as false you will be able to achieve your requirement to discard all those late messages. Unfortunately DSL doesn't support it yet. Hence it won't help even if you upgrade your project to use Spring Integration 4.1.
Another option for those "Reply message received but the receiving thread has already received a reply" is on the spring.integraton.messagingTemplate.throwExceptionOnLateReply = true option using spring.integration.properties file within the META-INF of one of jar.
Anyway I think that Scatter-Gather is the best solution for you use-case.
You can find here how to configure it from JavaConfig.
UPDATE
What about exceptions and error channel?
Since you get deal already with the throwExceptionOnLateReply I guess you send a message to the requestChannel via #MessagingGateway. The last one has errorChannel option. From other side the PublishSubscribeChannel has errorHandler option, for which you can use MessagePublishingErrorHandler with your errorChannel as a default one.
BTW, don't forget that Framework provides errorChannel bean and the endpoint on it for the LoggingHandler. So, think, please, if you really need to override that stuff. The default errorChannel is PublishSubscribeChannel, hence you can simply add your own subscribers to it.
I have a simple Spring Integration 4 Java DSL flow which uses a DirectChannel's LoadBalancingStrategy to round-robin Message requests to a number of possible REST Services (i.e. calls a REST service from one of two possible service endpoint URIs).
How my flow is currently configured:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
LoadBalancingStrategy loadBalancingStrategy = new RoundRobinLoadBalancingStrategy();
DirectChannel directChannel = new DirectChannel(loadBalancingStrategy);
return directChannel;
}
#Bean
public IntegrationFlow testLoadBalancing0Flow() {
return IntegrationFlows.from("test.load.balancing.ch")
.handle(restHandler0())
.channel("test.result.ch")
.get();
}
#Bean
public IntegrationFlow testLoadBalancing1Flow() {
return IntegrationFlows.from("test.load.balancing.ch")
.handle(restHandler1())
.channel("test.result.ch")
.get();
}
#Bean
public HttpRequestExecutingMessageHandler restHandler0() {
return createRestHandler(endpointUri0, 0);
}
#Bean
public HttpRequestExecutingMessageHandler restHandler1() {
return createRestHandler(endpointUri1, 1);
}
private HttpRequestExecutingMessageHandler createRestHandler(String uri, int order) {
HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(uri);
// handler configuration goes here..
handler.setOrder(order);
return handler;
}
My configuration works, but I am wondering whether there is a simpler/better way of configuring the flow using Spring Integration's Java DSL?
Cheers,
PM
First of all the RoundRobinLoadBalancingStrategy is the default one for the DirectChannel.
So, can get rid of the testLoadBalancingCh() bean definition at all.
Further, to avoid duplication for the .channel("test.result.ch") you can configure it on the HttpRequestExecutingMessageHandler as setOutputChannel().
From other side your configuration is so simple that I don't see reason to use DSL. You can achieve the same just with annotation configuration:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
return new DirectChannel();
}
#Bean(name = "test.result.ch")
public DirectChannel testResultCh() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "test.load.balancing.ch")
public HttpRequestExecutingMessageHandler restHandler0() {
return createRestHandler(endpointUri0, 0);
}
#Bean
#ServiceActivator(inputChannel = "test.load.balancing.ch")
public HttpRequestExecutingMessageHandler restHandler1() {
return createRestHandler(endpointUri1, 1);
}
private HttpRequestExecutingMessageHandler createRestHandler(String uri, int order) {
HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(uri);
// handler configuration goes here..
handler.setOrder(order);
handler.setOutputChannel(testResultCh());
return handler;
}
From other side there is MessageChannels builder factory to allow to simplify loadBalancer for your case:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
return MessageChannels.direct()
.loadBalancer(new RoundRobinLoadBalancingStrategy())
.get();
}
However, I can guess that you want to avoid duplication within DSL flow definition to DRY, but it isn't possible now. That's because IntegrationFlow is linear to tie endoints bypassing the boilerplate code for standard objects creation.
As you see to achieve Round-Robin we have to duplicate, at least, inputChannel, to subscribe several MessageHandlers to the same channel. And we do that in the XML, via Annotations and, of course, from DSL.
I'm not sure that it will be so useful for real applications to provide a hook to configure several handlers using single .handle() for the same Round-Robin channel. Because the further downstream flow may not be so simple as your .channel("test.result.ch").
Cheers