I have the following configuration which uses Redis as message store. I don't have java code for this module, but only this configuration file. This configuration has the following functionality: When this module receives a message from input channel, it will check the Redis store, if the message doesn't exist (the expression will be evaluated to TRUE), the message will be sent to the output channel which will be put into Redis; if the message already exists (the expression is evaluated to False), the message will be discarded.
Suppose this module is called RedisModule, so i have a stream:
RedisMdule | log
The problem is that: when I sent messages to this module, in the log file, it shows the messages #2, #4, #6 etc, the first message is missing, so are the odd number messages. Are there anything I have missed in this configuration file ? Many Thanks.
<int:channel id="input"/>
<int:channel id="output"/>
<int:filter input-channel="input"
output-channel="output"
discard-channel="nullChannel"
expression="#metadataStore.get(payload) == null"/>
<int:outbound-channel-adapter channel="output"
expression="#metadataStore.put(payload, '')"/>
From big height it looks like you need Idempotent Receiver, which does exactly similar logic but in the atomic manner. See MetadataStoreSelector source code:
return this.metadataStore.putIfAbsent(key, value) == null;
So, you can configure an <idempotent-receiver> with your RedisMetadataStore and use payload as a key-expression option.
It isn't clear by your post how you get logs, because <int:outbound-channel-adapter> is one-way component.
Maybe you are there in the Spring XD? Where you use the output channel for your own purpose, but that really should be an output of your module.
That's really may the reason why you see only even messages, because odd are sent to your <int:outbound-channel-adapter> and the DirectChannel uses round-robin balancing strategy by default.
With the <idempotent-receiver> you should have only <bridge input-channel="input" output-channel="output"/> in your RedisMdule.
Related
When I try to shutdown my spring-integration process, the flow using an inbound Jms.messageDrivenChannelAdapter throws the following error message:
"org.springframework.jms.listener.DefaultMessageListenerContainer - Rejecting received message because of the listener container been stopeed in the meantime"
my inbound adapter is defined as follows:
Jms.messageDrivenChannelAdapter(
Jms.container(jmsConnectionFactory, destinationName)
.concurrency(highConcurrency)
.get()
)
I believe that my problem is that the default "receiveTimeout" on my jms container is too small and that I need to increase that value to cater for my "high-concurrency" (right ?), as "receiveTimeout" seems to be the only value the container "doShutdown" method cares about.
Now, the sourceCode for the receiveTimeout property says "this value needs to be smaller than the transaction timeout". Also the spring-integration doco regarding inbound jms adapters says "if you want the entire flow to be transactional [...] consider using a jms-message-driven-channel-adapter with acknowledge set to transacted (the default)", which seems to imply that the jms adapter is transactional by default.
Hence, my main question is: even though I'm not using any explicit transaction manager, do I need to not only explicitely set "receiveTimeout" on my container but also "transactionTimeout" with transactionTimeout > receiveTimeout ?
Thanks a lot in advance for your expertise and your time.
Best Regards
That is not "throws". That is just warn:
protected void doExecuteListener(Session session, Message message) throws JMSException {
if (!isAcceptMessagesWhileStopping() && !isRunning()) {
if (logger.isWarnEnabled()) {
logger.warn("Rejecting received message because of the listener container " +
"having been stopped in the meantime: " + message);
}
rollbackIfNecessary(session);
throw new MessageRejectedWhileStoppingException();
}
And pay attention to that rollbackIfNecessary(session);. So, even if the received message slips somehow into this listener function, the whole environment makes it sure that the state is not broken and the data is not lost - the session is rolled back.
The transactionTimeout does not make sense if you don't use a transactionManager. Spring Integration makes it transacted exactly for the use-case we see around that warn log.
I am working with a project which is using JMS listener to receive incoming message, and then route to different destination, currently the process only pick one destination among below 3 for each incoming message. so the xml configuration is written as below
<integration:router ref="jmsRouter" input-channel="jmsFilterOutput" default-output-channel="jmsRouterOutput" />
<integration:service-activator id="serviceActivator1" input-channel="input1"
ref="messageProcessService" method="callMsgProcessor1" />
<integration:service-activator id="serviceActivator2" input-channel="input2"
ref="messageProcessService" method="callMsgProcessor2" />
<integration:service-activator id="serviceActivator3" input-channel="sharedInput"
ref="messageProcessService" method="callMsgProcessor3" output-channel="reqChannel" />
among above 3 serviceActivator, the output-channel of the last one is defined as IBM mq in another xml configuration file.
now my job is to generate a different message from sharedInput, and send to a different queue in parallel
so I add a line as below
<integration:service-activator id="serviceActivator4" input-channel="sharedInput"
ref="messageProcessService" method="callMsgProcessorNew" output-channel="reqChannelNew" />
however when running JMS, the message from sharedInput only goes to callMsgProcessor3, and the populated message is sent to reqChannel only as well, and ignore my new destination. if I comment out the third service activator, sharedInput can go to callMsgProcessorNew, and route to new queue.
can anyone advise how I should configure to push the sharedInput go to two processors (callMsgProcessor3 and callMsgProcessorNew), and also sent to their corresponding output mq channel in parallel?
I googled online, seems router splitter or recipient list router can solve my problem? but still feeling confused after reading the related doc, and not sure how to configure it in my case. appreciate if someone can help provide a sample
please let me know if I need to provide more info to clarify the issue.
You can make a sharedInput as a PublishSubscribeChannel and have another service activator subscribed to it so the same message will go to both of them. After that you can make absolutely different flows and do whatever logic you need to parallel. See docs for more info: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations-publishsubscribechannel.
Also respective EIP determination : https://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html
thanks for your reply, #Artem! I realized one thing, sharedInput only go to one destination is because it is one message. if I can duplicate the message, it will go to two destination. so I add recipient-list-router, and made change as below, and it worked!
<integration:recipient-list-router id="duplicateMsgRouter" input-channel="sharedInput"
timeout="1234"
ignore-send-failures="true"
apply-sequence="true">
<integration:recipient channel="channel1"/>
<integration:recipient channel="channel2"/>
</integration:recipient-list-router>
<integration:service-activator id="serviceActivator3" input-channel="channel1"
ref="messageProcessService" method="callMsgProcessor3" output-channel="reqChannel" />
<integration:service-activator id="serviceActivator4" input-channel="channel2"
ref="messageProcessService" method="callMsgProcessorNew" output-channel="reqChannelNew" />
My req. is to poll a directory for a specified time interval say 10 mins. If a file of a particular extension say *.xml is found in the directory then the it just consumes (i.e. picks and deletes) the file and prints the name else after the specified time (say 10 mins.) interval it sends out a mail that the file has not been picked (i.e. consumed) or the file has not come.
There are 2 options either I do it through Spring integration OR WatchService of Core Java. Following is the code in Spring Integration which I have written till now:
<int:channel id="fileChannel" />
<int:channel id="processedFileChannel" />
<context:property-placeholder location="localProps.properties" />
<int:poller default="true" fixed-rate="10000" id="poller"></int:poller>
<int-file:inbound-channel-adapter
directory="file:${inbound.folder}" channel="fileChannel"
filename-pattern="*.xml" />
<int:service-activator input-channel="fileChannel"
ref="fileHandlerService" method="processFile" output-channel="processedFileChannel"/>
<bean id="fileHandlerService" class="com.practice.cmrs.springintegration.Poll" />
The above code is successfully polling the folder for a particular file pattern. Now I have 2 things to do:
1) Stop polling after a particular time interval (configurable) say 10 mins.
2) Check whether a file with a particular extension is there in the folder ... if the file is there (it consumes and then deletes) else it sends an email to a group of people (email part is done.)
Please help me in the above 2 points.
You can use a Smart Poller to do things like that.
You can adjust the poller and/or take different actions if/when the poll results in a message.
Version 4.2 introduced the AbstractMessageSourceAdvice. Any Advice objects in the advice-chain that subclass this class, are applied to just the receive operation. Such classes implement the following methods:
beforeReceive(MessageSource<?> source)
This method is called before the MessageSource.receive() method. It enables you to examine and or reconfigure the source at this time. Returning false cancels this poll (similar to the PollSkipAdvice mentioned above).
Message<?> afterReceive(Message<?> result, MessageSource<?> source)
This method is called after the receive() method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be null if there was no message created by the source). You can even return a different message!
EDIT: Here is a gist showing my log. It appears that there is ReceiveMessage and then a preSend on inputChannel:
https://gist.github.com/louisalexander/04e7d95835521efdd15455c98075e2ea
Apologies for being so dense, but I can't seem to figure out how to properly make use of the sqs-message-driven-channel-adapter
In my context file, I am configuring it as such:
<int-aws:sqs-message-driven-channel-adapter
id="my-message-driven-adapter" sqs="sqs" queues="some-queue-of-mine"
max-number-of-messages="5" visibility-timeout="200" wait-time-out="10"
send-timeout="2000" channel="inputChannel" />
I observe that messages are properly making it into some-queue-of-mine (by removing the above bit of code and sending messages to the queue). I then restart my server, enabling the message driven adapter and I observe that all the messages are consumed from the queue, but where did they go? :-/
My expectation was that the messages would be funneled into a DirectChannel named inputChannel:
<int:channel id="inputChannel"/>
That I have a service-activator consuming from as follows:
<int:service-activator ref="myConsumer"
method='execute' input-channel="inputChannel" output-channel="outputChannel">
<int:request-handler-advice-chain>
...
</int:request-handler-advice-chain>
</int:service-activator>
But of course, I am never seeing myConsumer get invoked. I imagine my understanding of how the MessageProducer mechanism works is inadequate. Can someone please correct my thinking by providing a trivial example of XML wiring?
According to the Logs the message is consumed by the handler.AbstractMessageHandler (AbstractMessageHandler.java:115) - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor#493f49cd]. Although that might be a fully different story.
SqsMessageDrivenChannelAdapter can be supplied with the errorChannel to handle downstream exceptions. By default it is only logged.
The message sent from that adapter is like:
return new GenericMessage<>(message.getBody(), new SqsMessageHeaders(messageHeaders));
Where that message.getBody() is String. See QueueMessageUtils.createMessage().
So, be sure that your service-activator accepts that String as a paylaod and not any other type.
I am using spring integration to download files and to process them.
<int-sftp:inbound-channel-adapter channel="FileDownloadChannel"
session-factory="SftpSessionFactory"
remote-directory="/home/sshaji/from_disney/files"
filter = "modifiedFileListFilter"
local-directory="/home/sshaji/to_disney/downloads"
auto-create-local-directory="true" >
<integration:poller cron="*/10 * * * * *" default="true"/>
</int-sftp:inbound-channel-adapter>
<integration:transformer input-channel="FileDownloadChannel"
ref="ErrorTransformer"
output-channel="EndChannel"/>
The execution is started by the poller.
It calls the "FileDownloadChannel" and then tries to download files from the sftp server.
I want to specify an output channel for this inbound-channel-adaptor but it doesnot have any output-channel attribute.
So i named the transformer with the same name as that of inbound-channel-adaptor so that it will also be called once poller starts.
My issue is that the transformer gets called before the download happens and hence transformer wont get any inputs to process and causes error.
Is there any way we can specify "order" attribute for this two tasks. or is there any workaround for the output-channel for the inbound-channel adaptor?.
I would really appreciate any help on this.
You need to read the Spring Integration Reference Manual and work through some sample applications.
Channel adapters don't have input and output channels, they have channels. Channel adapters either produce or consume a message (inbound Vs outbound) on their channel. Elements such as transformers, service activators etc, that receive a message and produce a reply, have input and output channels.
"My issue is that the transformer gets called before the download happens and hence transformer wont get any inputs to process and causes error."
This statement makes no sense to me; if there's no file yet, there's nothing to "call" the transformer with.
"attribute for this two tasks."
There are not two "tasks".
The poller thread invokes the inbound adapter; then, when a file arrives, it is sent as a message to the configured channel which, with your configuration, means the poller thread invokes the transformer with the message.
A Channel Adapter is a Message Endpoint that enables connecting a single sender or receiver to a Message Channel.
In your case, output channel is 'FileDownloadChannel'
<integration:channel id="FileDownloadChannel"/>
<int-sftp:inbound-channel-adapter channel="FileDownloadChannel" ...>
<integration:poller fixed-rate="10000"/>
</int-sftp:inbound-channel-adapter>
In order to execute your tasks in order, you can use the Message Handler chain as follows:
<integration:channel id="outputChannel"/>
<chain input-channel="FileDownloadChannel" output-channel="outputChannel">
<filter ref="someSelector" throw-exception-on-rejection="true"/>
<transformer ref="ErrorTransformer"/>
<service-activator ref="someService" method="someMethod"/>
</chain>