I have poller configuration like this:
<int:poller id="myPoller" default="true" max-messages-per-poll="2" task-executor="executor"
error-channel="errorChannel" receive-timeout="20000" trigger="dynamicTrigger">
</int:poller>
But i need to trigger the poller with last time message from previous poll. For example of the flow:
If i have 4(A,B,C,D) Post rest the app will process 2(A,B) message first and then wait for the last response (either from A or B), afterward trigger the poll for next message (C,D).
How to achieve that condition ? Thx.
Well, I suggest you to take a look into poller's <advice-chain>. With some custom MethodInterceptor you perform your decision logic and call your polling source stop(). When you have a desired response somewhere downstream you just perform start() of that endpoint again to let proceed with a new bunch of messages.
With enough big polling-interval you even don't need that <advice-chain> and just perform stop() in the downstream as well, via some state checking.
Also consider to use Control Bus for stop/start operations.
Related
Our TransactionSynchronizationFactory is deleting the source file even before the flow ends and this is causing a failure in the flow. After reading the file, we split(), make a WebClient Gateway call, resequence() and then aggregate(). Just after aggregation the TransactionSynchronizationFactory is performing a commit. Please suggest why is the behavior?
syncProcessor.setAfterCommitExpression(parser.parseExpression("payload.delete()"));
syncProcessor.setAfterRollbackExpression(parser.parseExpression("payload.delete()"));
return Pollers.fixedDelay(Duration.ofMinutes(pollInterval))
.maxMessagesPerPoll(maxMessagesPerPoll)
.transactionSynchronizationFactory(transactionSynchronizationFactory)
.transactional(pseudoTransactionManager)
.advice(loggingAdvice);
The transaction synchronization is tied to a thread which has started transaction. Whenever you leave that thread (kinda unblock), the end of transaction is triggered. Be sure that you don't shift a message after that aggregate() to some other thread, e.g. via an ExecutorChannel or QueueChannel.
In addition I would look into some other solution where you are not tied to transaction and threading model. Just have the file stored in the headers and whenever you done call its delete()! No reason to deal with transaction with simple files.
I'm using an Azure Functions Queue Trigger to read in a queue message and process work in Python. I have my batchSize parameter set to 1 to limit concurrency. In the Python code, I have an additional check to see if the work is ready to be processed as in:
if work is ready:
do_work()
else:
dequeue the message and try again
I want to dequeue the message if the work isn't ready to be processed and try again in one minute. I think I can achieve the retry by just setting visibilityTimeout to 00:01:00. But what is the best way to dequeue the message? Would it just be:
if work is ready:
do_work()
else:
raise Exception("work is not ready, dequeue the message")
Would this allow the message to be dequeued and picked up for a retry in one minute? This use case is fairly rare and I'm wondering if there is a better way because I'm not sure if I want to see the Error in my logs for this case.
Edit: is it possible to use initialVisibilityDelay in my Python QueueTrigger? This would eliminate the need for me to throw an exception if I could delay the visibility of my messages by one minute.
is it possible to use initialVisibilityDelay in my Python
QueueTrigger?
How are you putting your messages to the queue initially? If it's with the Azure Queue storage output binding, then I think you can't easily set the initial visibility delay.
You can however archieve what you want by using the queue's put_message function:
put_message(queue_name, content, visibility_timeout=None, time_to_live=None, timeout=None)
As you can see, it accepts the visibility_timeout parameter to make a message "active" after a specific delay, which is what you're after if I understood correctly.
I'm trying to implement a Poller with a DynamicPeriodicTrigger which would backoff (increase duration between polls) if the pollable message source (e.g. FTP server) becomes unavailable, a bit like what is already done through SimpleActiveIdleMessageSourceAdvice but the advice would need to be able to catch the exception thrown during the poll. Unfortunately the invoke method of AbstractMessageSourceAdvice is final, so I can't overwrite it.
I also tried a different approach which is to catch the poll exception by having the poller forward it to an error-channel, where I can increase the duration of the trigger (that part works ok). The problem in that case is how to reset the trigger the next time the poll succeed (i.e. the message source is available again). I can't just reset the trigger in a downstream handler method because the message source may have recovered, but there could still be no message available (in which case my downstream handler method is never called to reset the duration of the trigger).
Thank you very much advance for your expertise and your time.
Best Regards
You don't have to override AbstractMessageSourceAdvice; as you can see its invoke method is pretty trivial; just copy it and add functionality as needed (just be sure to implement MessageSourceMutator so it's detected as a receive-only advice).
Maybe it's as simple as moving the invocation.proceed() to a protected non-final method.
If you come up with something that you think will be generally useful to the community, consider contributing it back to the framework.
I would like to make a flow initiated by syslog inbound-channel-adapter messages transactional. Unfortunately the adapter does not take a poller which could be made transactional (the typical approach used in many examples with other inbound adapters).
Is there any workaround?
EDIT
After some thinking I realized that my intent is a little different than initially described (hence change of the title). Basically all I want to do is some simple and straight-forward way of making an arbitrary part (starting from some arbitrary channel in the flow) of my message flow pseudo-transactional. Meaning - I want to execute some custom code if the flow completes without any exceptions (but note that I don't want my custom pseudo commit code to be a part (step) of a flow itself). And I want to execute some custom code if any exception occured.
Semantics of using TransactionSynchronizationFactory would suite me very well.
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.renameTo('/success/' + payload.name)" channel="committedChannel" />
<int:after-rollback expression="payload.renameTo('/failed/' + payload.name)" channel="rolledBackChannel" />
</int:transaction-synchronization-factory>
The only problem is how-to wire it together with the rest of the flow. What I tried is to define intermediate dummy service-activator endpoint that receives messages from the channel where I want the transaction to begin. And then add transactional poller to that service-activator. But this approach has problems of its own because in order to use poller you have to define the incoming channel as a queue channel which seem to make execution of the flow in a separate thread (or at least I observed some async behaveour).
Any flow from a message-driven adapter can be run in the scope of a transaction by making the adapter's channel start a transaction:
<tx:advice id="txAdvice" transaction-manager="transactionManager">
<tx:attributes>
<tx:method name="send"/>
</tx:attributes>
</tx:advice>
<aop:config>
<aop:advisor advice-ref="txAdvice" pointcut="bean(fromSyslog)"/>
</aop:config>
<int:channel id="fromSyslog" />
The channel (and all downstream channels) must be a DirectChannel (no queue channels, or task executors). After an async handoff to another thread, the transaction will commit.
I am sure you understand, but for the benefit of other readers, this does not make the syslog adapter itself transactional, just the downstream flow.
EDIT
Another technique is to use a mid-flow transactional gateway...
#Transactional
public interface TxGate {
void oneWay(Message<?> message);
}
<int:service-activator input-channel="fromSyslog" ref="txGate" />
<int:gateway id="txGate" service-interface="foo.TxGate"
default-request-channel="txSyslog" error-channel="foo" />
That way, you can handle an exception within the scope of the transaction and decide whether or not to commit (throwing an exception from the error flow will rollback).
The void return is important; since the downstream flow doesn't return a reply.
EDIT2
In response to your edited question.
So, it seems your issue with the solutions I provided (specifically in the mid-flow transactional gateway) only allows you to take some action if there is an error when you also want to take some (different) action after success.
There are two ways to do that.
Make the last channel in the subflow a publish-subscribe-channel; add a second consumer (use order to explicitly define the order in which they are called) and take your 'success' action on the second subscriber - he won't be called after an exception (by default) so you'd continue to handle the exception case on the gateway's error channel.
Use a ChannelInterceptor. Start the transaction in preSend(). The afterSendCompletion() method will be invoked after the subflow completes. The presence (or not) of the Exception argument is populated if the subflow throws an exception.
We could consider adding such an interceptor to the framework, if you want to consider contributing it.
in our application,consumer started polling continuously at the load of the application and therefore sometimes it impact the execution time for one of the
method by polling in between the method execution.
Method (let say test())which ideally take few millisecond to run in junit case is now taking few seconds for the execution in app.Therfore, would like to skip the polling
at this point of time,if possible.
In spring integration doc have seen something called PollSkipAdvice/PollSkipStrategy which says The PollSkipAdvice can be used to suppress (skip) a poll.
Could you please suggest,if this can be of any help in above scenario.Would be great, if explain using example.Thanks.
sample config:
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="false" channel="inputFromKafka">
<int:poller fixed-delay="10" time-unit="MILLISECONDS"
max-messages-per-poll="5" />
</int-kafka:inbound-channel-adapter>
You scenario isn't clear. Really...
We have here only one adapter with aggressive fixed-delay every 10 MILLISECONDS and only for the small amount of messages.
Consider to increase the poll time and make max-messages-per-poll as -1 to poll all of them for the one poll task.
From other side it isn't clear how that your test() method is involved...
Also consider to switch to <int-kafka:message-driven-channel-adapter> for better control over messages.
Regarding PollSkipAdvice... Really not sure which aim would you like to reach with it...
And one more point. Bear in mind all <poller>s use the same ThreadPoolTaskScheduler with the 10 as pool. So, maybe some other long-living task keeps threads from it busy...
This your <int-kafka:inbound-channel-adapter> takes only one, but each 10 millis, of course.