I have a flow that starts with a poller and hands off the message to several async flows downstream using task-executors to execute in parallel for a given dataset. A downstream aggregator completes the flow and notifies the poller that the flow is complete.
I would like to track every execution of the poller by using MDC so that the logs can be mapped to a particular execution of the flow.
I started by adding MDC to the poller thread (using Advice), however with this approach there could be a couple of issues:
How do I stamp the MDC on the executor thread when the async hand off happens?
Since executor uses a a thread pool, do I need to clear the MDC before the thread returns to the pool? Will there be any side effects?
Another approach would be to add MDC to the Message header and set it manually on the new thread during the async handoff. How to do that? For example, if I turn on the debug logs, the MDC should be stamped right from the beginning of the new thread execution and not from the point where my logic starts in the service activator.
How to set this on the task-executor thread (and probably also remove before returning to the pool) using XML configuration? Something like an MdcAwareThreadPoolExecutor seen here.
Also, I would not want the MDC logic to be spread across all the async handoff endpoints, may be there is some generic way to configure it?
Is there a better way to achieve this? Any known solutions?
I would like to track every execution of the poller by using MDC so that the logs can be mapped to a particular execution of the flow.
It is fully sound as "you would like to track the message journey in your flow". As you noticed there is the way to set some message header. So, why just don't map your logs by this specific header?
You can take a look into Message History pattern how to gather the whole path for the message, so then in logs you can track it back looking into message headers.
See here: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/system-management.html#message-history
If you really still insist on the MDC, then you definitely need to take a look into some MDCDelegatingExecutorDecorator. Some sample you can borrow from Spring Security and its DelegatingSecurityContextExecutor`: https://docs.spring.io/spring-security/site/docs/5.4.0/reference/html5/#concurrency
Related
Is there a way we could instruct Spring IntegrationFlow DSL to avoid concurrent execution, for example - first run is still not completed however as per the Poller it is time for the second run? Similar to #DisallowConcurrentExecution we use in Spring Batch Quartz scheduler.
Thanks
To avoid concurrent execution from the polling channel adapter, you really must not use anything that could lead for parallelism. First: don't use a TaskExecutor for the polling tasks. Second don't use fixedRate and just fixedDelay. The second one behaves the way that it does schedule the next polling task only when the previous has finished. And if you don't shift the work to other thread (see that TaskExecutor), anything is going to be performed on the same scheduled thread.
Technically what you are asking is there by default if you just use a fixedDelay for the poller and no more other options are configured.
Although you need to keep in mind that that the rest of your flow must be direct as well: no ExecutorChannel or QueueChannel in use!
Also see docs for conditional polling if you still cannot make your flow direct and blocking: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#conditional-pollers
I would like to know if I can have persistence in my Spring Integration setup when I use a aggregator, which is not backed by a MessageStore, by leveraging the persistence of AMQP (RabbitMQ) queues before and after the aggregator.
I imagine that this would use ack's: The aggregator won't ack a message before it's collected all the parts and sent out the resulting message.
Additionally I would like to know if this is ever a good idea :)
I am new working with queue's, and am trying to get a good feel for patterns to use.
My business logic for this is as follows:
I receive a messages on one queue.
Each message must result in two unrelated webservice calls (preferably in parallel).
The results of these two calls must be combined with details from the original message.
The combination must then be sent out as a new message on a queue.
Messages are important, so they must not be lost.
I was/am hoping to use only one 'persistent' system, namely RabbitMQ, and not having to add a database as well.
I've tried to keep the question specific, but any other suggestions on how to approach this are greatly appreciated :)
What you would like to do recalls me Scatter-Gather EI Pattern.
So, you get a message from the AMQP send it into the ScatterGather endpoint and wait for the aggregated reply. That's enough for to stick with the default acknowledge.
Right, the scatterChannel can be PublishSubscribeChannel with an executor to call Web Services in parallel. Anyway the gatherer process will wait for replies according the release strategy and will block the original AMQP listener do not ack the message prematurely.
We need to implement a Async web service.
Behaviour of web service:
We send the request for an account to server and it sends back the sync response with an acknowledgement ID. After that we get multiple Callback requests which contains that acknowldegment ID. The last callback request for an acknowledgement ID will contain a text(completed:true) in the response which will tell us that this is the last callback request for that account and acknowledgement ID. This will help us to know that async call for a particular account is completed and we can mark its final status. We need to execute this web service for multiple accounts. So, we will be getting callback requests for many accounts.
Question:
What is the optimal way to process these multiple callback requests coming for multiple accounts.
Solutions that we thought of:
ExecutorService Fixed Thread Pool: This will parallely process our callback requests but the concern is that it does not maintain the sequence. So it will be difficult for us to determine that the last callback request for an acknowledgment ID(account) has come. Hence, we will not be able to mark the final status of that account as completed with surity.
ExecutorService Single Thread Executor: Here, only one thread is there in the pool with an unbouded queue. If we use this then processing will be pretty slow as only one thread will be actually processing.
Please suggest an optimal way to implement requirement both memory and performance wise.
Let's be clear about one thing: HTTP is a blocking, synchronous protocol. Request/response pairs aren't asynch. What you're doing is spawning asynch requests and returning to the caller to let them know the request is being processed (HTTP 200) or not (HTTP 500).
I'm not sure that I know optimal for this situation, but there are other considerations:
Use an ExecutorServiceThreadPool that you can configure. Make sure you have a prefix that lets you distinguish these threads from others.
Add request task to a blocking dequeue and have a pool of consumer threads process them. You can tune the dequeue and the consumer thread pool sizes.
If processing is intensive, send request messages to a queue running on another server. Have a pool of queue listeners process the requests.
You cannot assume that the callbacks will return in a certain order. Don't depend on "last" being "true". You'll have to join all those threads together to know when they're finished.
It sounds like the web service should have a URL that lets users query for status.
In my spring integration application i have several stored-proc-outbound-gateway, i would to log how much time each call is taking, any help would be appreciated.
I would ideally like to be able to enable/disable logging for the parameters used, time taken and total rows retrieved (returning-resultset) to monitor and performance tuning purpose.
Thanks
You can add a ChannelInterceptor (subclass of ChannelInterceptorAdapter) to the request channel which will give you raw timing (preSend/postSend), but the time will include any processing downstream of the gateway (on direct channels).
Since you want to examine the results too, you could start a timer (e.g. Spring StopWatch) in the interceptor (preSend) on the request channel and stop the timer in an interceptor on the reply channel). If you use the same interceptor bean you can store the timer in a ThreadLocal.
You can turn on/off collection using a boolean property on the interceptor.
Alternatively, you can add a custom advice to the gateway.
EDIT
The advice is probably the best approach because with a ThreadLocal you will need to add code to the first interceptor to handle failures and clean up. With an around advice, the timer would just be a local method variable.
The SI workflow starts with an inbound-channel-adapter which runs under a new transaction started by a pollar.
The adapter triggers a data processing flow which kind of fans out, for example the adapter polls a database and get few rows, splits and next adapter makes another db call (hence for each row) as input which produces several other rows and so on.
Right now its running as single threaded behavior as I only want to commit the original transaction when everything went well.
Now I want to to speed up the processing by running it under more threads, so if one of my original adapter produced 3 rows, I want to process them simultaneously for downstream flow and so on.
is this possible and if yes, how can I define a global task executor with some configuration and allow the processing at various stages to be executed under this.
Thank you
You could insert a <gateway/> in the flow between the inbound adapter and the rest of the flow; the poller thread will be suspended in the gateway awaiting a reply. You would likely need an aggregator to send the reply to the gateway when the other tasks are complete. You would discard the reply.
<service-activator input-channel="fromAdapter" ref="gw" output-channel="nullChannel" />
<gateway default-request-channel="toRestOfFlow" />
The problem is if the reply is never received the thread will wait forever. If you add a reply-timeout, the transaction will commit after the timeout. So you may need some additional logic to handle that, perhaps adding your own bean that invokes the gateway and detects a null reply.
Of course, in all these scenarios; the async tasks cannot participate in the original transaction.