in our application,consumer started polling continuously at the load of the application and therefore sometimes it impact the execution time for one of the
method by polling in between the method execution.
Method (let say test())which ideally take few millisecond to run in junit case is now taking few seconds for the execution in app.Therfore, would like to skip the polling
at this point of time,if possible.
In spring integration doc have seen something called PollSkipAdvice/PollSkipStrategy which says The PollSkipAdvice can be used to suppress (skip) a poll.
Could you please suggest,if this can be of any help in above scenario.Would be great, if explain using example.Thanks.
sample config:
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="false" channel="inputFromKafka">
<int:poller fixed-delay="10" time-unit="MILLISECONDS"
max-messages-per-poll="5" />
</int-kafka:inbound-channel-adapter>
You scenario isn't clear. Really...
We have here only one adapter with aggressive fixed-delay every 10 MILLISECONDS and only for the small amount of messages.
Consider to increase the poll time and make max-messages-per-poll as -1 to poll all of them for the one poll task.
From other side it isn't clear how that your test() method is involved...
Also consider to switch to <int-kafka:message-driven-channel-adapter> for better control over messages.
Regarding PollSkipAdvice... Really not sure which aim would you like to reach with it...
And one more point. Bear in mind all <poller>s use the same ThreadPoolTaskScheduler with the 10 as pool. So, maybe some other long-living task keeps threads from it busy...
This your <int-kafka:inbound-channel-adapter> takes only one, but each 10 millis, of course.
Related
I just want to setup a very common server : it must accept connections and make some business calculations to return the answer. Calculations can be short or long -> I need some kind of ThreadPoolExecutor to execute these calculations.
In netty 3, that we were using since a long time, this was achieved very easily, by just using an ExecutionHandler in the pipeline, before my BusinessHandler.
But now trying to setup the same thing in netty 4, i read in the documentation that ExecutionHandler was not existing anymore, and that i add to specify a EventExecutor when adding my BusinessHandler to the channel pipeline.
DefaultEventExecutorGroup applicativeExecutorGroup = new DefaultEventExecutorGroup(10);
...
ch.pipeline().addLast(applicativeExecutorGroup, businessHandler);
It works for very basic scenarios (only short queries), but not in the following one. The reason is that DefaultEventExecutorGroup will not select a free worker, but any one based on a round-robin.
A first request (R1) comes, is assigned T1 (Thread 1 of the DefaultEventExecutorGroup), and will take a long time (say 1 minute).
Then a few other queries Ri (i=2 to 10) are received. They are assigned Ti, and are also processed successfully.
But when R11 comes, it is assigned again T1, due to the round-robin algorithm implemented in DefaultEventExecutorGroup, and the query is queued after the long R1. As a result, it will not start its processing before one minute, and that is clearly an unacceptable delay. In concrete scenarios, clients never get the answer, because they time out waiting for the answer before we start the processing.
And this continue like this. One query every 10 queries will just fail, because queued after the long one in the only busy thread, while all the other threads of the Group were just idle.
Is there another configuration of my pipeline that would work ? For example, does a implemntation of EventExecutor exist that would just work like a standard Executor (select a FREE worker).
Or is it just a bug in netty 4.1 ? It would looks very strange, as this looks as a very common scenario for any server.
Thanks for your help.
From what you explained above I think you want to use UnorderedThreadPoolEventExecutor as a replacement for DefaultEventExecutorGroup. Or if ordering is important NonStickyEventExecutorGroup.
I'm considering to use such a solution when Function is triggered by Queue on Java. I'm trying to understand how to configure batchSize and newBatchThreshold more efficiently. I would like to mention below what I managed to find out about it. Please correct me as soon as you find a mistake in my reasoning:
Function is executed on 1 CPU-core environment;
Function polls messages from Queue in batches with size 16 by default and executes them in parallel (right from the documentation);
so I make a conclusion that:
if messages need CPU-intensive tasks - they are executed sequentially;
so I make a conclusion that:
since processing of messages starts at the same time (when batch arrived) then processing of more last messages takes longer and longer (confirmed experimentally);
all these longer and longer processings are billable (despite Function's body execution lasts 10 times less);
so I make a conclusion that:
One should set both batchSize and newBatchThreshold to 1 for CPU-intensive tasks and can vary only for non-CPU intensive tasks (looks like only IO-intensive tasks).
Does it make sense?
I came across the following in the spring integration reference doc
The receiveTimeout property specifies the amount of time the poller should wait if no messages are available when it invokes the receive operation. For example, consider two options that seem similar on the surface but are actually quite different: the first has an interval trigger of 5 seconds and a receive timeout of 50 milliseconds while the second has an interval trigger of 50 milliseconds and a receive timeout of 5 seconds. The first one may receive a message up to 4950 milliseconds later than it arrived on the channel (if that message arrived immediately after one of its poll calls returned). On the other hand, the second configuration will never miss a message by more than 50 milliseconds. The difference is that the second option requires a thread to wait, but as a result it is able to respond much more quickly to arriving messages. This technique, known as long polling, can be used to emulate event-driven behavior on a polled source.
Based on my experience, the second option can cause a problem because an interval of 50 milliseconds will make the poller run every 50 millies, but if there's no messages to pick up each threads created will wait for 5 seconds for a message to appear. In that 5 seconds the poller will get executed another 100 times, potentially creating another 100 threads, etc.
This quickly runs away.
My question is did I misunderstand the way this all works? Because if I'm correct I think the reference documentation should be changed, or at least a warning added.
e<bean id="store" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore">
<property name="dataSource" ref="channelServerDataSource"/>
<property name="channelMessageStoreQueryProvider" ref="queryProvider"/>
<property name="region" value="${user.name}_${channel.queue.region:default}"/>
<property name="usingIdCache" value="false"/>
</bean>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="#store.removeFromIdCache(headers.id.toString())" />
<int:after-rollback expression="#store.removeFromIdCache(headers.id.toString())"/>
</int:transaction-synchronization-factory>
<int:channel id="transacitonAsyncServiceQueue">
<int:queue message-store="store"/>
<!-- <int:queue/> -->
</int:channel>
<bean id="rxPollingTrigger" class="org.springframework.scheduling.support.PeriodicTrigger">
<constructor-arg value="500"/>
<constructor-arg value="MILLISECONDS"/>
<property name = "initialDelay" value = "30000"/>
<!-- initialDelay important to ensure channel doesnt start processing before the datasources have been initialised becuase we
now persist transactions in the queue, at startup (restart) there maybe some ready to go which get processed before the
connection pools have been created which happens when the servlet is first hit -->
</bean>
<int:service-activator ref="asyncChannelReceiver" method="processMessage" input-channel="transacitonAsyncServiceQueue">
<int:poller trigger="rxPollingTrigger" max-messages-per-poll="20" task-executor="taskExecutor" receive-timeout="400">
<int:transactional transaction-manager="transactionManagerAsyncChannel" />
</int:poller>
<int:request-handler-advice-chain>
<ref bean="databaseSessionContext" />
</int:request-handler-advice-chain>
</int:service-activator>
<task:executor id="taskExecutor" pool-size="100-200" queue-capacity="200" keep-alive="1" rejection-policy="CALLER_RUNS" />
My question is did I misunderstand the way this all works?
Yes, you misunderstand.
The trigger (in this case a PeriodicTrigger with an interval of 50ms) is only consulted to calculate the next poll time when the current poll exits.
There is only one poller thread running concurrently. If there is no message, the poll thread is suspended for 5 seconds; the trigger is then consulted (t.nextExecutionTime()) and the next poll is scheduled for +50ms; so, with no data a single thread will run every 5.05 seconds.
When messages are present, and you wish to have a concurrency greater than one, you would use a task executor to allow the poller thread to hand off to another thread so that the trigger is immediately consulted for the next poll time.
Based on my experience
Please clarify "your experience" and show configuration, evidence etc.
If you have a suspected thread leak, the first step, generally, is to take a thread dump to figure out what they are all doing.
EDIT: (in response to your comments below).
There's not really a downside of CALLER_RUNS in this scenario because, although the current thread "jumps ahead" of the queued tasks, it's not like this poll has newer data than the queued tasks, it's just a poll after all. However, poller threads are a limited resource (although the limit can be changed) so long-running tasks on a poller thread are generally discouraged.
ABORT could cause some noise in the logs; an alternative is to configure a PollSkipAdvice where the advice can look at the task queue and silently ignore the current poll. In 4.2, we've added even more flexibility to the poller.
You will find many articles on the internet that say that using RDBMS as a queue is not the greatest idea; you might want to consider a JMS or rabbitmq-backed channel instead. If you are tied to JDBC, you should be sure to use the JdbcChannelMessageStore and not the JdbcMessageStore. The former is preferred for backing channels since it only uses 1 table; the latter has some performance issues when used to back a channel, because of contention on the message group table. See Backing Message Channels in the JDBC Support chapter for more information.
I have a question on Mule threading profile Exhausted_Action. From the documentation, I understand that when the action is WAIT, any new request beyond the maxActive will wait for a thread to be available. Whereas action of RUN, would cause use the original thread to process the request. From my understanding, I thought WAIT is better way to do it, rather than RUN. However, it appears MULE has all default values set to RUN. Just want to hear for comments on my understanding and differences between these two actions and how to decide which one to use when.
Your undertanding about WAIT and RUN is correct.
Reason why all the default values are RUN is that, The message processing is not stopped because of unavailability of flow thread. Because the original thread(or receiver thread) is anyway waiting for the Flow thread to take the message and process, why not process it. (This is my opinion).
But there is a downside for using RUN.
Ex:
No of receiver threads are restricted to 2.
<asynchronous=processing-strategy name="customAsynchronous" maxThreads="1" />
<flow name="sample" processingStrategy="customAsynchronous" >
<file:inbound-endpoint ......>
............
..........
</flow>
File sizes: 1MB, 50MB, 100MB, 1MB, 5MB.
In the above flow when there are 5 files coming in. 3 files are processed as there is 1 flow thread available and 2 File Receiver threads (Exhausted_Action = RUN). The flow thread will finish the processing fast as the first file is small and keeps waiting for the next message. Unfortunately the receiver thread whose job is to pick the next file and give it to Flow thread to process is busy processing the BIG file. This way there is a chance of receiver threads getting struck in time consuming processing while the flow threads are waiting.
So it is always depending on the usecase you are dealing with.
Hope this helps.
I'm struggling with how to affect (upward) the number of threads that a VM transport in Mule uses. I've read the Tuning & Performance page in Mule's documentation, but something isn't clicking with me.
I have one flow that executes a JDBC call and then shoves the result set into VM queues using a foreach.
<foreach>
<vm:outbound-endpoint exchange-pattern="one-way" path="checkService"/>
</foreach>
This is picked by another flow for processing, which consists of making an HTTPS call out and checking the return value.
<flow name="ExecuteTests" doc:name="ExecuteTests">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService"/>
<https:outbound-endpoint exchange-pattern="request-response"...
...etc.
</flow>
Some of those calls are quick, but some take up to 5 seconds. What I'd like is for the ExecuteTests flow to use more threads to do the processing, but I only ever see threads 02-05 in the logs. I would have expected to see the number of threads used closer to the dispatcher thread pool for the outbound HTTPS connector...which I thought defaulted to 16.
I tried the following:
<vm:connector name="vmConnector">
<receiver-threading-profile maxThreadsActive="100" maxThreadsIdle="100"/>
</vm:connector>
<flow name="ExecuteTests" doc:name="ExecuteTests">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService" connector-ref="vmConnector"/>
...etc.
but it made no difference.
Thinking that maybe the buffer for the inbound endpoint was messing with things, I tried:
<vm:connector name="vmConnector">
<receiver-threading-profile maxThreadsActive="100" maxThreadsIdle="100"/>
<vm:queue-profile maxOutstandingMessages="1"/>
</vm:connector>
but it didn't help either.
What am I missing?
From your flow it can be observed that the issue is not with the threads.(I believe).
Normally the VM receiver threads are fast and they scale up based on the number of requests coming in.
In the "ExecuteTests" flow there seems to be an outbound HTTP call. So it is the HTTP call that might be the cause for the delay.
But still if you want to increase the threads try adding a dispatcher threading profile for a HTTP connector.
Also inorder to increase the number of threads for your flow processing use Flow threading profiles.
Like
<asynchronous-processing-strategy name="executeTestsStrategy" maxThreads="40" minThreads="20" />
<flow name="ExecuteTests" doc:name="ExecuteTests" processingStrategy="executeTestsStrategy">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService"/>
<https:outbound-endpoint exchange-pattern="request-response"...
...etc.
</flow>
Hope this helps.