I'm struggling with how to affect (upward) the number of threads that a VM transport in Mule uses. I've read the Tuning & Performance page in Mule's documentation, but something isn't clicking with me.
I have one flow that executes a JDBC call and then shoves the result set into VM queues using a foreach.
<foreach>
<vm:outbound-endpoint exchange-pattern="one-way" path="checkService"/>
</foreach>
This is picked by another flow for processing, which consists of making an HTTPS call out and checking the return value.
<flow name="ExecuteTests" doc:name="ExecuteTests">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService"/>
<https:outbound-endpoint exchange-pattern="request-response"...
...etc.
</flow>
Some of those calls are quick, but some take up to 5 seconds. What I'd like is for the ExecuteTests flow to use more threads to do the processing, but I only ever see threads 02-05 in the logs. I would have expected to see the number of threads used closer to the dispatcher thread pool for the outbound HTTPS connector...which I thought defaulted to 16.
I tried the following:
<vm:connector name="vmConnector">
<receiver-threading-profile maxThreadsActive="100" maxThreadsIdle="100"/>
</vm:connector>
<flow name="ExecuteTests" doc:name="ExecuteTests">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService" connector-ref="vmConnector"/>
...etc.
but it made no difference.
Thinking that maybe the buffer for the inbound endpoint was messing with things, I tried:
<vm:connector name="vmConnector">
<receiver-threading-profile maxThreadsActive="100" maxThreadsIdle="100"/>
<vm:queue-profile maxOutstandingMessages="1"/>
</vm:connector>
but it didn't help either.
What am I missing?
From your flow it can be observed that the issue is not with the threads.(I believe).
Normally the VM receiver threads are fast and they scale up based on the number of requests coming in.
In the "ExecuteTests" flow there seems to be an outbound HTTP call. So it is the HTTP call that might be the cause for the delay.
But still if you want to increase the threads try adding a dispatcher threading profile for a HTTP connector.
Also inorder to increase the number of threads for your flow processing use Flow threading profiles.
Like
<asynchronous-processing-strategy name="executeTestsStrategy" maxThreads="40" minThreads="20" />
<flow name="ExecuteTests" doc:name="ExecuteTests" processingStrategy="executeTestsStrategy">
<vm:inbound-endpoint exchange-pattern="one-way" path="checkService"/>
<https:outbound-endpoint exchange-pattern="request-response"...
...etc.
</flow>
Hope this helps.
Related
Oi
I've got two bpel processes. Process A puts message in a queue and Process B consumes the messages and does some work.
What I'm looking for is a way to limit the number of messages being handled at the same time. So limiting the number of Processes B running simultaneously.
adapter.jms.receive.threads - this parameter indicates the number of poller threads that are created when an adapter endpoint is activated. The default is 1. Each poller thread receives its own message that is processed independently and thus allows for increased throughput.
I think this parameter does what i'm looking for but I see no difference with it.
What i'm doing to test it is pushing a bunch of messages into the queue and immediately its created an execution instance no matter what value i have in adapter.jms.receive.threads.
Shouldn't this property limit the number of requests being handled simultaneously? Can you think of any reason for it not working? Am I missing any configuration? Any compability issue?
You did not specify which exact version you are using but because you mentioned "adapter.jms.receive.threads" I assume you are at least on Oracle BPEL 11g+.
Described behaviour occurs if you don't override the default value of bpel.config.oneWayDeliveryPolicy property (which is set to "async.persist"). Changing bpel.config.oneWayDeliveryPolicy on your component to "sync" should solve your problem.
Precisely, add the following property to your component definition inside composite.xml file:
<property name="bpel.config.oneWayDeliveryPolicy" type="xs:string" many="false">sync</property>
At some point one someQueue's size is starting to grow up. The Messages are enqueued, but they are not deqeued. Such queue is consumed with <from uri="activemq:queue:someQueue?concurrentConsumers=5"/> and it seems that parallel Processors are working fine, because data seems to be processed - further routes are triggered.
I suppose that one of the Processor which works concurently somohow stuck. How to check what is the reason? How to check if it really stuck, without adding additional watch dogs/timer threads in Processor? How to check the processor working time? Is there way to find which message caused it?
After some time when queue is to big the route stop processing data, and no other paralell Processors are run.
What I used till now is just keeping the thread code safe, and displaying status with activemq:dstat. I also think about attaching the JPDA to the Karaf to see what is happaning insisde, but maybe there are other, simplier methods to find what is the problem.
<route id="someRoute" startupOrder="7">
<from uri="activemq:queue:someQueue?concurrentConsumers=5"/>
<process ref="someProcesor"/>
<choice>
<when>
<simple>${header.ProcesedSuccesfull}</simple>
<to uri="activemq:queue:otherQueue"/>
</when>
<otherwise>
<log loggingLevel="ERROR" message="error" loggerRef="myLogger"/>
</otherwise>
</choice>
</route>
You can see the inflight repository, and as well the await manager mbean [1]. The latter can tell you how many threads are blocked waiting for some external condition to trigger before continuing.
You can use hawtio [2] web console to see that from a web browser, and then trigger a stuck thread to unblock and continue and reject/fail the messaging, but that would leave the thread to continue running in your use-case so it can pickup new messages.
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/api/management/mbean/ManagedAsyncProcessorAwaitManagerMBean.java
http://hawt.io/
in our application,consumer started polling continuously at the load of the application and therefore sometimes it impact the execution time for one of the
method by polling in between the method execution.
Method (let say test())which ideally take few millisecond to run in junit case is now taking few seconds for the execution in app.Therfore, would like to skip the polling
at this point of time,if possible.
In spring integration doc have seen something called PollSkipAdvice/PollSkipStrategy which says The PollSkipAdvice can be used to suppress (skip) a poll.
Could you please suggest,if this can be of any help in above scenario.Would be great, if explain using example.Thanks.
sample config:
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="false" channel="inputFromKafka">
<int:poller fixed-delay="10" time-unit="MILLISECONDS"
max-messages-per-poll="5" />
</int-kafka:inbound-channel-adapter>
You scenario isn't clear. Really...
We have here only one adapter with aggressive fixed-delay every 10 MILLISECONDS and only for the small amount of messages.
Consider to increase the poll time and make max-messages-per-poll as -1 to poll all of them for the one poll task.
From other side it isn't clear how that your test() method is involved...
Also consider to switch to <int-kafka:message-driven-channel-adapter> for better control over messages.
Regarding PollSkipAdvice... Really not sure which aim would you like to reach with it...
And one more point. Bear in mind all <poller>s use the same ThreadPoolTaskScheduler with the 10 as pool. So, maybe some other long-living task keeps threads from it busy...
This your <int-kafka:inbound-channel-adapter> takes only one, but each 10 millis, of course.
Maybe this is a recurrent question, but I need some customization with my context.
I'm using Spring Batch 3.0.1.RELEASE
I have a simple job with some steps. One step is a chunk like this:
<tasklet transaction-manager="myTransactionManager">
<batch:chunk reader="myReader" processor="myProcessor" writer="myWriter" commit-interval="${commit.interval}">
</batch:chunk>
<bean id="myProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor" scope="step">
<property name="delegates">
<list>
<bean class="...MyFirstProcessor">
</bean>
<bean class="...MySecondProcessor">
</bean>
</list>
</property>
Reader: JdbcCursorItemReader
Processor: CompositeProcessor with my delegates
Writer: CompositeWriter with my delegates
With this configuration, my job works perfectly.
Now, I want to convert this to a multi-threaded job.
Following the documentation to basic multi-thread jobs, I included a SympleAsyncTaskExecutor in the tasklet, but it failed.
I have readed JdbcCursorItemReader does not work properly with multi-thread execution (is it right?). I have changed the reader to a JdbcPagingItemReader, and it has been a nightmare: job does not fail, writing process are ok, but data has mixed among the threads, and customer data were not right and coherent (customers have got services, addreses, etc. from others).
So, why does it happen? How could I change to a multi-thread job?
Are the composite processor and writer right for multithread?
How could I make a custom thread-safe composite processor?
Maybe could it be the JDBC reader: Is there any thread-safe JDBC reader for multi-thread?
I'm very locked and confused with this, so any help would be very appreciated.
Thanks a lot.
[EDIT - SOLVED]
Well, the right and suitable fix to my issue is to design the job for multithread and thread-safe execution from the beggining. It's habitual to practice first with one-thread step execution, to understand and know Spring Batch concepts; but if you consider you are leaving this phase behind, considerations like immutable objects, thread-safe list, maps, etc... must raise.
And the current fix in the current state of my issue has been the next I describe later. After test Martin's suggestions and taking into account Michael's guidelines, I have finally fix my issue as good as I could. The next steps aren't good practice, but I couldn't rebuild my job from the beggining:
Change itemReader to JdbcPagingItemReader with setState to false.
Change List by CopyOnWriteArrayList.
Change HashMap by ConcurrentHashMap.
In each delegated processor, get a new instance of every bean property (fortunately, there was only one injected bean) by passing the context (implements ApplicationContextAware) and getting a unique instance of the bean (configure every injected bean as scope="prototype").
So, if the delegated bean was:
<bean class="...MyProcessor">
<property name="otherBean" ref="otherBeanID" />
Change to:
<bean class="...MyProcessor">
<property name="otherBean" value="otherBeanID" />
And, inside MyProcessor, get a single instance for otherBeanID from the context; otherBeanID must be configurated with scope="protoype".
As I tell before, they're no good style, but it was my best option, and I can assert each thread has its own and different item instance and other bean instance.
It proves that some classes has not been well designed for a right multithread execution.
Martin, Michael, thanks for your support.
I hope it helps to anyone.
You have asked a lot in your question (in the future, please break this type of question up into multiple, more specific questions). However, item by item:
Is JdbcCursorItemReader thread-safe?
As the documentation states, it is not. The reason for this is that the JdbcCursorItemReader wraps a single ResultSet which is not thread safe.
Are the composite processor and writer right for multithread?
The CompositeItemProcessor provided by Spring Batch is considered thread safe as long as the delegate ItemProcessor implementations are thread safe as well. You provide no code in relation to your implementations or their configurations so I can't verify their thread safety. However, given the symptoms you are describing, my hunch is that there is some form of thread safety issues going on within your code.
You also don't identify what ItemWriter implementations or their configurations you are using so there may be thread related issues there as well.
If you update your question with more information about your implementations and configurations, we can provide more insight.
How could I make a custom thread-safe composite processor?
There are two things to consider when implementing any ItemProcessor:
Make it thread safe: Following basic thread safety rules (read the book Java Concurrency In Practice for the bible on the topic) will allow you to scale your components by just adding a task executor.
Make it idempotent: During skip/retry processing, items may be re-processed. By making your ItemProcessor implementation idempotent, this will prevent side effects from this multiple trips through a processor.
Maybe could it be the JDBC reader: Is there any thread-safe JDBC reader for multi-thread?
As you have noted, the JdbcPaginingItemReader is thread safe and noted as such in the documentation. When using multiple threads, each chunk is executed in it's own thread. If you've configured the page size to match the commit-interval, that means each page is processed in the same thread.
Other options for scaling a single step
While you went down the path of implementing a single, multi-threaded step, there may be better options. Spring Batch provides 5 core scaling options:
Multithreaded step - As you are trying right now.
Parallel Steps - Using Spring Batch's split functionality you can execute multiple steps in parallel. Given that you're working with composite ItemProcessor and composite ItemWriters in the same step, this may be something to explore (breaking your current composite scenarios into multiple, parallel steps).
Async ItemProcessor/ItemWriters - This option allows you to execute the processor logic in a different thread. The processor spins the thread off and returns a Future to the AsyncItemWriter which will block until the Future returns to be written.
Partitioning - This is the division of the data into blocks called partitions that are processed in parallel by child steps. Each partition is processed by an actual, independent step so using step scoped components can prevent thread safety issues (each step gets it's own instance). Partition processing can be preformed either locally via threads or remotely across multiple JVMs.
Remote Chunking - This option farms the processor logic out to other JVM processes. It really should only be used if the ItemProcessor logic is the bottle neck in the flow.
You can read about all of these options in the documentation for Spring Batch here: http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html
Thread safety is a complex problem. Just adding multiple threads to code that used to work in a single threaded environment will typically uncover issues in your code.
In my nlog configuration, I've set
<targets async="true">
with the understanding that all logging now happens asynchronously to my application workflow. (and I have noticed a performance improvement, especially on the Email target).
This has me thinking about log sequence though. I understand that with async, one has no guarantee of the order in which the OS will execute the async work. So if, in my web app, multiple requests come in to the same method, each logging their occurrence to NLog, does this really mean that the sequence in which the events appear in my log target will not necessarily be the sequence in which the log method was called by the various requests?
If so, is this just a consequence of async that one has to live with? Or is there something I can do to keep have my logs reflect the correct sequence?
Unfortunately this is something you have to live with. If it is important to maintain the sequence you'll have to run it synchronously.
But if it is possible for you to manually maintain a sequence number in the log message, it could be a solution.
I know this is old and I'm just ramping up on NLog but if you see a performance increase for the email client, you may want to just assert ASYNC for the email target?
NLog will not perform reordering of LogEvent sequence, by activating <targets async="true">. It just activates an internal queue, that provides better handling of bursts and enables batch-writing.
If a single thread writes 1000 LogEvents then they will NOT become out-of-order, because of async-handling.
If having 10 threads each writing 1000 LogEvents, then their logging will mix together. But the LogEvents of an individual thread will be in the CORRECT order.
But be aware that <targets async="true"> use the overflowAction=Discard as default. See also: https://github.com/nlog/NLog/wiki/AsyncWrapper-target#async-attribute-will-discard-by-default
For more details about performance. See also: https://github.com/NLog/NLog/wiki/performance