I have two inbound-channel-adapter which collect files from two distinct sources.
I'd like to process the incoming files one at a time, by the same instance of service-activator and in the same thread. At the moment, since there are two distinct Poller, they are actually processed by two different threads concurrently.
I thought that using a queueChannel to feed my service-activator would have solved the problem but I don't want to introduce another Poller (and hence, another delay).
Any idea?
Use an ExecutorChannel with an Executors.newSingleThreadExecutor().
You can also use a QueueChannel with a fixedDelay of 0; the poller blocks in the queue for 1 second by default (and can be increased - receiveTimeout) so with a 0 delay between polls, no additional latency will be added.
Related
Oi
I've got two bpel processes. Process A puts message in a queue and Process B consumes the messages and does some work.
What I'm looking for is a way to limit the number of messages being handled at the same time. So limiting the number of Processes B running simultaneously.
adapter.jms.receive.threads - this parameter indicates the number of poller threads that are created when an adapter endpoint is activated. The default is 1. Each poller thread receives its own message that is processed independently and thus allows for increased throughput.
I think this parameter does what i'm looking for but I see no difference with it.
What i'm doing to test it is pushing a bunch of messages into the queue and immediately its created an execution instance no matter what value i have in adapter.jms.receive.threads.
Shouldn't this property limit the number of requests being handled simultaneously? Can you think of any reason for it not working? Am I missing any configuration? Any compability issue?
You did not specify which exact version you are using but because you mentioned "adapter.jms.receive.threads" I assume you are at least on Oracle BPEL 11g+.
Described behaviour occurs if you don't override the default value of bpel.config.oneWayDeliveryPolicy property (which is set to "async.persist"). Changing bpel.config.oneWayDeliveryPolicy on your component to "sync" should solve your problem.
Precisely, add the following property to your component definition inside composite.xml file:
<property name="bpel.config.oneWayDeliveryPolicy" type="xs:string" many="false">sync</property>
I have business requirement where I have to process messages in a certain priority say priority1 and priority2
We have decided to use 2 JMS queues where priority1 messages will be sent to priority1Queue and priority2 messages will be sent to priority2Queue.
Response time for priority1Queue messages is that the moment message is in Queue, I need to read, process and send the response back to say another queue in 1 second. This means I should immediately process these messages the moment they are in priority1Queue, and I will have hundreds of such messages coming in per second on priority1Queue so I will definitely need to have multiple concurrent consumers consuming messages on this queue so that they can be processed immediately when they are in the queue(consumed and processed within 1 second).
Response time for priority2Queue messages is that I need to read, process and send the response back to say another queue in 1 minute. So the response time of priority2 is lower to priority1 messages however I still need to respond back in a minute.
Can you suggest best possible approach for this so that I can concurrently read messages from both the queue and give higher priority to priority1 messages so that each priority1 message can be read and processed in 1 second.
Mainly how it can be read and fed to a processor so that the next message can be read and so on.
I need to write a java based component that does the reading and processing.
I also need to ensure this component is highly available and doesn't result in OutOfMemory, I will be having this component running across multiple JVMS and multiple application servers thus I can have multiple clusters running this Java component
First off, the requirement to process within 1 second is not going to be dependent on your messaging approach, but more about the actual processing of the message and the raw CPUs available. Picking up 100s of messages per second from a queue is child's play, the JMS provider is most likely not the issue. Depending on your deployment platform (Tomcat, Mule, JEE, whatever), there should be a way to have n listeners to scale up appropriately. Because the messages exist on the queue until you pick it up, doubtful you'll run out of memory. I've done these apps, processed many more messages without problems.
Second, number of strategies for prioritizing messages, not necessarily requiring different queues, using priorities. I'm leaning towards using message priorities and message filters, where one group of listeners take care of the highest priority messages and another listener filters off lower priority but makes sure it does enough to get them out within a minute.
You could also do something where a lower priority message gets rewritten back to the same queue with a higher priority, based on how close to 1 minute you are. I know that sounds wrong, but reading/writing from JMS has very little overhead (at least compared to do the equivalent, column-driven database transactions), but the listener for lower priority messages could just continually increase the priority until it has to be processed.
Or simpler, just have more listeners on the high priority queue/messages than the lower priority ones, and imbalance in number of processes for messages might be all it needs.
Lots of possibilities, time for a PoC.
I am using spring integration to process some directories for files and each file goes through a "flow". I would like to set the overall processing of files in such a way that a file poller monitors a directory (or multiple) for new files. Once a new file has been picked up poller it should be passed to the subsequent components in the flow where this new file is processed while the polling process is not held. The second aspect of processing is that all new files go through a few steps before they are aggregated by an aggregator based on e.g. number of files (the criteria changes across directories). Once enough files have been accumulated then they are released from aggregated and then processed in some time consuming steps after aggregator. So the overall process looks like this
file-A picked up
file-A passed from poller to step1
file-A passed from step1 to aggregator
file-B picked up
file-B passed from poller to step1
file-B passed from step1 to aggregator
file-C picked up
file-C passed from poller to step1
file-C passed from step1 to aggregator
files A,B and C are released from aggregator
files A,B and C are processed by final-step
so overall there are two requirements
process each file in a separate thread to maximize the number of currently being processed files
files released from aggregator belong to a correlation id, we want only one group of messages using the same correlation id to be processed by final-step
How I attempted to satisfy these two requirements is for #1 i simply used a queue after file poller where the new files are dropped and step-a picks up files from queue. This detaches the polling process and the idea was to use a thread-executor in step-a service activator to process each file in a single thread
Second requirement was automatically handled by simply executing final-step after aggregator in the same thread as aggregator. Since the aggregator places a lock based on correlation id if another group is released for same correlation id it just simply waits before the previous instance of same group is being processed.
The problem I ran into is that #1 wasnt being fulfilled because the service activator was waiting until the end of thread completion before attempting to create another thread for second file. This is kind of not helpful because this way having a thread executor on service activator is not useful. It only seems to create second therad after completing the first thread. So to fix this I replaced queued channel with a dispatcher channel and palced the executor on the dispatcher channel. Now each file was being processed in a separate thread and multiple files were being processed at the same time.
Now for second part, since the components after aggregator are time consuming I wanted to disconnect that process from first part so I placed a queued channel after aggregator but now with this approach the locking behavior that I was previously getting with aggregator is gone because the thread that released the messages from aggregator dies/completes in the queued channel before final time consuming step.
Any thoughts on the overall process. How can I accomplish both of my requirements while running things in parallel.
Thanks
It's not at all clear what your question is. Yes, for the downstream flow to run under the lock, it must run on the final "releasing" thread (the thread processing the last inbound message that completes the group), you can't use a queue or executor channel downstream of the aggregator.
However, this has no impact on the threads from the executor channel; other groups (with different correlation) will process. However, if you are using the same correlation id for the "next" group, its threads will block.
If you are saying you want to assemble the next group (with the same correlationid) while the first one is processing downstream, you'll have to use some other mechanism to enforce single threading downstream - such as an executor channel with a single-thread executor, or use another lock registry.
I am using a producer / consumer pattern backed with a BlockingCollection to read data off a file, parse/convert and then insert into a database. The code I have is very similar to what can be found here: http://dhruba.name/2012/10/09/concurrent-producer-consumer-pattern-using-csharp-4-0-blockingcollection-tasks/
However, the main difference is that my consumer threads not only parse the data but also insert into a database. This bit is slow, and I think is causing the threads to block.
In the example, there are two consumer threads. I am wondering if there is a way to have the number of threads increase in a somewhat intelligent way? I had thought a threadpool would do this, but can't seem to grasp how that would be done.
Alternatively, how would you go about choosing the number of consumer threads? 2 does not seem correct for me, but I'm not sure what the best # would be. Thoughts on the best way to choose # of consumer threads?
The best way to choose the number of consumer threads is math: figure out how many packets per minute are coming in from the producers, divide that by how many packets per minute a single consumer can handle, and you have a pretty good idea of how many consumers you need.
I solved the blocking output problem (consumers blocking when trying to update the database) by adding another BlockingCollection that the consumers put their completed packets in. A separate thread reads that queue and updates the database. So it looks something like:
input thread(s) => input queue => consumer(s) => output queue => output thread
This has the added benefit of divorcing the consumers from the output, meaning that you can optimize the output or completely change the output method without affecting the consumer. That might allow you, for example, to batch the database updates so that rather than making one database call per record, you could update a dozen or a hundred (or more) records with a single call.
I show a very simple example of this (using a single consumer) in my article Simple Multithreading, Part 2. That works with a text file filter, but the concepts are the same.
I have single ActorSystem, which has several subscribers to it's eventStream. Application may produce thousands of messages per second, and some of the messages are more important than the rest of. So they should be handled before all.
I found that every ActorSystem has single eventStream attached, thus it seems that I need to register same actor class with two (or more) ActorSystems, in order to receive important messages in dedicated eventStream.
Is this preferred approach, or there are some tricks for this task? May be classifiers can also tweak message priorities somehow?
EventStream is not a datastructure that holds events, it just routes events to subscribers, hence you should use PriorityMailbox for the listener actors, see the documentation for how to use priority mailboxes: http://doc.akka.io/docs/akka/2.0.3/scala/dispatchers.html#Mailboxes