New at MSMQ and WCF.
I want to be able to process incoming MSMQ messages at a high rate. I want to make it Multithreaded (and transactional).
What is the best way of doing this? Any examples, code snippets, theories are very much welcome.
Also, how is WCF able to know if there is a message in the MSMQ? Or would I have to create a Windows Service that polls the MSMQ, then for messages found, start it on a new thread and invoke the WCF service and pass the message to it?
What is the best way?
Many thanks
Answer here was to use WCF and create a data contract of service known types.
These known types are objects it would be expecting from the queue being read from.
To make it multi threaded and transactional, not only does the queue need to be transactional but also decorate the service attribute:
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMode.PerSession, ReleaseServiceInstanceOnTransactionComplete = false)]
The InstanceContextMode IS perSession by default.
you also need to set up the bindings on your config file
example: http://msdn.microsoft.com/en-us/library/ms751493.aspx
Related
I am now working on the application saving data into the database using the REST API. The basic flow is: REST API -> object -> save to database. I wanted to introduce the queue to the application, having in mind the idea of the producer and consumer being a part of one, abovementioned application.
Is it possible for the Node.js application to act as both producer and consumer of the queue? Knowing that Node.js is single-threaded language, does it give me any other choice instead of creating two applications - one producing to the queue and the second one - waiting actively for messages in a queue and saving to the database?
Also, the requirement here would be for an application to process any item that hasn't been acknowledged on the queue on the restart. That also makes me think that the 'two applications' architecture is the best idea here.
Thank you for the help.
Yes, nodejs is able to do that and is well suited for every I/O intensive application use case. The point here is "what are you trying to achieve"? message queues are meant to make different applications communicate together, while if you need an in-process event bus is a total overkill. There are many easier and efficient ways to propagate messages between decoupled components of the same nodejs app; one of these way is EventEmitter that let your components collaborate in a pubsub fashion
If you are convinced that an AMQP broker is you solution, you just need to
Define a "producer" class that publishes data on an exchange myExchange
Define a "consumer" queue that declares a queue myQueue
Create a binding at application startup between myExchange and myQueue, based on some routing key. Then, when a message is received from "consumer" you need to acknowledge after db saving. When a message is acked, it will be destroyed since it's already been consumed. You can decide, after an error, to recover the message via NACK
There are nodejs libraries that make code easier, such as Rascal
Short answer: YES and use two separate connections for publishing and consuming
Is it possible for the NodeJS application to act as both producer and consumer of the queue?
I would even state that it is a good usecase matching extremely well with NodeJS philosophy and threading mechanism.
Knowing that Node.js is single-threaded language, does it give me any other choice instead of creating two applications - one producing to the queue and the second one - waiting actively for messages in a queue and saving to the database?
You can have one application handling both, just be aware that if your client is publish too fast for the server to handle, RabbitMQ can apply back pressure on the TCP connection, thus consuming on a back-pressured TCP connection would greatly affect consumer performance.
I recently created an error manager to take logged errors from clients on our network and put them into an MSMQ for processing. I have a separate Windows Service running on the server to pick items off the queue and push them into a database.
When I wrote it and tested it everything worked great; however I neglected to consider that at deploy-time, having 100 clients all sending to a public queue might not be performant, best-case, and worst-case there could be all kinds of collisions, it seems to me.
My thought right now is to front the MSMQ with a WCF service and make everyone go through that. The logic being that at that point I could employ some locking, etc. If I went with a service I think I could employ a private queue instead of a public one, which would be tons faster, as well.
What I'm not sure is, am I overthinking it? MSMQ is pretty robust and the methods I think are thread-safe. Should I just leave it alone and see what happens? If I do put in the service, how much management would I need to have in place?
I recently created an error manager to take logged errors from clients
on our network and put them into an MSMQ for processing
I assume you're using System.Messaging for this? If so there is nothing at all wrong with your approach.
having 100 clients all sending to a public queue might not be
performant
MSMQ was designed from the bottom up to handle high load. Depending on the size of the individual messages and the storage threshold of the machine, a queue can hold 10's of thousand of messages without any noticeable performance impact.
Because a "send" in MSMQ involves the queue manager on each machine writing messages locally before transmission (in a store and forward messaging pattern), there is almost no chance of "collisions" or any other forms of contention happening; if the sender is unable to transmit the message it simply "sends" it to a temporary local queue and then the actual transmission happens in the background and is mediated by the fault tolerant and very reliable msmq protocol.
My thought right now is to front the MSMQ with a WCF service and make
everyone go through that
This would be a valid choice if you were starting from nothing. As another poster has stated, WCF does hide you from some of the msmq-voodoo by removing the necessity to use System.Messaging. However, you've already written the code so I see little benefit exposing a netMsmqBinding endpoint.
If I went with a service I think I could employ a private queue
instead of a public one
As far as I understand it from your description, there's nothing to stop you using a private queue in your current scenario. In fact I'd recommend always using private queues as they're much simpler.
If I do put in the service, how much management would I need to have
in place?
You will have more management overhead with a wcf service. Because you're wrapping each end of a send-receive with the WCF stack, there is more code to spin up and therefore potentially fail. WCF stack exceptions are famously difficult to troubleshoot without full service logging enabled.
EDIT - in response to comments
I think for a private queue you have to actually be writing FROM the
machine the queue sits on, which would not work in a networked
environment
Untrue. MSMQ supports transactional reads to and writes from any private queue, regardless of whether the queue is local or remote.
This is because any time a message is sent from one machine to another in msmq, regardless of the queue address, the following happens:
Queue manager on sending machine writes the message to a temporary local "outbound" queue.
Queue manager on sending machine contacts queue manager on receiving machine and transmits the message.
Queue manager on receiving machine places the message into the destination queue.
If you are using transactions, the above steps will comprise 3 distinct transactions.
Something to remember: the safest paradigm in exchanging messages between queues on different machines is send remote, read local.
So this means when you send a message, you're instructing msmq to send to a remote queue address. However, when someone sends something to you, they must do the same. So you end up reading only from local queues, and sending only to remote queues.
This way you get the most reliable messaging setup, because when reading, a local queue will always be available.
Try it! I've been using msmq for cross machine communication for nearly 10 years and I've never used a public queue. I don't even know what they're for!
I would expose an WCF "IsOneWay" method.
And then host your WCF in IIS.
The IsOneWay will wire up to MSMQ.
This way...you have the robustness of IIS hosting. You can expose any endpoint you want.
But eventually the request makes it to MSMQ.
One of hte reasons is the ease of using msmq with wcf. Having written and used msmq "pre-wcf" I found the code (pulling messages off the queue and error handling) to be difficult and problematic. That alone would push me to WCF hosting.
And as you mention, the security around a local-queue is much easier to deal with.
Bottom line, let WCF handle the msmq-voodoo for you.
Simple example below.
[ServiceContract]
public interface IMyControllerController
{
[OperationContract(IsOneWay = true)]
void SubmitRequest( MyObject obj );
}
http://msdn.microsoft.com/en-us/library/ms733035%28v=vs.110%29.aspx
http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.isoneway%28v=vs.110%29.aspx
What happens in WCF to methods with IsOneWay=true at application termination
http://blogs.msdn.com/b/tomholl/archive/2008/07/12/msmq-wcf-and-iis-getting-them-to-play-nice-part-1.aspx
I have a web application that interfaces with another application through a message queue. So, my web application has a service-actibator that is bound to an inbound message driven channel adapter; currently it is is always listening for messages on the queue.
However, there may be times where it is desiarable to turn that listening off without bouncing the application itself. For example, if the queue gets a backlog of messages and for whatever reason the web application that is listening for these messages begins to have performance issues and we want to isolate the application from the queue to help identify if that is the source of the performance problem or not.
The bottom line is we are trying to proactivey look for ways to help our support staff when needing to diagnose potential inter-system issues...without having to necessarily bounce the servers for a configuration change.
Then if it is determined that the interface to the external system should be turned back on then we would want to be able to re-start the service activator.
Is anything like this possible? Or is there an approach that I'm not thinking of that would allow this type of runtime start/stop capability?
Yes, it is possible.
All Endpoints in the Spring Integration implement org.springframework.context.SmartLifecycle.
From other side SI has a component for this purpose - Control Bus
So, it very simple:
<channel id="controlBusChannel"/>
<control-bus input-channel="controlBusChannel"/>
<service-activator input-channel="stopMyServiceActivatorChannel"
output-channel="controlBusChannel" expression="'#myServiceActivator.stop()'"/>
<service-activator id="myServiceActivator" input-channel="myInputChannel"
output-channel="myOutupChannel"/>
I was using AzureQueue to communicate between roles. My messages like "GoToMaintenanceMode", "StopSendingEmails", "DoNotAcceptRequests" etc. But I realized that, it won't work for my scenario when I have multiple instances due to queue message will shows up only 1 instance at a time.
So my question is beside the options below is there an elegant way to handle this issue something like Role.AllInstances.Run() etc.?
the method I'm using it right now:
instance peeks the message, adds it's own instance id to the message and puts it back to the queue, and does not peek the message if it contains it's own instance id.
P.S. I do not want to implement TCP listener, asking for native solution if there is one.
You could use Windows Azure Service Bus Topics/Subscriptions instead of queues. They support multicasting (i.e. multiple receivers).
A short how-to guide can be found here
http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/
Basically, your queue would become a Topic all your instances would become Subscriber to the Topic.
As you do not want TCP listener option or Service bus option. How about extending your same idea with multiple queues. Instance1 will read from Queue1 and Instance2 will read from Queue2 and so on. The only thing you need to handle is the number of queues and simultaneous adding of queue messages to all the queues.
We are thinking of speparate Queues for:
Request (RequestQueue)
Response (ResponseQueue)
Scenario:
Worker role will putMessage to RequestQueue e.g. GetOrders
Third party will monitor RequestQueue. If they see GetOrders
request they will getMessage, process them and put the response in
ResponseQueue.
Question:
If I putMessage to RequestQueue, I will like to get results back from ResponseQueue. Is there easy way to achieve this and how?
Thank you.
No, this is not possible. If you put a message in a queue, you must pop the message from the same queue (it will not magically appear in any other queue). Perhaps if you explained more why you think you need two separate queues here for push/pop, there might be a more expansive answer and suggestion.
EDIT: Perhaps I misunderstood your intent. I guess I don't get the question now - can you help clarify. You seem to be asking how to put a message on one queue, acknowledge it by putting another message on another queue, and have someone read the acknowledgment from the second queue? What is the question here? I should point out that you won't want some 3rd party to read directly from a Windows Azure queue as that would require sharing the master storage key with them (a non-starter). Perhaps you are looking for how to have 3rd parties read from a queue?
EDIT 2: Sounds like you want to consume messages with a 3rd Party. Windows Azure queues probably are not a good fit as I mentioned due to security reasons (you need to share the master key). Instead, you could either layer a WCF service over the queue (using queues via proxy) or use the queueing from the Service Bus - that will allow you to have separate credentials. Using the Service Bus capability might be the right choice here in terms of simplicity. Take a look here for demos.
Have a worker of some sort monitor the question queue, then post an answer to the answer queue. Interface out the queue managers and you shouldn't have any problems using any sort of queue tech. Also, the worker doesn't really need to use a queue for answers..
Caveats:
Worker service has access to both queues
Each queue item contains a serialized foreign key to identify themselves.