We need to implement a Async web service.
Behaviour of web service:
We send the request for an account to server and it sends back the sync response with an acknowledgement ID. After that we get multiple Callback requests which contains that acknowldegment ID. The last callback request for an acknowledgement ID will contain a text(completed:true) in the response which will tell us that this is the last callback request for that account and acknowledgement ID. This will help us to know that async call for a particular account is completed and we can mark its final status. We need to execute this web service for multiple accounts. So, we will be getting callback requests for many accounts.
Question:
What is the optimal way to process these multiple callback requests coming for multiple accounts.
Solutions that we thought of:
ExecutorService Fixed Thread Pool: This will parallely process our callback requests but the concern is that it does not maintain the sequence. So it will be difficult for us to determine that the last callback request for an acknowledgment ID(account) has come. Hence, we will not be able to mark the final status of that account as completed with surity.
ExecutorService Single Thread Executor: Here, only one thread is there in the pool with an unbouded queue. If we use this then processing will be pretty slow as only one thread will be actually processing.
Please suggest an optimal way to implement requirement both memory and performance wise.
Let's be clear about one thing: HTTP is a blocking, synchronous protocol. Request/response pairs aren't asynch. What you're doing is spawning asynch requests and returning to the caller to let them know the request is being processed (HTTP 200) or not (HTTP 500).
I'm not sure that I know optimal for this situation, but there are other considerations:
Use an ExecutorServiceThreadPool that you can configure. Make sure you have a prefix that lets you distinguish these threads from others.
Add request task to a blocking dequeue and have a pool of consumer threads process them. You can tune the dequeue and the consumer thread pool sizes.
If processing is intensive, send request messages to a queue running on another server. Have a pool of queue listeners process the requests.
You cannot assume that the callbacks will return in a certain order. Don't depend on "last" being "true". You'll have to join all those threads together to know when they're finished.
It sounds like the web service should have a URL that lets users query for status.
Related
We have our HTTP layer served by Play Framework in Scala. One of our APIs is something of the form:
POST /customer/:id
Requests are sent by our UI team which calls these APIs through a React Framework.
The issue is that, sometimes, the requests are issued in batches, successively one after the other for the same customer ID. When this happens, different threads process these requests and so our persistent layer (MySQL) reaches an inconsistent state due to the difference in the timestamp of the handling of these requests.
Is it possible to configure some sort of thread affinity in Play Scala? What I mean by that is, can I configure Play to ensure that requests of a particular customer ID are handled by the same thread throughout the life-cycle of the application?
Batch is
put several API calls into a single HTTP request.
A batch request is a set of command in one HTTP request, like here https://developers.facebook.com/docs/graph-api/making-multiple-requests/
You describe it as
The issue is that, sometimes, the requests are issued in batches, successively one after the other for the same customer ID. When this happens, different threads process these requests and so our persistent layer (MySQL) reaches an inconsistent state due to the difference in the timestamp of the handling of these requests.
This is a set of concurrent requests. Play framework usually works as a stateless server. I assume you also organize it as stateless. There is nothing that binds one request to another, you can't control order. Well, you can, if you create a special protocol, like "opening batch request", request #1, #2, ... "closing batch request". You need to check if exactly all request was correct. You also need to run some stateful threads and some queues ... Thought akka can help with this but I am pretty sure you wan't do it.
This issue is not a "play-framework" depended. You will reproduce it in any server. For example, the general case: Is it possible to receive out-of-order responses with HTTP?
You can go in either way:
1. "Batch" the command in one request
You need to change the client so it jams "batch" requests into one. You also need to change server so it processes all the commands from the batch one after another.
Example of the requests: https://developers.facebook.com/docs/graph-api/making-multiple-requests/
2. "Pipeline" requests
You need to change the client so it sends the next request after receive the response from the previous.
Example: Is it possible to receive out-of-order responses with HTTP?
The solution to this is to pipeline Ajax requests, transmitting them serially. ... . The next request sent only after the previous one has returned successfully."
I have a web service that accepts post requests. A post request specifies a specific job to be executed in the background, that modifies a database used for later analysis. The sender of the request does not care about the result, and only needs to receive a 202 acknowledgment from the web service.
How it was implemented so far:
Flask Web service will get the http request , and add the necessary parameters to the task queue (rq workers), and return back an acknowledgement. A separate rq worker process listens on the queue and processes the job.
We have now switched to aiohttp, and realized that the web service can now schedule the actual job request in its own event loop, by using the aiohttp.ensure_future() method.
This however blurs the lines between the web-server and the task queue. On the positive side, it eliminates the need of having to manage the rq workers.
Is this considered a good practice?
If your tasks are not CPU heavy - yes, it is good practice.
But if so, then you need to move them to separate service or use run_in_executor(). In other case your aiohttp event loop will be blocked by this tasks and server will not be able to accept new requests.
Situation: A high-scale Azure IIS7 application, which must do this:
Receive request
Place request payload onto a queue for decoupled asynchronous processing
Maintain connection to client
Wait for a notification that the asynchronous process has completed
Respond to client
Note that these will be long-running processes (30 seconds to 5 minutes).
If we employ Monitor.Wait(...) here, waiting for a callback to the same web application, from the asynchronous process, to invoke Monitor.Pulse(...) on the object we invoked Monitor.Wait() on, will this effectively create thread starvation in a hurry?
If so, how can this be mitigated? Is there a better pattern to employ here for awaiting the callback? For example, could we place the Response object into a thread-safe dictionary, and then somehow yield, and let the callback code lock the Response and proceed to respond to the client? If so, how?
Also, what if the asynchronous process dies, and never invokes the callback, thus never causing Monitor.Pulse() to fire? Is our thread hung now?
Given the requirement you have, I would suggest to have a look at AsyncPage/AsyncController (depends on whether you use ASP.NET WebForms or ASP.NET MVC). These give you the possibility to execute long running tasks in IIS without blocking I/O threads.
How can the disruptor be used effectively on processes where there are variable-duration "business logic" tasks? Has it been done before?
Can it be done with a second ring buffer processing the response stage? If so, how do I go about that.
I understand the Disruptor and see some specific parts of my call chain where I could apply the concept. To be specific, the application is a middleware-type application which performs following steps:
read inbound message, unmarshall to request
identify customer details for request, determine workflow steps to process
call backend system to execute steps
collate responses, log response, marshall, and return to consumer
The issue is that some instances of the backend steps can take a "long" time, which potentially forces the response stage for short running tasks to wait for longer running tasks. Assume call of backend system can be done either async or sync - so the idea would be that the backend system call is simply a consumer that triggers an async request to backend.
Back end system response time can be anywhere from 5ms (some requests), 50ms (90% of requests) to - 5000ms (1% of requests) (think large disk I/O).
I can see the Disruptor as potentially highly efficient but can't get my head around this hurdle to keep average latency down.
When a Web Role places a message onto a Storage Queue, how can it poll for a specific, correlated response? I would like the back-end Worker Role to place a message onto a response queue, with the intent being that the caller would pick the response up and go from there.
Our intent is to leverage the Queue in order to offload some heavy processing onto the back-end Worker Roles in order to ensure high performance on the Web Roles. However, we do not wish to respond to the HTTP requests until the back-end Workers are finished and have responded.
I am actually in the middle of making a similar decision. In my case i have a WCF service running in a web role which should off-load calculations to worker-roles. When the result has been computed, the web role will return the answer to the client.
My basic data structure knowledge tells me that i should avoid using something that is designed as a queue in a non-queue way. That means a queue should always be serviced in a FIFO like manner. So basically if using queues for both requests and response, the threads awaiting to return data to the client will have to wait untill the calculation message is at the "top" of the response queue, which is not optimal. If storing the responses by using Azure tables, the threads poll for messages creating unnecessary overhead
What i belive is a possible solution to this problem is using a queue for the requests. This enables use of the competeing consumers pattern and thereby load-balancing. On messages sent into this queue you set the correlationId property on the message. For reply the pub/sub part ("topics") part of Azure service bus is used togehter with a correlation filter. When your back-end has processed the request, it published a result to a "responseSubject" with the correlationId given in the original request. Now this response ca be retrieved by your client by calling CreateSubscribtion (Sorry, i can't post more than two links apparently, google it) using that correlation filter, and it should get notified when the answer is published. Notice that the CreateSubscribtion part should just be done one time in the OnStart method. Then you can do an async BeginRecieve on that subscribtion and the role will be notified in the given callback when a response for one of it's request is available. The correlationId will tell you which request the response is for. So your last challenge is giving this response back to the thread holding the client connection.
This could be achieved by creating Dictionary with the correlationId's (probably GUID's) as key and responses as value. When your web role gets a request it creates the guid, set it as correlationId, add it the hashset, fire the message to the queue and then call Monitor.Wait() on the Guid object. Then have the recieve method invoked by the topic subscribition add the response to the dictionary and then call Monitor.Notify() on that same guid object. This awakens your original request-thread and you can now return the answer to your client (Or something. Basically you just want your thread to sleep and not consume any ressources while waiting)
The queues on the Azure Service Bus have a lot more capabilities and paradigms including pub / sub capabilities which can address issues dealing with queue servicing across multiple instance.
One approach with pub / sub, is to have one queue for requests and one for the responses. Each requesting instance would also subscribe to the response queue with a filter on the header such that it would only receive the responses targeted for it. The request message would, of course contain the value to the placed in the response header to drive the filter.
For the Service Bus based solution there are samples available for implementing Request/Response pattern with Queues and Topics (pub-sub)
Let worker role keep polling and processing the message. As soon as the message is processed add an entry in Table storage with the required corelationId(RowKey) and the processing result, before deleting the processed message from the queue.
Then WebRoles just need to do a look up of the Table with the desired correlationId(RowKey) & PartitionKey
Have a look at using SignalR between the worker role and the browser client. So your web role puts a message on the queue and returns a result to the browser (something simple like 'waiting...') and hook it up to the worker role with SignalR. That way your web role carries on doing other stuff and doesn't have to wait for a result from asynchronous processing, only the browser needs to.
There is nothing intrinsic to Windows Azure queues that does what you are asking. However, you could build this yourself fairly easily. Include a message ID (GUID) in your push to the queue and when processing is complete, have the worker push a new message with that message ID into a response channel queue. Your web app can poll this queue to determine when processing is completed for a given command.
We have done something similar and are looking to use something like SignalR to help reply back to the client when commands are completed.