I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};
Related
I need to have a listener app in .NET Core which will continuously look into new messages in Azure Service Bus Queue. I've tried so far with different approaches, but only able to read messages from ASB queue which are already published to queue before my listener application has started consuming. But once all the existing messages are processed and then the listener is not able to read any new messages from ASB queue.
I'm trying to figure out whether there is any method in the namespace "Azure.Messaging.ServiceBus" which will continuously pull messages from ASB queue.
I do not want to create a WHILE loop or a long running Task.
Any help is appreciated.
There are 4 ways to create an "Azure Service Bus Queue". This could be the long procedure to understand. To make it precise, I am attaching the links to follow for different methods of creation.
ARM Template : https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-resource-manager-namespace-queue
Powershell: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quickstart-powershell
Azure CLI: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quickstart-cli
Azure Portal: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quickstart-portal
The Suitable answer for the requirement:
The receive modes are two types.
Receive and Delete
Peek lock
We can receive the message and pull it to read using Azure.Messaging.ServiceBus as required. Using the namespaces and utilizing the services of Azure.Messaging.ServiceBus
I've an API (python-flask app) running on an app service in azure and want to implement a queuing system using Azure Service Bus such that requests from API are sent to a simple FIFO queue managed/ran by the service bus. Another resource in Azure will be pulling from this queue and running the jobs based on the contents of the json/payload contained in the message in the queue element.
When this element has been processed by the other resource I want to encode the job status/metadata (e.g., "finished" along with metadata such as the location where resulting data was stored). I read about such a system that makes use of the lightweight database offered by Redis, however, I'm wondering if something like this lightweight database/cache system of job status/ids/metadata is available through Azure Service Bus? I'm aware that Redis can be run standalone on a VM in Azure, however, if this can all be managed via the service bus that would be ideal. I couldn't find specifics on this being offered within Azure Service Bus and due to how this job metadata is later being accessed I cannot just push metadata messages to a new queue.
Does anyone have any insight on this or potential alternatives? If Redis can be run alongside flask within the same App Service then that would be ideal, but again I wasn't able to find anything explicit on this and it doesn't seem possible to simultaneously run a flask server/app and Redis server at the same time on an App Service.
Thanks.
I'm wondering if something like this lightweight database/cache system
of job status/ids/metadata is available through Azure Service Bus?
Azure Service Bus is a fully managed enterprise message broker, Azure Redis is a NoSQL database with steroids. It also offers queue mechanism and some other data structures.
it doesn't seem possible to simultaneously run a flask server/app and
Redis server at the same time on an App Service.
You can, but inside containers.
Please check if this can help you: https://stackoverflow.com/a/39008342/1384539
I have multiple WebApi that are hosted in Azure but some of the API taking a long time to process.
Instead of a calling from the scheduler function in Azure, I was advised to use the Messaging Queue.
Is this a good approach?
Also, let say my API URL.. https://testwebapi.net/api1 and https://testwebapi.net/api2, how can we communicate these API using Message queue and also how to call the WebAPI individually from Messaging Queue?
Is this Azure Bus service is the same as Messaging Queue for Azure.
I would strongly recommend you look at the respective docs on Azure, and start doing some quick tutorials and post more specific questions.
Now, to answer, what I believe is your main question, Azure Bus Service and Messaging Queue.
The thing is, there is no one Messaging Queue, rather three options for Azure messaging - Service Bus, Event Grid, and Event Hubs. Also, there is something called 'Azure Queue Storage', but I dont think you are thinking of that, considering your context.
There is a very cool comparison chart available here - https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
But, I think, my guess, is you are looking for ultimately looking to use Azure Service Bus.
Also, please understand that one does not invalidate the other. As your project develops, you may end up using Azure Service Bus with other options like Event Hubs or Grids or all of them in different places.
Also, you keep using the incorrect terms. for example, you say 'Azure Bus service' but its really 'Azure Service Bus'. Also, there is no thing called 'Messaging Queue for Azure.' as there are so many different messaging services for Azure. This is what I want to add here. Spent some time with the docs and post more specific questions as you continue your project.
We are currently using Azure more like IAAS, rather than cloud services.
As a start, I would like to utilise Azure Messaging Queues to process some database actions and Web API calls.
I am assuming I would need to write another piece of code that subscribes to the queues, so when messages arrive, it knows to process the transaction?
Is that piece of code, a console app? runs on a scheduled task? a windows service? or a function app within azure?
What is the Best Practice for this architecture?
You can write console app and schedule web job to monitor the queue. However better way is to use Azure Functions. You don't have to monitor the queue then. Whenever the message arrives in the queue it will trigger Azure Function and you can process the message. The benefit is it's server-less.
I am trying to read messages from a service bus queue using logic app. I do not find any documentation on how the logic app will scale based on the number of messages in the queue. Any help or pointer very much appreciated.
Right now, the answer is...it just does. Meaning, unless you are using a singleton LogicApp, Azure will spin up as many LogicApp instances as needed.
In other words, yes, it scales based on demand using any Trigger, not just Service Bus.
Consequently, there is no built in way to limit the number of instances.