I have a local machine that reads RabbitMQ queue messages.
I wish to move it to cloud. Which Azure service can be used in this case?
I went through event hubs, but I am not sure, if it would read messages from rabbitMQ continuously.
Any suggestions for the service that should be put to use.
You should take a look at Azure Service Bus. It has got FIFO queues as well as publish/subscribe capabilities. However if using Azure managed service is not a strict requirement you can use RabbitMQ on a VM (or a cluster for high availability) as well.
UPDATE: Your response means you want a managed service. There are 2 options - if you want to go with RabbitMQ but do not want to manage the infrastructure you can go for 3rd party service provider like CloudAMQP who will manage it on your behalf. The other option is to go for Cloud native messaging - meaning if you are on Azure you change your messaging service to Azure Service Bus. This would mean changing you code as well.
Related
I've an API (python-flask app) running on an app service in azure and want to implement a queuing system using Azure Service Bus such that requests from API are sent to a simple FIFO queue managed/ran by the service bus. Another resource in Azure will be pulling from this queue and running the jobs based on the contents of the json/payload contained in the message in the queue element.
When this element has been processed by the other resource I want to encode the job status/metadata (e.g., "finished" along with metadata such as the location where resulting data was stored). I read about such a system that makes use of the lightweight database offered by Redis, however, I'm wondering if something like this lightweight database/cache system of job status/ids/metadata is available through Azure Service Bus? I'm aware that Redis can be run standalone on a VM in Azure, however, if this can all be managed via the service bus that would be ideal. I couldn't find specifics on this being offered within Azure Service Bus and due to how this job metadata is later being accessed I cannot just push metadata messages to a new queue.
Does anyone have any insight on this or potential alternatives? If Redis can be run alongside flask within the same App Service then that would be ideal, but again I wasn't able to find anything explicit on this and it doesn't seem possible to simultaneously run a flask server/app and Redis server at the same time on an App Service.
Thanks.
I'm wondering if something like this lightweight database/cache system
of job status/ids/metadata is available through Azure Service Bus?
Azure Service Bus is a fully managed enterprise message broker, Azure Redis is a NoSQL database with steroids. It also offers queue mechanism and some other data structures.
it doesn't seem possible to simultaneously run a flask server/app and
Redis server at the same time on an App Service.
You can, but inside containers.
Please check if this can help you: https://stackoverflow.com/a/39008342/1384539
I would like to setup an azure web service and an azure vm that is running msmq. Can I send messages to the vm's msmq? I noticed in the documentation for msmq on azure, that sending messages with http to msmq on azure is not supported, but I thought there might be some other means of sending messages to msmq from an azure web app.
Has anyone actually set this up? If this is not possible do we need to use service bus instead?
I would suggest you to use Azure Storage Queue or Azure Service Bus Queue instead of MSMQ. But still if you want to use MSMQ on Azure Virtual Machine, you have the official documentation here.
Also, the difference between storage queue and service bus queue is here, so you can check out which one suits you the best.
I have implemented a Proof of Concept in Azure Service fabric that uses Azure Service Bus as a message queue. I'm using nServiceBus within the application to send and respond to messages which makes it very easy to change from one queuing technology to another.
I'd like to know if it's possible to use MSMQ instead of Service Bus in Azure Service Fabric as the nodes that are created are just windows 2016 servers and I'm not sure I need something like Service Bus. It's a question I want to answer with my POC.
Does anyone know whether MSMQ is included in an Azure Service Fabric node or how I could turn it on and if it's a viable solution?
Short answer - MSMQ is not suitable for Azure Service Fabric.
MSMQ is store and forward technology. It's using local file system to persist messages and then forward to another machine. When Service Fabric is going to move service from one node to another, it will not move the file system along. Meaning you'll lose messages.
I would recommend to stay with Azure Service Bus unless there's a good reason you're looking for an alternative.
I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};
We are looking to use Windows Azure to host our existing SaaS platform and extend our functionality and capability. WE will be taking adavantage of both the data storage and application and web service functionality of Azure.
My question is as follows:
Some of our clients will not want Public CLoud access. Since our datastore stores sensitive client data many of them will require our whole system to be hosted internally on their own network and servers.
If we setup a full Azure setup of database and connected applications and processes how difficult is it to be able to duplicate that system for a specific client on their own servers and network using existing Microsoft technologies?
I know its a vague question and I also have a liminted understanding of Azure so whatever information you can provide here would be most appreciated.
Thank you
It sounds like you need the flexibility of a hybrid cloud/on-prem solution. Likely the best solution is the Windows Azure Service Bus. Essentially, you configure a WCF web service in the cloud (SOAP, REST, etc) that performs asynchronous brokered messaging between your on-premise application and your web application. This can be performed using queue messages, for example:
The web application (cloud) requests resources from the brokering service (cloud) by sending a queue message
The service handles the queue message and makes it available to the consuming (on-prem) service
On-prem service checks for new messages from the brokering service, gets the request for data, and returns desired data from DB
On-prem service sends message to brokering service with desired data
Web app (cloud) checks for new messages from the brokering service, then uses the data from on-prem service
Service bus is secure, asynchronous, fault-tolerant, and ensures that both components are decoupled.
Another method is to use Windows Azure Connect, which is a VPN solution that sets up network-level connnectivity. I recommend Service Bus because it promotes a more robust and scalable architecture, and fault-tolerance is high.