I'm playing around with the Application Map feature of Application Insights and I'm struggling to generate what I expected to see.
I have a "Sender" (HTTP Trigger Azure Function) that writes to a Service Bus Queue via a binding, and a "Receiver" (Service Bus Trigger Azure Function) that reads from the same queue.
I was expecting to see an Application Map that is something like:
Sender -> Service Bus -> Receiver
But I'm struggling. The sending bit seems to work out of the box, as I get:
Image showing Sender -> Service Bus
Unfortunately I'm unable to generate an arrow that leaves the Service Bus. I've tried putting the receiver in a separate project to the sender, as well as the receiver using a separate App Insights resource. This latter attempt just gave me two different app insights to read and all I can see if the function being executed.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-end-to-end-tracing?tabs=net-standard-sdk-2#trace-message-processing suggests it's possible in normal .NET code. But I can't find an equivalent resource for functions.
Is what I'm expecting possible?
Does this work if you disable the preview mode in Application Map? If so, it might be because you need to send more than just 1 call at a time. The preview mode requires that you send data for more than 1 minute on a new Application Insights resource. I tried an out-of-the-box solution using C# Script and it looks like correlation should work at least with Service Bus extension version: Microsoft.Azure.WebJobs.Extensions.ServiceBus 4.2.1 (from the default Functions 2.0 extension bundle).
Distributed tracing of Azure ServiceBus was only release in version v2.21 - you can see the information in the release notes for beta 2
Added support for distributed tracing with Azure.Messaging.ServiceBus
Sorry for all the links, but doesn't make much sense without them.
Related
I am having trouble to identify the best "tool" to solve the problem. I am using a python library which publishes to its data via Sever-Sent-Events (SSE) (see https://github.com/wattsight/wapi-python/blob/development/wapi/events.py).
I would like to constantly listen for new events. However, I am not sure which tool in Azure in appropriate. An Azure Function would have to run continously which seems like a misuse, SignalR requires control over the "sender" of events and I don't know if EventHub would be able to manage that job.
Thank you for letting my learn from your experience.
The azure Eventhub is the right service to receive new events. Besides that, it can also provide other benefits like scalability, events storage etc.
And you can also consider using azure function with eventhub trigger. But it has a limitation about the incoming message max size.
I am using Azure Functions V2 with a Service Bus trigger using 1.0.23 of the C# Functions SDK. I'm using the following approach to get secrets from KeyVault and use them within the settings of the triggers: How to map Azure Functions secrets from Key Vault automatically
The function, especially when it has done nothing for a while, doesn't fire when there are messages on the subscription. If I then go to the portal and execute manually (yes, that particular execution is fired with a null message) it kicks it into life and picks up the other messages on the queue and processes them correctly.
This obviously isn't ideally for our automated tests. Has anybody seen this, or know of anything that will help?
Also, the Function App is running on a consumption plan.
App Service Plan
If you're using App Service plan then it's simple, just make use of Always on
Consumption Plan
If you're using Consumption plan, the issue could be that your triggers did not sync properly with the Azure Infrastructure (Central Listener). It could have happened due to the way you deployed/edited your trigger related settings as explained in issue #210 below.
When you access the function directly from Portal, it might be forcing your function app to come alive, but as you can see that's only a workaround. Something similar is mentioned here
Take a look at these issues:
Service Bus Topic Trigger goes to sleep - Consumption Plan
They also mention that it wakes up only on accessing it via the portal or calling a HTTP triggered function in the same app, which is similar to the behavior you are seeing.
Issue #210
Issue #681
There are 3 suggested ways to resolve it, mentioned as part of Issue #210 above
In order to synchronize triggers when these deployment options are
used, open the Azure Portal and click the Refresh button, or make a
API call to the sync triggers endpoint:
https://github.com/davidebbo/AzureWebsitesSamples/blob/master/ARMTemplates/FunctionsWebDeploy.json#L90
Powershell sample:
https://github.com/davidebbo/AzureWebsitesSamples/blob/master/PowerShell/HelperFunctions.ps1#L360-L365
I've had a similar issue. ServiceBus connection was injected using ServiceBus value in ConnectionStrings section of Function configuration. This is enough when Function is in hot state but after transitioning to cold state AzureWebJobsServiceBus value is used to connect to service bus. So in my case setting AzureWebJobsServiceBus to ServiceBus connection string in Function configuration fixed this.
Team,
I would like to monitor a azure service bus dead letter queue length using normal C#. it should throw an exception when the receiver is not able/late to process messages from the active queue and due to time delay the count in the dead letter queue increases.
Is there a way without using ApplicationInsights ?
While using the full framework .NET client still provides message counts, according to the Azure Service Bus team the advised way is to use Azure Monitor service. The service has a .NET client that can be used to obtain the needed information (example). Service Bus team has also published a sample here. The client has not provided all the information in the past, but that is work in progress and could be different now than before.
In case you're still planning to use Service Bus client to retrieve message counts, I highly advise to use .NET Standard client rather than full framework client. The "new" client doesn't have NamespaceManager, but it has an equivalent, ManagementClient that will provide the functionality you're looking for, including improvements over its predecessor and bug fixes moving forward. The "old" client is on a limited support only.
If you are using the "old" Service Bus SDK, you can get it from MessageCountDetails:
var msg = NamespaceManager.CreateFromConnectionString(connectionString);
var queue = msg.GetQueue(queueName);
var dlqCount = queue.MessageCountDetails.DeadLetterMessageCount;
It is possible to fetch the Count of Messages(both active and dead-letter) in a Queue with the help of the latest Azure Monitor Metrics. Or you can make use of the Azure Monitor in Azure portal, which allows you to configure dashboards and alerts.
I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};
I am working with Azure Service Bus Topics and Subscriptions. It's being used to send control messages across the application. The message listeners (subscribers) are running in a worker role and they are picking up the messages and processing the request. Each message in the bus can be picked up by only once, even if there are multiple listeners running simultaneously.
There is no issue in using the Service Bus; however we are facing some issues while debugging/testing the application in local. We have 2 service bus, one for the cloud and one for local debugging. Now if multiple people are debugging the application simultaneously, the message is being picked by only one of the system (at random). This is the intended behavior, but it's causing a nuisance while debugging.
Is there any way how I can use a local emulator for the Service Bus? I did some research but I couldn't find any reliable solution for this. Is there any way how we debug the application in isolation?
Unfortunately, there is no Azure SB local emulator. That was asked before, and you can try to use Service Bus for Windows Server but it is slightly behind the cloud service in terms of features support/functionality, etc. Still, it supports Azure SDK, for example. MSDN link for SB for WS.
Azure Service Bus is a broker with competing consumers. Having multiple developers debugging using the same namespace will be be tough (message lock duration expired and another developer that happened to be debugging got that message).
I would suggest to look into a namespace per developer. With MSDN license you're given enough Azure credit to have each developer work in a "sandboxed" namespace. As to how to get it working, you can read from a configuration file, environment variable, etc.
On the ASB for Windows Server - currently it's on version 1.1 where Azure SB is on 3+. Hosted version will be always ahead of the on-premises. Something to consider.
The problem should probably be solved by abstracting the service bus implementation so that it can be swapped for something that you actually can run locally (or in memory). I would suggest not to re-invent that wheel though and choose a library like Masstransit https://masstransit-project.com.
We ended up using a different topic name for each developer while debugging which is working pretty good.
We use the devlopers machine name in the path for alot of dev things. for example a queueName like "TheEventMessageQueue" can be "TheEventMessageQueue-Machine123" and we use a dev subscription so we dont make a mess :)
By using the machine name its easy to se who the que belongs to if its going out of control