I have looked around the Azure portal and searched the net, but I have been unable to find an answer. Is there a way, perhaps via the api or powershell, to get metrics on webjobs? Such as average runtime per individual job? I would also like to find out the average queued time of a message that a webjob triggers from (though that is probably a storage metric not a webjob metric). Any pointers would be appreciated.
As Igorek said, I don't think it is possible either. There are many tools to monitor application. Two of them have Azure integration:
Application Insights
New relic
I have used Application Insights to send metric from a webjob. You can follow this tutorial to setup Application insights n your webjob:
Application Insights on Windows Desktop apps, services and worker roles
If you want to calculate the time to process a message from a queue, you can do something like that:
public async Task ProcessAsync([ServiceBusTrigger("queueName")] BrokeredMessage incommingMessage)
{
var stopwatch = Stopwatch.StartNew();
// Process your message
...
stopwatch.Stop();
// You should only instantiate the TelemetryClient once in your application.
var telemetryClient = new TelemetryClient() { InstrumentationKey = "MyInstrumentationKey"};
//Send your metric
telemetryClient.TrackMetric("ProcessQueueMessageElapsedTime", stopwatch.ElapsedMilliseconds);
}
Don't think this is possible without 3rd party services.. in fact, the only one I know that does this stuff specifically is CloudMonix, which I'm affiliated with.
Related
Team,
I would like to monitor a azure service bus dead letter queue length using normal C#. it should throw an exception when the receiver is not able/late to process messages from the active queue and due to time delay the count in the dead letter queue increases.
Is there a way without using ApplicationInsights ?
While using the full framework .NET client still provides message counts, according to the Azure Service Bus team the advised way is to use Azure Monitor service. The service has a .NET client that can be used to obtain the needed information (example). Service Bus team has also published a sample here. The client has not provided all the information in the past, but that is work in progress and could be different now than before.
In case you're still planning to use Service Bus client to retrieve message counts, I highly advise to use .NET Standard client rather than full framework client. The "new" client doesn't have NamespaceManager, but it has an equivalent, ManagementClient that will provide the functionality you're looking for, including improvements over its predecessor and bug fixes moving forward. The "old" client is on a limited support only.
If you are using the "old" Service Bus SDK, you can get it from MessageCountDetails:
var msg = NamespaceManager.CreateFromConnectionString(connectionString);
var queue = msg.GetQueue(queueName);
var dlqCount = queue.MessageCountDetails.DeadLetterMessageCount;
It is possible to fetch the Count of Messages(both active and dead-letter) in a Queue with the help of the latest Azure Monitor Metrics. Or you can make use of the Azure Monitor in Azure portal, which allows you to configure dashboards and alerts.
I have the following resources
One Mobile/API app
One MVC app
Three Logic apps
One Azure function deployment with 5 functions
I want to have a single tracking number (correlation ID) to track across all instances at the same time. I'm looking at the Contoso Insurance sample, but I'm rebuilding it by hand (not using Azure Deploy scripts).
I've read the deployment code, but I'm not sure if I can merge app insight logs together, or if it's a hack of some sort.
Observations
When I right click on visual studio, I can only associate to Application insights instances that aren't already connected to a *app (web | mobile | api).
However, in the configuration, I can give Application insights a direct GUID which might allow me to achieve the goal of one App Insights activity log for the entire process
Question
Is it possible to have one app insights log among all Mobile/API/Logic/MVC sites?
Is there a way to have (or should I have) one standard app insights instance per web app, then a special dedicated shared app insights instance for my code to call into and log?
What is contoso insurance doing with Azure App Insights?
Jeff from Logic Apps team here -- So the answer is yes - but there are some caveats. We are working to make the experience seamless and automatic, but for now it will require the following. First as a heads up:
First, for Logic Apps we have what's called the client tracking ID -- this is a header you can set on an incoming HTTP Request or Service Bus message to track and correlate events across actions. It will be sent to all steps (functions, connectors, etc.) with the x-ms-client-tracking-id header.
Logic Apps emits all logs to Azure Monitor - which unfortunately today only has a sink into Event Hubs, Storage, and Log Analytics -- not App Insights.
With all of that in-mind, here's the architecture we see many following:
Have your web apps just emit to App Insights directly. Use some correlation ID as needed. When firing any Logic Apps, pass in the x-ms-client-tracking-id header so you can correlate events.
Log your events to App Insights in the Function app. This blog details some of how to do that, and it is also being worked on for a better experience soon.
In your logic app - either write a Function to consume events off of Azure monitor and push to App Insights, or write a function that is an App Insight "logger" that you can call in your workflow to also get the data into App Insights.
This is how Contoso Insurance is leveraging App Insights as far as I understand. We are working across all teams (App Insights, Azure Monitor, Azure Functions, Logic Apps) to make this super-simple and integrated in the coming weeks/months, but for now achievable with above. Feel free to reach out for any ?s
I'm simply trying to work out how best to retrieve messages as quickly as possible from an Azure Service Bus Queue.
I was shocked that there wasn't some way to properly subscribe to the queue for notifications and that I'm going to have to poll. (unless I'm wrong in which case the documentation is terrible).
I got long polling working, but checking a single message every 60 seconds looks like it'll cost around £900 per month (again, unless I've misunderstood that). And if I add a redundant/second service to poll it'll double.
So I'm wondering what the best/most cost efficient way of doing it is.
Essentially I just want to take a message from the queue, perform an API lookup on some internally held data (perhaps using hybrid services?) and then perhaps post a message back to a different queue with some additional information .
I looked at worker roles(?) -- is that something that could do it?
I should mention that I've been looking at doing this with node.js.
Check out these videos from Scott Hanselman and Mark Simms on Azure Queues.
It's C# but you get the idea.
https://channel9.msdn.com/Search?term=azure%20queues%20simms#ch9Search
Touches on:
Storage Queues vs. Service Bus Queues
Grabbing messages in bulk vs. one by one (chunky vs. chatty)
Dealing with poison messages (bad actors)
Misc implementation details
Much more stuff i can't remember now
As for your compute, you can either do a VM, a Worker Role (Cloud Services), App Service Webjobs, or Azure Functions.
The Webjobs SDK and Azure Functions bot have a way to subscribe to Queue events (notify on message).
(Listed from IaaS to PaaS to FaaS - Azure Functions - if such a thing exists).
Azure Functions already has sample code provided as templates to do all that with Node. Just make a new Function and follow the wizard.
If you need to touch data on-prem you either need to look at integrating with a VNET that has site-to-site connectivity back to your prem, or Hybrid Connections (App Service only!). Azure Functions can't do that yet, but every other compute is a go.
https://azure.microsoft.com/en-us/documentation/articles/web-sites-hybrid-connection-get-started/
(That tutorial is Windows only but you can pull data from any OS. The Hybrid Connection Manager has to live on a Windows box, but then it acts as a reverse proxy to any host on your network).
To deal with Azure ServiceBus Queue easily, the best option seems to be Azure Webjob.
There is a ServiceBusTrigger that allows you to get messages from an Azure ServiceBus queue.
For node.js integration, you should have a look at Azure Function. It is built on top of the webjob SDK and have node.js integration :
Azure Functions NodeJS developer reference
Azure Functions Service Bus triggers and bindings for queues and topics
In the second article, there is an example on how get messages from a queue using Azure Function and nodejs :
module.exports = function(context, myQueueItem) {
context.log('Node.js ServiceBus queue trigger function processed message', myQueueItem);
context.done();
};
I have a MVC website that gets published to Azure where it uses an Azure SQL Database.
The time has come where we need to run a scheduled task to send SMS reminders. I was under the impression that Azure Web Jobs was a good fit for this but am having some issues getting it up and running.
I have added a console app to my website solution and referenced by EF data model from the console app (which I would like to publish as a web job).
The current console app looks as follows:
class Program
{
static void Main(string[] args)
{
JobHost host = new JobHost();
host.RunAndBlock();
}
public static void ProcessNotifications()
{
var uow = new KirkleesDatabase.DAL.UnitOfWork();
uow.CommunicationRepository.SendPALSAppointmentReminders();
}
}
Running the console app will then throw the exception:
Additional information: User account connection string is missing. This can be set via the 'AzureJobsData' connection string or via the constructor.
This suggests that the Web Job is specifically looking for a connection string that points at a storage account. However, I would like the web job to query an Azure database rather than work with storage.
Is this doable?
Thanks,
As Victor wrote, the sdk events are triggered by blobs, queues and tables.
A simple solution for your need: write a console application with a pollig approach. No sdk needed. Details at the beginning of http://blog.amitapple.com/post/73574681678/git-deploy-console-app/
while(true)
{
if ("isthereworktodo?")
{
var uow = new KirkleesDatabase.DAL.UnitOfWork();
uow.CommunicationRepository.SendPALSAppointmentReminders();
}
Thread.Sleep(10000); // set an appropriate sleep interval
}
Unfortunately the WebJobs SDK does not support SQL databases as triggers. For triggers, it only supports Azure Storage (blobs, queues and tables).
You can access Azure SQL Databases from the web job as demonstrated in this answer.
To create a web job you don't have to use the webjobs sdk, you can use several types of executables (.exe, .cmd, .js, .php, .py, ...).
Try using this new visual studio add-on: http://visualstudiogallery.msdn.microsoft.com/f4824551-2660-4afa-aba1-1fcc1673c3d0, follow the steps there to link between your web application and your console application which will be deployed as a webjob when published to your Azure site.
I'd like to understand how much of the following scenario is supported by the Azure Service Bus:
Worker server notifies an unknown number of webservers of an event
Each web server processes the message (processing takes sometime - 10-30 minutes)
When every web server is done with processing of the first message, all of the web servers need to receive a new event. Basically I'm trying to synchronize a number of web roles after performing a long-running job on each web role.
How much of this can I get "for free" from Azure service bus?
Azure Service Bus has a lot of rich messaging features to help with both the Pub/Sub aspect of your requirement as well as the request/response correlation. The concept of sessions (grouped/related) messages along with session state can be very helpful here. Following are some specific links that may help:
MSDN article on sessions: http://msdn.microsoft.com/en-us/magazine/jj863132.aspx
Sample for using sessions: http://code.msdn.microsoft.com/windowsazure/Brokered-Messaging-Session-41c43fb4
Request/Response sample: http://code.msdn.microsoft.com/windowsazure/Brokered-Messaging-Request-0ce8fcaf
Talk on correlation etc.: http://channel9.msdn.com/Blogs/Subscribe/Service-Bus-Messaging-Deep-Dive
You may also want to take a look at the recently released Service Bus Durable Task Framework preview here. For some samples of how to use this framework, see here.
Basically, using this framework on top of Service Bus features such as sessions etc you could write some C# orchestration code that does something roughly like this:
...
// phase 1
List<Task> taskList = new List<Task>();
foreach (var serverName in serverList)
{
taskList.Add(context.ScheduleTask<object>(typeof(ExecutePhase1OnWebServerActivity), serverName));
}
// wait for all of the executions to finish
await Task.WhenAll(taskList);
// phase 2
taskList = new List<Task>();
foreach (var serverName in serverList)
{
taskList.Add(context.ScheduleTask<object>(typeof(ExecutePhase2OnWebServerActivity), serverName));
}
await Task.WhenAll(taskList);
...
Based on how i interpret your question, it Sounds to me like the complicated bit would be not sending the 2nd message until you are sure all of the unknown number of servers had processed their message.
While service bus would offer pub/sub and correlation features which could help you here, to me your describing a pattern where servers would register their interest and also acknowledge that they have processed their message so something would then count up the ack's and once everyone had ack'd the 2nd message would be sent.
This orchestration isnt something service bus can help you with. This puts you in the place of building something in a worker role or possibly using BizTalk when its available in a VM role.
Something like a long running BizTalk orchestration to handle the registration, tracking of acknowledgements and publishing a new message combined with a service bus topic for pub sub to the web servers could be a way to do this
This looks to me like a combination of Workflow Foundation and Service Bus.