Looking at the Kudu Azure WebJobs API docs https://github.com/projectkudu/kudu/wiki/WebJobs-API I see there are several calls I can do to manage WebJobs programmatically.
What is missing is a call to get, for a continuous webjob, the details of a single invocation. With invocation I mean the single execution of the function for a given message.
What I am trying to do is, for a message getting in the poison queue, to get the exception message of the parent invocation. In the poison message I get the id of the parent invocation with the json prop $AzureWebJobsParentId.
I would like to manage the poison queue with a function that emails the details of the error and moves the message in a dead-letter queue.
Any idea if this is possible?
The Azure WebJobs SDK Core extensions contain a binding for ExecutionContext which allows you to access invocation specific system information in your function. An example showing how to access the function Invocation ID:
public static void ProcessOrder(
[QueueTrigger("orders")] Order order,
TextWriter log,
ExecutionContext context)
{
log.WriteLine("InvocationId: {0}", context.InvocationId);
}
The invocation ID is used in the Dashboard logs, so having access to this programatically allows you to correlate an invocation to those logs.
Create a function specific error handler that will only handle errors for one function. This is done by naming convention based on an "ErrorHandler" suffix.
public static void ProcessOrderErrorHandler(
[ErrorTrigger()] TraceFilter filter,
TextWriter log)
{
var lastMessage = filter.Message;
var lastMessages = filter.GetDetailedMessage(5);
}
For email notifications
public static void ErrorMonitor(
[ErrorTrigger("0:30:00", 10, Throttle = "1:00:00")] TraceFilter filter,
[SendGrid] SendGridMessage message)
{
message.Subject = "WebJobs Error Alert";
message.Text = filter.GetDetailedMessage(5);
}
There's not an official way to do this yet, but there's an issue to track exposing stable APIs (C# and REST) for reading the individual function instances: See https://github.com/Azure/azure-webjobs-sdk/issues/880 for status
Given a WebJobs SDK invocation ID, you can access the details of that execution via the WebJobs Dashboard. You can access the Dashboard via the WebJobs blade in the portal (the link will be in the "LOGS" column).
Or in the browser you can form the URL yourself, for example (substituting your app name and invocation ID):
https://<yourapp>.scm.azurewebsites.net/azurejobs/#/functions/invocations/<invocation-id>
That is how you would access these details manually.
Related
I'm taking over responsibility for an Azure Service Bus deployment. Documentation is incomplete in that there are undocumented Queues and Topic Subscriptions present.
What information can Azure provide me about the function that is consuming messages sent to a queue/topic subscription? I'd like ultimately to be able to locate the source code which is processing the messages.
e.g. a class library name, class name or even just a namespace of the function that receives the messages. Something. Anything.
Or am I asking the wrong question?
Seems like an oversight on Azure that all information about Service Bus ends at the definition of the "inbox" and offers nothing about the functions that deal with received messages.
Problem is, the whole idea of using a Service Bus is to decouple (parts of) applications, so the sender does not have a tight relationship with the receiver of the message. From your story it seems your consumers are probably Azure Functions but in other situations receiver might be running on-premises or in an other cloud environment like AWS or Google Cloud. The Azure Service Bus does simply not know what kind of hosting environment is consuming the messages. Tools like Application Insights can generate this kind of information but only if enabled on the consuming side.
What information can Azure provide me about the function that is consuming messages sent to a queue/topic subscription? I'd like ultimately to be able to locate the source code which is processing the messages.
e.g. a class library name, class name or even just a namespace of the function that receives the messages. Something. Anything.
If the producing/consuming applications have an Application Insights integration the Application Map might reveal this information.
Enabling Azure Functions to consume Service Bus messages is all based on configuration so you could scan your code repositories based topic names but there are plenty of other locations where this configuration can be stored like Azure Function Application Settings, Azure Key Vault, Azure App Configuration etc.
Even a statement "There are no consumers on this queue/subscription" would be a help.
The closest you can get is see whether there are active messages waiting to be delivered:
If there is no (active) consuming subscription while messages are pushed I would expect an increasing value of the Active Message Count metric. But do mind the Message Time to Live for the specific topic.
Or am I asking the wrong question?
It is a valid question and I definitely feel your pain but there is no satisfying answer to your question. For loosly coupled systems you need documentation, distributed tracing and possibly infrastructure as code to tackle issues like this.
Getting a list of Azure Functions and their bindings
In your comments you mention that it would help to get a list of functions and their bindings. I do know how to do that using the management SDK:
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.AppService;
internal class Program
{
private static async Task Main(string[] args)
{
var client = new ArmClient(new DefaultAzureCredential(new DefaultAzureCredentialOptions()
{
ExcludeVisualStudioCredential = true,
ExcludeVisualStudioCodeCredential = true
}));
var azureFunctions = new List<SiteFunctionResource>();
foreach (var subscription in client.GetSubscriptions())
{
var resourceGroups = subscription.GetResourceGroups();
await foreach (var resourceGroup in resourceGroups.GetAllAsync())
{
var websites = resourceGroup.GetWebSites();
azureFunctions.AddRange(
websites
.Where(ws => ws.Data.Kind.Contains("functionapp", StringComparison.InvariantCultureIgnoreCase))
.SelectMany(ws => ws.GetSiteFunctions().ToList())
);
}
}
foreach (var azureFunction in azureFunctions)
{
var bindings = azureFunction.Data.Config.ToObjectFromJson<Config>().bindings;
Console.WriteLine(azureFunction.Id);
Console.WriteLine("bindings:");
foreach (var binding in bindings)
{
Console.WriteLine($"\t{binding.name}: {binding.type}");
}
Console.WriteLine("-----------------------------------");
}
Console.ReadLine();
}
public class Binding
{
public string type { get; set; }
public string route { get; set; }
public List<string> methods { get; set; }
public string authLevel { get; set; }
public string name { get; set; }
}
public class Config
{
public List<Binding> bindings { get; set; }
}
}
It needs the following NuGet packages:
<PackageReference Include="Azure.Identity" Version="1.7.0-beta.1" />
<PackageReference Include="Azure.ResourceManager" Version="1.3.0" />
<PackageReference Include="Azure.ResourceManager.AppService" Version="1.0.0-beta.3" />
Sample output:
Azure Event Grid is used to capture all Azure Resource Write Success activities. I'm trying to filter users by email address.
data.clamins.name gives the username in similar way when tried data.claims.'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name' the filter is not working as expected.
Should there be escape characters to make 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name' filter work ?
This is used for below use case of tagging newly created resources for deletion
Event grid invoke azure automation script when there is a Resource Write Success event in Azure
Script tag resources with expire tag
A second script take care of deletion.
I think, there is no either documented or undocumented way/workaround using this kind of naming in the advanced filtering. It will be nice to have it for claims, for instance:
data.claims.schemas_ws200505identity_emailaddress
where the schemas_ws200505identity_ represents value of http://schemas.xmlsoap.org/ws/2005/05/identity/claims/
Anyway, for your solution the place for filtering this property is in the subscription handler. The following code snippet shows this example:
public static async Task Run(JObject eventGridEvent, ILogger log)
{
// ...
var emailaddress = eventGridEvent["data"]?["claims"]?["http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"]?.Value<string>();
log.LogInformation($"emailaddress = {emailaddress}");
// ...
await Task.CompletedTask;
}
I know you can track normal operations using the standard API: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-async-operations
However I was wondering if there was a known way to utilize the Fluent Azure Management Libraries to track long async operations such as VM operations etc. For example the VM restart method is a void Task which does not return an operation-id for tracking.
async Task IVirtualMachineScaleSetVM.RestartAsync(CancellationToken cancellationToken)
{
await this.RestartAsync(cancellationToken);
}
Cheers!
AFAIK, it seems that it's hard to trace VM restart status which does not return an operationId.
Logging in the fluent Azure management libraries for .NET leverages the underlying AutoRest service client tracing.
Create a class that implements Microsoft.Rest.IServiceClientTracingInterceptor. This class will be responsible for intercepting log messages and passing them to whatever logging mechanism you're using.
class ConsoleTracer : IServiceClientTracingInterceptor
{
public void ReceiveResponse(string invocationId, HttpResponseMessage response) { }
}
Before creating the Microsoft.Azure.Management.Fluent.Azure object, initialize the IServiceClientTracingInterceptor you created above by calling ServiceClientTracing.AddTracingInterceptor() and set ServiceClientTracing.IsEnabled to true. When you create the Azure object, include the .WithDelegatingHandler() and .WithLogLevel() methods to wire up the client to AutoRest's service client tracing.
ServiceClientTracing.AddTracingInterceptor(new ConsoleTracer());
ServiceClientTracing.IsEnabled = true;
var azure = Azure
.Configure()
.WithDelegatingHandler(new HttpLoggingDelegatingHandler())
.WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
.Authenticate(credentials)
.WithDefaultSubscription();
For more details, you could refer to this article.
We are facing issue with web API which is hosted in Azure web app. We are inserting a message in Azure Service Bus . Getting intermittent Timeout issue when message is sent to Service Bus queue through Azure web app but when debugging the same code locally it works fine. The service bus is same in both cases. Not able to find the reason for same.Below is code snippet.
Logger.Log(ErrorLevel.Info, "Started Sending Message to Service Bus for Job Id :" + newjobObject.JobId);
ServiceBusRepository.SendMessageToQueue(jobEntity.JobId);
Logger.Log(ErrorLevel.Info, "Completed Sending Message to Service Bus for Job Id :" + newjobObject.JobId);
This is code for SendMessageToQueue method
public static void SendMessageToQueue(string Message)
{
try
{
var message = new BrokeredMessage(Message);
Logger.Log(ErrorLevel.Info,"Before SendMessagetoQueue= " + message);
sendQueueClient.Send(message);
Logger.Log(ErrorLevel.Info,"After SendMessagetoQueue= " + message);
}
catch(Exception ex)
{
Logger.Log(ex.Message.ToString());
}
}
In logs only first logging statement is getting logged not the second one. So it means while sending message some exception is occurring.Anyone can provide pointers to debug this issue.
Intermittent timeout exceptions (System.TimeoutException) are transient exceptions and are acceptable. According to documentation, you should add a retry logic fro any transient exception.
In addition to that, for timeout exceptions, you might want to adjust messaging factory settings (see documentation here).
When the client throws an exception, have a closer at that exception. It will contain info that could help. Would not hurt to post it along with the question.
I have an MVC app running on local IIS server. I have an existing App Insights resource on Azure where I want Telemetry from my app to show up. However, Trace messages are not showing up on Azure Portal. I have added ApplicationInsights to my project, specified the resource where Telemetry should show up in ApplicationInsights.config and also written a TelemetryWrapper that I use in my code to send out the actual Telemetry Info and Error messages.
I initialize the Telemetry service through the wrapper:
TelemetryWrapper.InitializeTelemetry("InstrumentationKey", "ApplicationName");
And send out messages from the wrapper too
TelemetryWrapper.TelemetryInfo(requestId, action, userId, ultimateId, systemId, payload);
An overview of the TelemetryWrapper:
public static void InitializeTelemetry(string apiKey, string appId)
{
bool postBackground = true;
_telemetry = new TelemetryHelper(apiKey, appId, false) { PostBackground = postBackground };
}
public static void TelemetryInfo(Guid requestId, string action, Guid userId, string ultimateId, string systemId,
string payload)
{
var telem = CreateInstance(requestId, action, ultimateId, userId, systemId, payload);
_telemetry.Info(telem);
}
What am I possibly doing wrong?
Need more information. all you've shown is your code, there's no AppInsights code here to look at. so it could be that your TelemetryHelper class isn't setting the ikey correctly, the ikey could be invalid.
However: The one thing that jumps out at me is your usage of the terms apiKey and appId in your code.
Those 2 things have specific meaning in application insights, but that is for reading data from the AI API services (and apiKey in that context is something you could consider a secret, as if someone has those keys they can read all of your data!) I'm not sure if you're using them in a different context here but that jumped out at me right away.
There's a different thing, "Instrumentation Key" or iKey that you use to send data to application insights. Are you possibly switching those things around and trying to use an AI appId and apiKey in place of an iKey?
One way to check is to use fiddler or some other tool locally to watch outbound http traffic. configure it to decrypt https, and watch for outbound calls to dc.services.visualstudio.com, which is where your telemetry is going. If you have outbound calls, then it is at least trying to send your telemetry. If those calls are failing, the response might tell you that the iKey you are using is invalid, if this is the case.
You do not need