I've tried to come up with something from the example in the WebJobsSDK gitHub
var eventHubConfig = new EventHubConfiguration();
string eventHubName = "MyHubName";
eventHubConfig.AddSender(eventHubName,"Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=SendRule;SharedAccessKey=xxxxxxxx");
eventHubConfig.AddReceiver(eventHubName, "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=yyyyyyy");
config.UseEventHub(eventHubConfig);
JobHost host = new JobHost(config);
But I'm afraid that's not far enough for someone of my limited "skillset"!
I can find no instance of JobHostConfiguration that has a UseEventHub property (using the v1.2.0-alpha-10291 version of the Microsoft.AzureWebJobs package), so I can't pass the EventHubConfiguration to the JobHost.
I've used EventHub before, not within the WebJob context. I don't see if the EventHostProcessor is still required if using the WebJob triggering...or does the WebJob trigger essentially act as the EventHostProcessor?
Anyway, if anyone has a more complete example for a simpleton like me that would be really sweet! Thanks
From the documentation here, you should have all the information you need.
What you are missing is a reference of the Microsoft.Azure.WebJobs.ServiceBus.1.2.0-alpha-10291 nuget package.
The UseEventHub is an extension method that is declared in this package.
Otherwise your configuration seems ok.
Here is an example on how to receive or send messages from/to an EventHub:
public class BasicTest
{
public class Payload
{
public int Counter { get; set; }
}
public static void SendEvents([EventHub("MyHubName")] out Payload x)
{
x = new Payload { Counter = 100 };
}
public static void Trigger(
[EventHubTrigger("MyHubName")] Payload x,
[EventHub("MyHubName")] out Payload y)
{
x.Counter++;
y = x;
}
}
EventProcessorHost is still required, as the WebJob just provides the hosting environment for running it. As far as I know, EventProcessorHost is not integrated so deeply into WebJob, so its triggering mechanism cannot be used for processing EventHub messages. I use WebJob for running EventProcessorHost continuously:
public static void Main()
{
RunAsync().Wait();
}
private static async Task RunAsync()
{
try
{
using (var shutdownWatcher = new WebJobsShutdownWatcher())
{
await Console.Out.WriteLineAsync("Initializing...");
var eventProcessorHostName = "eventProcessorHostName";
var eventHubName = ConfigurationManager.AppSettings["eventHubName"];
var consumerGroupName = ConfigurationManager.AppSettings["eventHubConsumerGroupName"];
var eventHubConnectionString = ConfigurationManager.ConnectionStrings["EventHub"].ConnectionString;
var storageConnectionString = ConfigurationManager.ConnectionStrings["EventHubStorage"].ConnectionString;
var eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, consumerGroupName, eventHubConnectionString, storageConnectionString);
await Console.Out.WriteLineAsync("Registering event processors...");
var processorOptions = new EventProcessorOptions();
processorOptions.ExceptionReceived += ProcessorOptions_ExceptionReceived;
await eventProcessorHost.RegisterEventProcessorAsync<CustomEventProcessor>(processorOptions);
await Console.Out.WriteLineAsync("Processing...");
await Task.Delay(Timeout.Infinite, shutdownWatcher.Token);
await Console.Out.WriteLineAsync("Unregistering event processors...");
await eventProcessorHost.UnregisterEventProcessorAsync();
await Console.Out.WriteLineAsync("Finished.");
}
catch (Exception ex)
{
await HandleErrorAsync(ex);
}
}
}
private static async void ProcessorOptions_ExceptionReceived(object sender, ExceptionReceivedEventArgs e)
{
await HandleErrorAsync(e.Exception);
}
private static async Task HandleErrorAsync(Exception ex)
{
await Console.Error.WriteLineAsync($"Critical error occured: {ex.Message}{ex.StackTrace}");
}
Related
I am looking into how the app insights work with different types of dependencies. I have a question on using the AppInsights ASP Net Core sdk with Messaging Service Bus sdk.
How can I capture messages to service bus while sending or receiving using this sdk in dependency? I understand that this is not something we would like to log all the time and I will make it configurable.
Thanks
You can create your own sender and implement the desired logic inside the SendMessageAsync method:
public class TelemetryEnabledServiceBusSender : ServiceBusSender
{
private readonly TelemetryClient telemetryClient;
internal TelemetryEnabledServiceBusSender(ServiceBusClient client, string topic, TelemetryClient telemetryClient) : base(client, topic)
{
this.telemetryClient = telemetryClient;
}
public override async Task SendMessageAsync(ServiceBusMessage message, CancellationToken cancellationToken = default)
{
telemetryClient.TrackTrace(message.Body.ToString());
await base.SendMessageAsync(message, cancellationToken);
}
}
use it like this:
var serviceBusSender = new TelemetryEnabledServiceBusSender(serviceBusClient, serviceBusData.Topic, telemetryClient);
await serviceBusSender.SendMessageAsync(message);
Logging processed messages is even simpler and can be done using the ServiceBusProcessor
serviceBusProcessor.ProcessMessageAsync += ProcessMessageAsync;
...
private async Task ProcessMessageAsync(ProcessMessageEventArgs arg)
{
telemetryClient.TrackTrace(arg.Message.Body.ToString());
...
}
Adding my approach as an answer here as it is too long to comment.
var telemetry = new DependencyTelemetry(
"Azure Service Bus",
_serviceBusClient.FullyQualifiedNamespace,
"ServiceBusReceiver.Receive",
string.Empty
);
using var operation =
_telemetryClient.StartOperation(telemetry);
try
{
var receivedMessage = await _serviceBusReceiver.ReceiveMessageAsync();
if (receivedMessage?.Body is not null)
{
message = receivedMessage.Body.ToString();
telemetry.Data = message;
// do something
}
telemetry.Success = true;
//Consider set telemetry duration
}
catch (Exception e)
{
// log exception
_telemetryClient.TrackException(e);
telemetry.Success = false;
throw;
}
finally
{
_telemetryClient.TrackTrace("Done");
_telemetryClient.StopOperation(operation);
}
Thanks to #Peter Bons
I have an Azure durable function triggered by a message that then uses the client to start an Orchestration trigger which starts several activity functions. I have set breakpoints in Orchestration client , trigger and each activity function.
But, it only hits the breakpoints in Orchestration client function and the others are getting ignored. But underneath it seems to execute the activity functions although the breakpoints are not hit.
Investigative information
Programming language used = C#
Visual Studio Enterprise 2019 version = 16.8.3
Azure Functions Core Tools
Core Tools Version: 3.0.3216
Function Runtime Version: 3.0.15193.0
Below here, I have included a code snippet. (have not added every activity function)
[FunctionName(nameof(InitiateExport))]
public static async Task InitiateExport(
[ServiceBusTrigger("%ExportQueueName%", Connection = "AzureSBConnection")]Message message,
[DurableClient(TaskHub = "%FunctionHubName%")] IDurableOrchestrationClient orchestrationClient,
[Inject]IServiceProvider rootServiceProvider, ILogger log)
{
var DataQueuedDetails = JsonConvert.DeserializeObject<DataQueuedDetails>(Encoding.UTF8.GetString(message.Body));
using (var scope = rootServiceProvider.CreateScope())
{
log.LogInformation($"{nameof(ExportData)} function execution started at: {DateTime.Now}");
var services = scope.ServiceProvider;
services.ResolveRequestContext(message);
var requestContext = services.GetRequiredService<RequestContext>();
await orchestrationClient.StartNewAsync(nameof(TriggerDataExport), null, (DataQueuedDetails, requestContext));
log.LogInformation($"{nameof(ExportData)} timer triggered function execution finished at: {DateTime.Now}");
}
}
[FunctionName(nameof(TriggerDataExport))]
public static async Task TriggerDataExport(
[OrchestrationTrigger] IDurableOrchestrationContext orchestrationContext,
[Inject] IServiceProvider rootServiceProvider, ILogger log)
{
using (var scope = rootServiceProvider.CreateScope())
{
var services = scope.ServiceProvider;
var (DataOperationInfo, requestContext) = orchestrationContext.GetInput<(DataQueuedDetails, RequestContext)>();
if (!orchestrationContext.IsReplaying)
log.LogInformation($"Starting Export data Id {DataOperationInfo.Id}");
var blobServiceFactory = services.GetRequiredService<IBlobServiceFactory>();
requestContext.CustomerId = DataOperationInfo.RelatedCustomerId;
try
{
await orchestrationContext.CallActivityAsync(
nameof(UpdateJobStatus),
(DataOperationInfo.Id, DataOperationStatus.Running, string.Empty, string.Empty, requestContext));
// some other activity functions
---
---
} catch (Exception e)
{
await orchestrationContext.CallActivityAsync(
nameof(UpdateJobStatus),
(DataOperationInfo.Id, DataOperationStatus.Failed, string.Empty, string.Empty, requestContext));
}
}
}
[FunctionName(nameof(UpdateJobStatus))]
public static async Task RunAsync(
[ActivityTrigger] IDurableActivityContext activityContext,
[Inject]IServiceProvider rootServiceProvider)
{
using (var scope = rootServiceProvider.CreateScope())
{
try
{
var (DataOperationId, status, blobReference, logFileBlobId, requestContext) = activityContext.GetInput<(string, string, string, string, RequestContext)>();
var services = scope.ServiceProvider;
services.ResolveRequestContext(requestContext.CustomerId, requestContext.UserId, requestContext.UserDisplayName, requestContext.Culture);
var dataService = services.GetRequiredService<IDataService>();
var DataOperationDto = new DataOperationDto
{
Id = DataOperationId,
OperationStatusCode = status,
BlobReference = blobReference,
LogBlobReference = logFileBlobId
};
await dataService.UpdateAsync(DataOperationDto);
}
catch (Exception e)
{
throw e;
}
}
}
While you are debugging your Function, you should make sure that you are either using the local storage emulator, or an Azure Storage Account that is different from the one that is also being used by the Function already deployed in Azure.
If you are using the same storage account that another running Function is using, then it could be that parts of the execution is actually happening in Azure instead of your dev machine.
I'm facing a strange issue, and I ran out of the possible causes. The scenario is
Fetch incoming message from queue
Process it and then add new message to another queue
but the thing is, if I finish the long running task for the incoming message, and then try to add new message to another queue, I don't receive it. If I just add a face message to that another queue, then I am able to receive the real message after the long-running operation is finished. But why ? I don't want to put any fake messages to the queue, but without that my scenario doesn't work. Any ideas ?
public class WorkerRole : RoleEntryPoint
{
// QueueClient is thread-safe. Recommended that you cache
// rather than recreating it on every request
Microsoft.ServiceBus.Messaging.QueueClient Client;
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
public override void Run()
{
MyResult result = null;
var queueClient = new Microsoft.Azure.ServiceBus.QueueClient("QueueConnectionString", "QueueName");
Client.OnMessage(async (receivedMessage) =>
{
try
{
using (Stream stream = receivedMessage.GetBody<Stream>())
{
using (StreamReader reader = new StreamReader(stream))
{
string json = reader.ReadToEnd();
OCRQueueItem_Incoming item = JsonConvert.DeserializeObject<IncomingClass>(json);
var someClass = new OCRManager();
var message = new Message(Encoding.UTF8.GetBytes("test 1"));
await queueClient.SendAsync(message);
result = new SomeManager().RunLongRunningTask(item); //it runs for 1-2min
}
}
}
catch (Exception ex) { }
finally
{
var json = JsonConvert.SerializeObject(result);
var message = new Message(Encoding.UTF8.GetBytes(json));
await queueClient.SendAsync(message);
}
});
CompletedEvent.WaitOne();
}
public override bool OnStart()
{
ServicePointManager.DefaultConnectionLimit = 12;
string connectionString = CloudConfigurationManager.GetSetting("Queue.ConnectionString");
Client = Microsoft.ServiceBus.Messaging.QueueClient.Create(connectionString);
return base.OnStart();
}
public override void OnStop()
{
Client.Close();
CompletedEvent.Set();
base.OnStop();
}
}
I have an Azure job which asynchronously saves records to the database. I am finding that it does not actually save anything to the database. I am definitely using async/awaits everywhere. I am adding parent (market) and child records. My database has referential constraints so the parent has to exist before the child, but that should be fine as I am doing them in the right order. I have no try-catches around my methods and there is nothing in the azure logs so it appears the job is succeeding. I have called Method1 with an await from a Winforms exe and it works fine from that. What can be wrong?
public static async Task MyJob([TimerTrigger("00:02:00", RunOnStartup = true)] TimerInfo timerInfo, TextWriter log)
{
await Jobs.Method1(Client, Logger);
}
public static async Task Method1(IClient client, ILogger logger)
{
await DataRepository.AddMarket(event.Id, event.MarketId);
await DataRepository.AddMarketChild(event.MarketId, 999);
}
public static async Task<Market> AddMarket(string eventId, string marketId)
{
using (var ctx = BTBEntities.CreateContext())
{
var market = new Market()
{
MarketId = marketId,
EventId = eventId,
};
ctx.Markets.Add(market);
await ctx.SaveChangesAsync();
return market;
}
}
public static async Task<HorseBet> AddHorseBet(string marketId, long selectionId)
{
using (var ctx = BTBEntities.CreateContext())
{
var bet = new MarketChild()
{
MarketId = marketId,
SelectionId = selectionId,
};
ctx.MarketChilds.Add(bet);
await ctx.SaveChangesAsync();
return bet;
}
}
Turns out there was a database problem it was just being swallowed by Azure. This is how I fixed it.
public override System.Threading.Tasks.Task<int> SaveChangesAsync()
{
try
{
return base.SaveChangesAsync();
}
catch (DbEntityValidationException ex)
{
var errorMessages = ex.EntityValidationErrors
.SelectMany(x => x.ValidationErrors)
.Select(x => x.ErrorMessage);
var fullErrorMessage = string.Join("; ", errorMessages);
var exceptionMessage = string.Concat(ex.Message, " The validation errors are: ", fullErrorMessage);
throw new DbEntityValidationException(exceptionMessage, ex.EntityValidationErrors);
}
}
I have an Azure Web Job built using the Azure SDK whose only job is to call a web service (Web API) and then log a response based on the return value (a class). The problem is that as soon as it calls the HttpClient PostAsJsonAsync method to call the service, it exits out of the web job without executing any of the response handling. My code is:
public class Result
{
// Properties ---------------------------------------------------------
public bool Success { get; set; }
public string Error { get; set; }
}
public class Functions
{
// This function will be triggered based on the schedule you have set for this WebJob
// This function will enqueue a message on an Azure Queue called queue
[NoAutomaticTrigger]
public async static void ManualTrigger(TextWriter log, int value)
{
using (var client = new HttpClient())
{
var rootUrl = ConfigurationManager.AppSettings.Get("WebJobTargetUrl");
client.BaseAddress = new System.Uri(rootUrl);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Console.WriteLine("Call service");
var response = await client.PostAsJsonAsync("api/Reminder/ProcessDueReminders", new { ItemID = 1 });
Console.WriteLine("After service");
var result = await response.Content.ReadAsAsync<Result>();
Console.WriteLine("After result");
if (result.Success)
Console.WriteLine("Reminders Processed");
else
Console.WriteLine("Reminder process error: " + result.Error);
}
}
}
and the execution logs from the portal are:
I believe it has something to do with the asynchronous operation but I can't figure out a pattern that will work. Any help would be appreciated.
You must define the return value of your own async method as Task instead of void.
On a related note, you should suffix the name of your method with Async. That's not going to solve the problem, but it indicates that you're using the async/await pattern.
There is probably an exception in your PostAsJsonAsync call. Try to put a try catch around it to and log the error:
try {
var response = await client.PostAsJsonAsync("api/Reminder/ProcessDueReminders", new { ItemID = 1 });
} catch (Exception ex){
Console.WriteLine("Exception: "+ ex);
}
Console.WriteLine("After service");