I wanted to create a console app as a WebJob using .NET Core but the WebJobs SDK is not yet available in .NET Core.
I've been advised to handle the scenario of reading messages from Azure Storage Queue manually. Looks like all the WebJobs SDK does is to keep polling the queue anyway.
Is the following code the basic idea in doing this? It doesn't look very sophisticated but not sure how it can be more sophisticated.
static void Main(string[] args)
{
var runContinuously = true;
while (runContinuously)
{
ReadAndProcessMessage();
System.Threading.Thread.Sleep(1000);
};
}
private static void ReadAndProcessMessage()
{
// Read message
ReadMessage();
// Process message and handle the work
HandleWork();
}
That will work. And I like simplicity.
The QueueTriggerAttribute makes use of a random exponential back-off algorithm to help minimize your transaction costs. If you'd like to trace through the logic of how this is accomplished, starting with the QueueListener class is a good way to go. Clone the project and then hop over to the RandomizedExponentialBackoffStrategy class.
Related
I have an Azure webjob that is triggered by a queue (inherited, not originally written by me). The queue usually only has one item placed on it at a time, but on the first of every month has many items.
On these occasions it has always processed one queue item at a time, finishing processing before picking up the next.
I noticed, however, that as of a couple of months ago it started processing two files at any one time, which is causing problems.
Whilst i could refactor the code to allow for this, I really don't have the time, and the return would be minimal. I simply want it to process one item at a time again, but i cant find anything that may have caused this to change.
Are there any settings in the azure portal I should be aware of? I don't believe any code relating to the trigger itself has changed.
Thank you in advance
Sure, this can be done. Note that a WebJob can be triggered by either a Service Bus Queue or an Azure Storage Queue. Here's info for both.
For Azure Storage Queues
By default, a QueueTrigger will grab 16 messages at a time and process them in parallel. If you don't want this, you need to set the JobHostConfiguration instance's BatchSize property to 1 in your WebJob's static void Main method. Example:
static void Main(string[] args)
{
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.BatchSize = 8;
JobHost host = new JobHost(config);
host.RunAndBlock();
}
For Service Bus Queues
Similarly, you'll set properties in the JobHostConfiguration. If you're using Service Bus, there's a little more setup. Example:
static void Main(string[] args)
{
JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration();
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);
JobHost host = new JobHost(config);
host.RunAndBlock();
}
In my case, the problem was that someone had scaled out the Azure app service to 2 instances. However, I'm marking Rob's answer as the most helpful.
I have an ASP.NET Core a website with a lot of simultaneous users which crashes many times during the day and I scaled up and out but no luck.
I have been told my numerous Azure support staff that the issue is that I'm sending out a lot of database calls although database utilization improved after creating indexes. Can you kindly advise what you think the problem is as I have done my best...
I was told that I have "socket leaks".
Please note:
I don't have any external service calls except to sendgrid
I have not used ConfigureAwait(false)
I'm not using "using" statements or explicitly disposing contexts
This is my connection string If it may help...
Server=tcp:sarahah.database.windows.net,1433;Initial Catalog=SarahahDb;Persist Security Info=False;User ID=********;Password=******;MultipleActiveResultSets=True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Max Pool Size=400;
These are some code examples:
In Startup.CS:
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
Main class:
private readonly ApplicationDbContext _context;
public MessagesController(ApplicationDbContext context, IEmailSender emailSender, UserManager<ApplicationUser> userManager)
{
_context = context;
_emailSender = emailSender;
_userManager = userManager;
}
This an important method code for example:
string UserId = _userManager.GetUserId(User);
var user = await _context.Users.Where(u => u.Id.Equals(UserId)).Include(u => u.Messages).FirstOrDefaultAsync();
// some other code
return View(user.Messages);
Please advise as I have tried my best but this is very embarrassing to me in font of my customers.
Without the error messages that you're seeing, here's a few ideas that you can check.
I'd start with going to your Web App's Overview blade in the Azure Portal. Update the monitoring graph to a time period when you're experiencing problems. Are you CPU bound? Have you exhausted memory? Also, check the HTTP Queue length. If your HTTP queue is really long, it's because your server is choking trying to service the requests and users are experiencing timeout issues.
Next, jump over to your SQL Server's Overview blade in the Azure Portal, and look at the resource utilization chart. Set the time period on the chart to when you're experiencing problems. Have you pegged out your DTUs for your database? If so, it's a sign of poor indexing, poor schema design, or you're just undersized and need to scale up.
Turn on ApplicationInsights if you haven't already. You can use the ApplicationInsights API to insert your own trace statements into your code. Or, you might be able to see exceptions causing the issue without having to do your own tracing.
Check the Kudu logs for your Web Apps.
I agree with Tseng - your usage of EF and .NET Core's DI framework looks correct.
Let us know how the troubleshooting goes and provide additional information on exactly what kind of errors you're seeing. Best of luck!
It looks like a DI issue to me. You are injecting ApplicationDbContext context. Which means the ApplicationDbContext will be resolved from the DI container meaning it will stay open the entire request (transient) as Tseng pointed out. It should be a scoped.
You can inject IServiceScopeFactory scopeFactory in your controller and do something like:
using (var scope = _scopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
}
Note that if you are using ASP.NET Core 1.1 and want to be sure that all your services are being resolved correctly change your ConfigureService method in the Startup to:
public IServiceProvider ConfigureServices(IServiceCollection services)
{
// Register services
return services.BuildServiceProvider(validateScopes: true);
}
Trying to get Azure Webjobs to react to incoming Service Bus event, Im running this by hitting F5. Im getting the error at startup.
No job functions found. Try making your job classes and methods
public. If you're using binding extensions (e.g. ServiceBus, Timers,
etc.) make sure you've called the registration method for the
extension(s) in your startup code (e.g. config.UseServiceBus(),
config.UseTimers(), etc.).
My functions-class look like this:
public class Functions
{
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([ServiceBusTrigger("test-from-dynamics-queue")] BrokeredMessage message, TextWriter log)
{
log.WriteLine(message);
}
}
I have every class and method set to public
I am calling config.UseServiceBus(); in my program.cs file
Im using Microsoft.Azure.WebJobs v 1.1.2
((Im not entirely sure I have written the correct AzureWebJobsDashboard- and AzureWebJobsStorage-connectionstrings, I took them from my only Azure storage-settings in Azure portal. If that might be the problem, where should I get them ))
According to your mentioned error, it seems that you miss parameter config for ininitializing JobHost. If it is that case, please use the following code.
JobHost host = new JobHost(config)
More detail info about how to use Azure Service Bus with the WebJobs SDK please refer to the document.The following is the sample code from document.
public class Program
{
public static void Main()
{
JobHostConfiguration config = new JobHostConfiguration();
config.UseServiceBus();
JobHost host = new JobHost(config);
host.RunAndBlock();
}
}
Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.
I'm trying to build an event driven Azure Queue where a event is to fired every time a message is put in the Azure Queue. With AzureXplorer I see that the messages are put in the Azure Queue properly but the CloudQueueClient.ResponseReceived Event never fires. I'm using Azure V1.4. This is the code from my Worker role:
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
while (true)
{
Thread.Sleep(10000);
}
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
var queuDataSource = new AzureQueueDataSource();
queuDataSource.GetCloudQueueClient().ResponseReceived +=new EventHandler<ResponseReceivedEventArgs>(WorkerRole_ResponseReceived);
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
return base.OnStart();
}
void WorkerRole_ResponseReceived(object sender, ResponseReceivedEventArgs e)
{
var i = 1; // Breakpoint here never happends
}
}
Windows Azure Queues need to be polled for new messages. See SDK samples or code here for examples on how to query queues for new messages.
Quick list of things to take into account:
Because polling is counted as a
transaction in Windows Azure, you
will be paying for those.
It is usually better to implement some kind of retry mechanism if no messages are found (e.g. exponential back-off, etc)
It is usually good to retrieve messages in batches (less round trips, less transactions, etc)
Remember that messages can be delivered more than once (plan for duplicate messages)
Use the "dequeuecount" property to deal with "poison messages".
There's plenty of coverage on all these. See the documentation/samples in the link above. This article is pretty good too: http://blogs.msdn.com/b/appfabriccat/archive/2010/12/20/best-practices-for-maximizing-scalability-and-cost-effectiveness-of-queue-based-messaging-solutions-on-windows-azure.aspx