Azure Webjobs and Queues - azure

Question

I am working with an Azure Service Bus Queue (or potentially a topic if required), and would like to know how a Web Job can be used with the Queue.
When a message comes onto the queue it represents a process that will run within the web job (or be started from the webjob). This process might be quick, 30 seconds, or it might be slow, 1 hour etc.
Can I use a single Web Job for this and somehow say that it should be running no more than 10 of these processes at a time?

Answer 1

Yes you can use a WebJob. I have created a simple WebJob with Storage Queue to just guide how it can be done. The below workflow will run only ten process at a time and keep all the other requests in memory of ConcurrentQueue. You will have to implement the logic to dequeue it and consume it
public class Functions
{
public delegate void CompletedProcessHandler(object sender, CompletedProcessHandlerArgs args);
static readonly Dictionary<int, CustomProcess> _dictionary =
new Dictionary<int, CustomProcess>();
static readonly ConcurrentQueue<ProcessEntity> _remaining =
new ConcurrentQueue<ProcessEntity>();
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("testqueue")] ProcessEntity msg,
TextWriter log)
{
if (_dictionary.Count <= 10)
{
var newProcess = new CustomProcess((_dictionary.Last().Key) + 1,
msg.Duration);
}
else
{
_remaining.Enqueue(msg);
}
}
public static void CompletedProcess(object sender, CompletedProcessHandlerArgs args)
{
_dictionary[Int32.Parse(args.ProcessID)].Dispose();
_dictionary.Remove(Int32.Parse(args.ProcessID));
}
}
public class CustomProcess : IDisposable
{
public event Functions.CompletedProcessHandler OnProcessCompleted;
private CancellationTokenSource _token;
private string _id;
private Timer _timer;
public CustomProcess(int i, int duration)
{
_timer = new Timer { Enabled = true, Interval = duration * 1000 };
_timer.Elapsed += Timer_Elapsed;
_id = i.ToString();
_token = new CancellationTokenSource();
Task.Factory.StartNew(() => WriteMessages());
_timer.Start();
OnProcessCompleted += Functions.CompletedProcess;
}
private void Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
_token.Cancel();
OnProcessCompleted?.Invoke(this, new CompletedProcessHandlerArgs(_id));
}
private void WriteMessages()
{
while (!_token.Token.IsCancellationRequested)
{
Console.WriteLine("Test Message from process " + _id);
}
}
public void Dispose()
{
_token.Dispose();
_timer.Dispose();
}
}
public class CompletedProcessHandlerArgs : EventArgs
{
public string ProcessID { get; set; }
public CompletedProcessHandlerArgs(string ID)
{
ProcessID = ID;
}
}
public class ProcessEntity
{
public int Duration { get; set; }
}
In the app.config of the web job you need to provide the two app settings
<add name="AzureWebJobsDashboard"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
<add name="AzureWebJobsStorage"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
The Program file is the default one from the Visual Studio template
public class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
WebJob will keep dequeue the message the moment it comes. Since you want only 10 to run at a time you will have to enqueue the message in memory and wait for running process to complete before you start a new one
As #Rick has mentioned you can set the is_Singleton property to true in settings.job file of the web job

Answer 2

Yes, you can trigger a web job with an Azure Service Bus Queue or Topic. A good example to look at to get you going would be the Service Bus quick start project template in Visual Studio.
In particular, you want to look at the ServiceBusTrigger attribute that the Web Jobs SDK provides.
As for the scalability of the web job, this will scale according to your web app instances. So, if you had say 5 instances of your web app with always on enabled, then you would have 5 instances of your web job. As an additional comment on this, if you wanted just one instance of the web job in an environment of 5 web app instances, then you could set the is_singleton property to true in the settings.job file.