I like to change the timespan a Q-message is invisible in the scenario where a webjob is triggered upon arrival of a message in the Queue.
I need this in the dev-stage while debugging. Sometimes an old Q-msg re-appears due to a bug but I already cleaned up the blob-storage-item the message was refering to. Hence WebJob crashes because it tries to get that not existing blob-object.
If I could set that the Q-messages are again visible after 1 min, that would be helpfull.
public partial class Functions
{
public static void ProcessBulkFileQ(
[QueueTrigger(AppConst.Q_BULKFILES)] JobInfo JobInfo,
[Blob(AppConst.CONTAINER_BULKFILES + "/{BlobName}", FileAccess.Read)] Stream InputStream,
[Blob(AppConst.CONTAINER_BULKFILES + "/{BlobName}")] CloudBlockBlob BlobToDelete
)
{
In the latest v1.1.0 release of the SDK we've added the ability for you to add custom QueueProcessors to control various aspects of message processing. You can see an example of what you're looking to do here in our tests.
In that example, for failed messages, you can explicitly set the visibility to your own value based on the error details. You can register custom QueueProcessors by creating your own factory and setting it on JobHostConfiguration.Queues.QueueProcessorFactory.
Related
This question is similar to others out there with answers, but I think things may have changed with AF v2 (I'm jumping into v2 without v1 experience). It seems that the assembly that AF v2 is integrated with for interacting with ServiceBus is Microsoft.Azure.ServiceBus, which has a class called "Message", which is not quite the same as "BrokeredMessage" found in other microsoft servicebus assemblies. One key difference between the two is that BrokeredMessage (which is referenced in almost all of the documentation, examples, and any other thread out there that I can find) has a .Deadletter() method on it, while Message does not. How can I deadletter a message intentionally when I don't have access to the Client or Receiver that received it?
(I have seen suggestions out there related to just cloning the message, putting on my own "dead letter" queue, and letting AF commit the original - I do not feel this is an adequate solution.)
I have yet to prove it out in the functions environment, but according to the ticket I opened on github (https://github.com/Azure/azure-webjobs-sdk/issues/1986), webjobs v3 supports binding both the Message and MessageReceiver that received it in the function trigger, and then the MessageReceiver can be used to deadletter the message.
This was a real pain to find out as it's still not documented. The API reference on the Azure SDK for .NET project readme even points to the documentation from the old namespace 🤦♂️
I had to search through the source code to find the correct way to get the LockToken.
[FunctionName("MyFunction")]
public static async Task Run([ServiceBusTrigger("myqueue", Connection = "myconnectionstring")] Message message,
ILogger log, MessageReceiver messageReceiver)
{
await messageReceiver.DeadLetterAsync(message.SystemProperties.LockToken);
}
Putting MessageReceiver as a parameter with automatically bind it, no additional set up needed.
Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.
I am developing a triggered webjob that use TimerTrigger.
Before the webjob stops, I need to dispose some objects but I don't know how to trigger the "webjob stop".
Having a NoAutomaticTrigger function, I know that I can use the WebJobsShutdownWatcher class to handle when the webjob is stopping but with a triggered job I need some help...
I had a look at Extensible Triggers and Binders with Azure WebJobs SDK 1.1.0-alpha1.
Is it a good idea to create a custom trigger (StopTrigger) that used the WebJobsShutdownWatcher class to fire action before the webjob stops ?
Ok The answer was in the question :
Yes I can use the WebJobsShutdownWatcher class because it has a Register function that is called when the cancellation token is canceled, in other words when the webjob is stopping.
static void Main()
{
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(() =>
{
Console.Out.WriteLine("Do whatever you want before the webjob is stopped...");
});
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
EDIT (Based on Matthew comment):
If you use Triggered functions, you can add a CancellationToken parameter to your function signatures. The runtime will cancel that token when the host is shutting down automatically, allowing your function to receive the notification.
public static void QueueFunction(
[QueueTrigger("QueueName")] string message,
TextWriter log,
CancellationToken cancellationToken)
{
...
if(cancellationToken.IsCancellationRequested) return;
...
}
I was recently trying to figure out how to do this without the
WebJobs SDK which contains the WebJobShutdownWatcher, this is what I
found out.
What the underlying runtime does (and what the WebJobsShutdownWatcher referenced above checks), is create a local file at the location specified by the environment variable %WEBJOBS_SHUTDOWN_FILE%. If this file exists, it is essentially the runtime's signal to the webjob that it must shutdown within a configurable wait period (default of 5 seconds for continuous jobs, 30 for triggered jobs), otherwise the runtime will kill the job.
The net effect is, if you are not using the Azure WebJobs SDK, which contains the WebJobsShutdownWatcher as described above, you can still achieve graceful shutdown of your Azure Web Job by monitoring for the shutdown file on an interval shorter than the configured wait period.
Additional details, including how to configure the wait period, are described here: https://github.com/projectkudu/kudu/wiki/WebJobs#graceful-shutdown
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.
I've configured the Azure Diagnostics so that the logs get uploaded to a storage table. I'm using Trace.TraceXxx from my code and all works well.
Now I'm trying to add tracing from the Role OnStart() and OnStop() methods. I know that the tracing works as I see the lines in the Debug window when running in the emulator. But from the cloud deployment, it seems that these trace lines never get uploaded to the table. My guess is that it is somewhat related to TraceSources, as the only trace lines I've in the table come from the w3wp.exe source... Any hint ?
Thanks
Like you said you can add the trace listener using the WaIISHost.exe.config, but besides that you can also add the trace listener in code (you'll need a reference to Microsoft.WindowsAzure.Diagnostics.dll):
public class WebRole : RoleEntryPoint
{
public override void Run()
{
var listener = new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener();
Trace.Listeners.Add(listener);
...
}
}
Another way of setting up diagnostics is through the configuration file. If you created a VS solution recently, it will automatically create the diagnostics plug-in and configuration for the trace listener. With the config file (diagnostics.wadcfg) there is no code that needs to be written for the different data sources. Here is a link where you can get started and a sample file:
http://msdn.microsoft.com/en-us/library/gg604918.aspx
You cannot include custom performance counters right now and you need to make sure that you don’t try to allocate more than 4GB of buffer to anything (you can leave at 0), or it tends to fail.
Note, the time interval format (e.g PT1M). That is a serialization format, so PTXM is X minutes, while PTXS is X in seconds. You need to mark this as content and copy always in Visual Studio (place at root of project) so it gets packaged.
And here is a link to the three ways to setup diagnostics
http://msdn.microsoft.com/en-us/library/windowsazure/hh411541.aspx
Ranjith
http://www.opstera.com