I've written a SharePoint 2010 app that uses a TimerJob to trigger processing of some documents in a list. The timer is set to trigger every minute but the the processing may take more than a minute. I'm just wondering if the next trigger of the timer job will be started using a new thread, or will the timer service just wait until the first thread has completed. I have no idea how Sharepoint manages threads for TimerJobs and I can't really find any relevant information.
This is possibly a problem given that my TimerJob definition has the following:
public override void Execute(Guid contentDbId)
{
try
{
SPWebApplication webApplication = this.Parent as SPWebApplication;
SPContentDatabase contentDb = webApplication.ContentDatabases[contentDbId];
using (SPSite site = contentDb.Sites[0])
{
using (SPWeb web = site.RootWeb)
{
PRManager.TriggerProcessing(web); // ?
}
}
}
catch (Exception)
{
}
}
}
The PRManager.TriggerProcessing() is a static method, obviously, and while it does contain mechanisms to limit only one thread at a time entering the method body, I'm just wondering IF SharePoint does create multiple threads in the event that those at-minute-interval calls to execute overlap.
Well, it is not so much a "thread" thing as much as it is a "job" thing.
SharePoint stores all jobs in a database table and it uses this table to track what is running and where it is running. It has a built in synchronization engine that is responsible for making sure the jobs execute as the job instructions say.
For example take the deployment ask
which is nothing more than a job. The
deployment task only allows One job to
run for a given solution at a time.
It makes sure that all of the tasks
finish on each server in the farm
before the overall job is reported as
done.
So the answer will depend on how your job configuration properties are set. There is a property on that job that tells SharePoint to only allow one instance of that job to run at a time. So if the job is currently executing another instance of it will not be started.
Related
We have a series of 4 Service Bus queues, each queue has a web job that processes messages and passes it on to the next queue. Though we're running on a single core, each webjob is async and allows the other jobs to continue while it queries a database or endpoint.
we have set MaxConcurrentCalls = 3 in the ServiceBusConfiguration
However, now all the messages are in the final queue, it's not spinning up multiple instances of the final Web Job to process them faster and instead executing synchronously. I'd like to know how to configure my Web Jobs to run the same web job in parallel.
I notice this article from 2014 which suggests we have to implement our own parallel processing but more recent articles contradict this information saying it is supported OOTB.
Only for Continuous WebJobs is available to scale out multi instances.
Which is determining whether the program or script runs on all instances or just one instance.
The option to run on multiple instances doesn't apply to the free or shared price tiers.
In your webjob, you will find an instance of the JobHostConfiguration object. This object is used to configure the properties of your webjob.
Here is a configuration:
static void Main()
{
var config = new JobHostConfiguration();
config.UseTimers();
config.Queues.MaxDequeueCount = 2;
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(4);
config.Queues.BatchSize = 2;
var host = new JobHost(config);
host.RunAndBlock();
}
So lets break down the items into pieces:
config.UseTimers();
The config.UserTimers(); allows us to use a timer trigger in our functions.
config.Queues.MaxDequeueCount = 2;
The MaxDequeueCount is the number of times your function will try process a message if it errors out.
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(4);
MaxPollingInterval is the max amount of time the WebJob will check the queue.
If of this is not desirable you can change this setting like I have above so that the WebJob will check the queue maximum every 4 seconds.
config.Queues.BatchSize = 2;
The BatchSize property is the amount of items your WebJob will process at the same time. The items will be processed asynchronously.
So if there is 2 items in the queue they will be processed parallel. If you set this one to 1 then you are creating a Synchronous flow as it will only take one item out of the queue at a time.
For more detail, you could refer to this article to learn run webjob in parallel.
Update:
The method BeginReceiveBatch/EndReceiveBatch allows you to retrieve multiple "items" from Queue (Async) and then use AsParallel to convert the IEnumerable returned by the previous methods and process the messages in multiple threads.
var messages = await Task.Factory.FromAsync<IEnumerable<BrokeredMessage>>(Client.BeginReceiveBatch(3, null, null), Client.EndReceiveBatch);
messages.AsParallel().WithDegreeOfParallelism(3).ForAll(item =>
{
ProcessMessage(item);
});
That code retrieves 3 messages from queue and processes then in "3 threads" (Note: it is not guaranteed that it will use 3 threads, .NET will analyze the system resources and it will use up to 3 threads if necessary.)
For more details, you could refer to this case.
By setting ServiceBusConfiguration.PrefetchCount and ServiceBusConfiguration.MessageOptions.MaxConcurrentCalls, I have been able to see that a single webjob will dequeue multiple messages and process them in parallel.
Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.
So i'm trying to get a small project of mine going that I want to host on azure, it's a web app which works fine and I've recently found webjobs which I now want to use to have a task run which does data gathering and updating, which I have a Console App for.
My problem is that I can't set a schedule, since it is published to the web app which dosen't support scheduling, so I tried using the Azure Webjobs SDK and using a timer but it wont run without a AzureWebJobsStorage connection string which I cannot get since my Azure account is a Dreamspark account and I cannot make a Azure Storage Account with it.
So I was wondering if there is some way to get this webjob to run on a time somehow (every hour or so). Otherwise if I just upgraded my account to "Pay-As-You-Go"? would I still retain my free features? namely SQL Server.
Im not sure if this is the right palce to ask but I tried googling for it without success.
Update: Decided to just make the console app run oin a infinate loop and ill just monitor it through the portal, the code below is what I am using to made that loop.
class Program
{
static void Main()
{
var time = 1000 * 60 * 30;
Timer myTimer = new Timer(time);
myTimer.Start();
myTimer.Elapsed += new ElapsedEventHandler(myTimer_Elapsed);
Console.ReadLine();
}
public static void myTimer_Elapsed(object sender, ElapsedEventArgs e)
{
Functions.PullAndUpdateDatabase();
}
}
The simplest way to get your Web Job on a schedule is detailed in Amit Apple's blog titled "How to add a schedule to a triggered WebJob".
It's as simple as adding a JSON file called settings.job to your console application and in it describing the schedule you want as a cron expression like so:
{"schedule": "the schedule as a cron expression"}
For example, to run your job every 30 minutes you'd have this in your settings.job file:
{"schedule": "0 0,30 * * * *"}
Amit's blog also goes into details on how to write a cron expression.
Caveat: The scheduling mechanism used in this method is hosted on the instance where your web application is running. If your web application is not configured as Always On and is not in constant use it might be unloaded and the scheduler will then stop running.
To prevent this you will either need to set your web application to Always On or choose an alternative scheduling option - based on the Azure Scheduler service, as described in a blog post titled "Hooking up a scheduler job to a WebJob" written by David Ebbo.
I have created my first azure webjob that runs continously;
I'm not so sure this is a code issue, but for the sake of completeness here is my code:
static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Functions).GetMethod("ProcessMethod"));
host.RunAndBlock();
}
And for the function:
[NoAutomaticTrigger]
public static async Task ProcessMethod(TextWriter log)
{
log.WriteLine(DateTime.UtcNow.ToShortTimeString() + ": Started");
while (true)
{
Task.Run(() => RunAllAsync(log));
await Task.Delay(TimeSpan.FromSeconds(60));
}
log.WriteLine(DateTime.UtcNow.ToShortTimeString() + "Shutting down..");
}
Note that the async task fires off a task of its own. This was to ensure they were started quite accurately with the same interval. The job itself is to download an url, parse and input some data in a db, but that shouldnt be relevant for the multiple instance issue I am experiencing.
My problem is that once this has been running for roughly 5 minutes a second ProcessMethod is called which makes me have two sessions simoultaniously doing the same thing. The second method says it is "started from Dashboard" even though I am 100% confident I did not click anything to start it off myself.
Anyone experienced anything like it?
Change the instance count to 1 from Scale tab of WebApp in Azure portal. By default it is set to 2 instances which is causing it to run two times.
I can't explain why it's getting called twice, but I think you'd be better served with a triggered job using a CRON schedule (https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/#CreateScheduledCRON), instead of a Continuous WebJob.
Also, it doesn't seem like you are using the WebJobs SDK, so you can completely skip that. Your WebJob can be as simple as a Main that directly does the work. No JobHost, no async, and generally easier to get right.
I have a very long running query that takes too long to keep my client connected. I want to make a call into my DomainService, create a new worker thread, then return from the service so that my client can then begin polling to see if the long running query is complete.
The problem I am running into is that since my calling thread is exiting right away, I am getting exceptions thrown when my worker tries to access any entities since the ObjectContext gets disposed when the original thread ends.
Here is how I create the new context and call from my Silverlight client:
MyDomainContext context = new MyDomainContext();
context.SearchAndStore(_myParm, SearchQuery,
p => {
if (p.HasError) { // Do some work and return to start
} // polling the server for completion...
}, null);
The entry method on the server:
[Invoke]
public int SearchAndStore(object parm)
{
Thread t = new Thread(new ParameterizedThreadStart(SearchThread));
t.Start(parms);
return 0;
// Once this method returns, I get ObjectContext already Disposed Exceptions
}
Here is the WorkerProc method that gets called with the new Thread. As soon as I try to iterate through my query1 object, I get the ObjectContext already Disposed exception.
private void WorkerProc(object o)
{
HashSet<long> excludeList = new HashSet<long>();
var query1 = from doc in this.ObjectContext.Documents
join filters in this.ObjectContext.AppliedGlobalFilters
.Where(f => f.FilterId == 1)
on doc.FileExtension equals filters.FilterValue
select doc.FileId;
foreach (long fileId in query1) // Here occurs the exception because the
{ // Object Context is already disposed of.
excludeList.Add(fileId);
}
}
How can I prevent this from happening? Is there a way to create a new context for the new thread? I'm really stuck on this one.
Thanks.
Since you're using WCF RIA. I have to assume that you're implementing two parts:
A WCF Web Service
A Silverlight client which consumes the WCF Service.
So, this means that you have two applications. The service running on IIS, and the Silverlight running on the web browser. These applications have different life cycles.
The silverlight application starts living when it's loaded in the web page, and it dies when the page is closed (or an exception happens). On the other hand (at server side), the WCF Web Service life is quite sort. You application starts living when the service is requested and it dies once the request has finished.
In your case your the server request finishes when the SearchAndStore method finishes. Thus, when this particular method starts ,you create an Thread which starts running on background (in the server), and your method continues the execution, which is more likely to finishes in a couple of lines.
If I'm right, you don't need to do this. You can call your method without using a thread, in theory it does not matter if it takes awhile to respond. this is because the Silvelight application (on the client) won't be waiting. In Silverlight all the operations are asynchronous (this means that they're running in their own thread). Therefore, when you call the service method from the client, you only have to wait until the callback is invoked.
If it's really taking long time, you are more likely to look for a mechanism to keep the connection between your silverlight client and your web server alive for longer. I think by modifying the service configuration.
Here is a sample of what I'm saying:
https://github.com/hmadrigal/CodeSamples/tree/master/wcfria/SampleWebApplication01
In the sample you can see the different times on client and server side. You click the button and have to wait 30 seconds to receive a response from the server.
I hope this helps,
Best regards,
Herber