Priority of Sitecore scheduled jobs - multithreading

Question

I have two Sitecore agents that run as scheduled jobs. Agent A is a long-running task that has low priority. Agent B is a short-running task that has high priority. B runs with an interval that is shorter than A's interval.
The problem is that B is never run, when A is already running.
I have implemented this to be able to run agents manually inside the content editor. When I do this, I am able to run B although A is already running (even though I set them to the same thread priority in the custom dialog).
How can I specify in my configuration file that B has a higher priority than A? Or make my scheduled job setup multi-threaded so simultaneously running jobs are possible in Sitecore? Is there a standard way to do this?
I have tried something like this where I set the thread priority inside the agent implementation, but this code is never invoked in B, when A is already running. So the prioritizing should somehow be done "before" the job implementation themselves.

Answer 1

As already mentioned in the other answer, Sitecore Scheduled Tasks are run sequentially, so each agent runs only after the previous one has finished.
However, tasks defined in the Database can be run async which means you can schedule multiple tasks to run in Parallel.
You'll need to create a Command and define a Schedule:
Defining a Command
Commands are defined in the Sitecore client.
In Content Editor navigate to /sitecore/system/Tasks/Commands
Create a new item using the Command template
Specify the Type and Method fields.
Defining a Schedule
The database agent will execute the command based on the settings in the schedule.
In Content Editor navigate to /sitecore/system/Tasks/Schedules
Create a new item using the Schedule template
For the Command field select the Command item you just created
If the task applies to specific items, you can identify those items in the Items field. This field supports a couple of formats.
For the Schedule field, you identify when the task should run. The value for this field is in a pipe-separated format.
On the schedule, check the field marked Async.
You can read more about the Database Scheduler in the Sitecore Community Docs.
Note that this may lead to performance issues if you schedule too many tasks to run in parallel.
The downside to running scheduled tasks from the Database if you cannot pass in parameters like you can for tasks defined in config. If you cannot simply access config settings from code (and they need to be passed in) then for a scheduled task defined in config you could then invoke a Sitecore Job from your Scheduled Task code. Sitecore Jobs run as a thread and each job will spin up a new thread so can be run in parallel, just make sure that Job names are unique (jobs of the same name will queue up).

Answer 2

The reason is because Sitecore Scheduled Job runs in sequence. So if you have a job being executed, it will not trigger the other jobs until it finishes.
If I am not mistaken, sitecore will queued the other jobs that will need to be executed after the current running job.
Since you trigger the job using the Run Agent Tool, it will run because you are forcing it to execute. It will not check if there is another job being ran except for publishing, it will queued because it is transferring item from Source to Target database.
EDIT:
You can check the <job> from the Web.config for Sitecore v7.x or Sitecore.config for Sitecore v8.x. You will see the pipeline being used for the Job. If I am not mistaken, I think you will need to check the code for the scheduler. The namespace is Sitecore.Tasks.Scheduler, Sitecore.Kernel
Thanks

Answer 3

As you might already understand from the answer from Hishaam (not going to repeat that good info), using the Sitecore agents might not be the best solution for what you are trying to do. For a similar setup (tasks that need to perform import, export or other queued tasks on an e-commerce site) I used an external scheduling engine (in my case Hangfire which did the job fine, but you could use an other as well) that called services in my Sitecore solution. The services performed as a layer to get to Sitecore.
You can decide how far you want to go with these services (could even start new threads) but they will be able to run next to each other. This way you will not bump into issues that another process is still running. I went for an architecture where the service was very thin layer towards the real business logic.
You might need to make sure though that the code behind one service cannot be called whilst already running (I needed to do that in case of handling a queue), but those things are all possible in .net code.
I find this setup more robust, especially for tasks that are important. It's also easier to configure when the tasks need to run.

Answer 4

I ended up with the following solution after realising that it was not a good solution to run my high priority scheduled task, B, as a background agent in Sitecore. For the purpose of this answer I will now call B: ExampleJob
I created a new agent class called ExampleJobStarter. The purpose of this job was simply to start another thread that runs the actual job, ExampleJob:
public class ExampleJobStarter
{
public override void Run()
{
if (ExampleJob.RunTheExampleJob) return;
ExampleJob.RunTheExampleJob = true;
Task.Run(() => new ExampleJob().Run());
}
}
public class ExampleJob
{
public static bool RunTheExampleJob;
public override void Run()
{
while (RunTheExampleJob)
{
DoWork();
Thread.Sleep(10000);
}
}
private void DoWork()
{
... // here I perform the actual work
}
}
ExampleJobStarter was now registered in my Sitecore config file to run every 10 minutes. I also removed ExampleJob from the Sitecore config, so it will not run automatically (thus, it is no longer a Sitecore job per se). ExampleJobStarter will simply ensure that ExampleJob is running in another thread. ExampleJob itself will do its work every 10 seconds, without being interfeered by the low-priority job agent, that still run as normal background agent.
Be aware of deadlock-issues if you go down this path (not an issue for the data I am working with in this case).