At SharePoint Saturday in Lisle, IL this weekend, Robert Bogue said there's a difference between active and running workflows. I've looked on the web, but can someone clarify?
If I can have up to millions of active workflows on the server, why can I only have 15 or so running workflows per server?
Yes, there is a difference:
"Running" Workflows are all which currently are doing something (i.e. executing an activity).
"Active" Workflows are simply all which are "running" but currently are not doing anything - e.g. waiting for OnItemChanged or DelayActivity.
The key to understand this is WorkflowEventDeliveryThrottle (here for SP2007, because the documentation for 2010 doesn't exist). The standard value for this is property is 15. That means that there are only 15 concurrent workflow which can run at the same time. After this limit is reached the workflows get queued to the OWSTimer which executes the workflows after some arbitrary time (I think the workflow timer job is set to every 5 minutes).
This Throttle can be changed by using stsadm (AFAIK Powershell doesn't work - you can change the property via code of course setting SPWebService.WorkflowEventDeliveryThrottle):
stsadm -o setproperty -pn workflow-eventdelivery-throttle -pv "20"
Now the maximum number of "running" workflows (better would be "maximum number of workflow events that can be processes simultaneously") would be 20. See some other SO post where someone plays with the parameter.
There is a nice technical blog post to understand Workflow Event Processing: About the “workflow-eventdelivery-throttle” parameter.
Similar to the throttle is the WorkflowEventDeliveryBatchSize which denotes the maximum number of workflow events that are processed in a batch.
Summary:
You can have thousands of active workflows, e.g. all waiting for the workflow item to be changed. They are not running, not finished - simply active.
There is a limited number of workflow events that can be processed at the same time (you called it "running" workflows)
You could also have thousands of running workflows, e.g. all of them might get triggered by a delay activity set to 5 minutes, but only a limited number of them is running simultaneously, the rest of them gets queued for later execution.
Related
In this Node app I'm working on, it's possible for users to book appointments.
When an appointment is booked, the users will later get a reminder by mail X hours before the actual appointment.
I'm thinking about using Node-schedule for this task.
For each appointment: Set up a future Date, send the reminder mail once and the delete this particular scheduled job
But... there might be ALOT of appointments booked when the app grows, and this means there will be ALOT of Node-schedule processes simultaneously sleeping and waiting to fire...
On a regular day, lets pretend we have 180 appointments booked for the future per clients, and lets pretend the app right now has 50 clients. This gives us around 9000 scheduled tasks sleeping and waiting to fire.
Question: Is this perfectly OK? ... or will all these simultaneously scheduled task be to much/many?
Short answer: 9000 is not a lot, you're good to go. However, I would advise you to write a benchmark to see for yourself.
I checked node-schedule's source and sure enough, it delegates scheduling to setTimeout for date-based tasks. This means that your jobs are scheduled using node.js internal event loop.
This code is very efficient, as node.js is tailored to handle several thousands of requests simultaneously.
Regardless of the number of pending jobs, node.js will only care about the next task and sleep until its date (unless an async I/O interrupts it), so scheduling a task is essentially O(1).
A workflow is started upon instantiation of the the entity Hazaa. It waits for a while and then creates a new instance of Hazaa. After that, it's put to sleep as successful.
I'd expect it to fire perpetually creating a bunch of Hazaas. However, I only get 15 new ones before the procreations cease. Together with the original one that I create manually to set off the workflow-flow, there's 16 instances in total. I've tested with longer delays (up to several hours) but the behavior is consistent.
That's for CRM On-line. On premise, the behavior is similar but limited to 8 instances in grand total.
According to the harvest of links I've found, there's a setting in CRM to control the number of iterations. The problem is that my solution will be mainly deployed for on-line customers so unless I own the cloud, that's a show stopper.
I understand it's CRM protecting against the recurrence. What can I do about it?
The best solution I can think of at the moment is to set up a super workflow, firing the sub workflow 16 times. Then I'd need to have a super super workflow etc. Not a braggable in my view.
A CorrelationToken contains a counter and a one-hour "self-destruct" timer.
When the first workflow runs, a new CorrelationToken is created. The counter is set to 1 and the timer is set to one hour.
When the second workflow is started from the first workflow (even indirectly, such as in your case), this same CorrelationToken is used if its self-destruct timer has not already expired. If it has, a new CorrelationToken is created. If it hasn't, it increments the counter and resets the timer. Lather, rinse, repeat.
The second (and subsequent) workflows will only execute if the counter is 8 or less (On-Premise) or 16 or less (CRM Online)
What this really means is that in practice, if your child workflows are executing sooner than one hour apart, the CorrelationToken never gets a chance to expire, which means eventually the counter increments past the limit. It does not mean that you can execute up to 8 (or 16) of these workflows every hour.
It sounds like you already figured most of this out, but I wanted to give other readers background. So, to answer your question: if your design includes looping workflows that are executed sooner than one hour apart, you will need to consider an alternate design. It will definitely involve an external process or service.
If I'm understanding you correctly, it sounds like you're creating an infinite loop, which is why the CRM kills workflows like these, since otherwise they'll never end. On what condition would you stop making more Hazaa records? You could add a number field and increment that field on each new Hazaa and when it reaches a certain number stop the workflow.
I have an application that has to launch jobs repeatingly. But (yes, that would have been to easy without a but...) I would like users to define their backup frequency in application.
In worst case, they would have to choose between :
weekly,
daily,
every 12 hours,
every 6 hours,
hourly
In best case, they should be able to use crontab expressions (see documentation for example)
How to do this? Do I launch a job every minutes that check for last execution time, frequency and then launches another job if needed? Do I create a sort of queue that will be executed by a masterjob?
Any clues, ideas, opinions, best pratices, experiences are welcome!
EDIT : Solved this problem using Akka scheduler. Ok, this is a technical solution not a design answer but still everything works great.
Each user defined repetition is an actor that send messages every period to a new actor to execute the actual job.
There may be two ways to do this depending on your requirements/architecture:
If you can only use Play:
The user creates the job and the frequency it will run (crontab, whatever).
On saving the job, you calculate the first time it will have to be run. You then add an entry to a table JOBS with the execution time, job id, and any other information required. This is required as Play is stateless and information must be stored in the DB for later retrieval.
You have a job that queries the table for entries whose execution date is less than now. Retrieves the first, runs it, removes it from the table and adds a new entry for next execution. You should keep some execution counter so if a task fails (which means the entry is not removed from DB) it won't block execution of the other tasks by the job trying again and again.
The frequency of this job is set to run every second. That way while there is information in the table, you should execute the request around as often as they are required. As Play won't spawn a new job while the current one is working if you have enough tasks this one job will serve all. If not, it will be killed at some point and restored when required.
Of course, the crons of the users will not be too precise, as you have to account for you own cron delays plus execution delays on all the tasks in queue, which will be run sequentially. Not the best approach, unless you somehow disallow crons which run every second or more often than every minute (to be safe). Doing a check on execution time of the crons to kill them if they are over a certain amount of time would be a good idea.
If you can use more than Play:
The better alternative I believe is to use Quartz (see this) to create a future execution when the user creates the job, and reproram it once the execution is over.
There was a discussion on google-groups about it. As far as I remember you must define a job which start every 6 hours and check which backups must be done. So you must remember when the last backup job was finished and make the control yourself. I'm unsure if Quartz can handle such a requirement.
I looked in the source-code (always a good source ;-)) and found a method every, where I think this should be do what you want. How ever I'm unsure if this is a clever design, because if you have 1000 user you will have then 1000 Jobs. I'm unsure if Play was build to handle such a large number of jobs.
[Update] For cron-expressions you should have a look into JobPlugin.scheduleForCRON()
There are several ways to solve this.
If you don't have a really huge load of jobs, I'd just persist them to a table using the required flexibility. Then check all of them every hour (or the lowest interval you support) and run those eligible. Simple.
Or, if you prefer to use cron syntax anyway, just write (export) jobs to a user crontab using a wrapper which calls back to your running app, or starts the job in a standalone process if that's possible.
Is it possible to start a workflow on a document in SharePoint 2007 that begins after a certain time or on a certain date?
Thank you.
Here is an example of how you can start a workflow programmatically, it kicks the workflow off from an EventReceiver.
And here is an example of how to create a SharePoint timer job (these can be run at a certain time of day or after a period of time).
With a bit of work these two concepts should get you what you want. In your case instead of kicking the workflow off from an event receiver you will be kicking it off from the timer jobs Execute call.
I'm currently developing on a project which uses some TimerJobs. One of the jobs should check the MySites of some special users about every 2 minutes. So I create a SPMinuteSchedule object and set the BeginSecond property to 0 and the Interval property to 2. I think the use of both properties seem to be obvious but I'm not really sure how to interpret the EndSecond property.
If EndSecond is set to 30 and BeginSecond to 0, does it mean that the Timer Service will start the job some where within these 30 seconds and the job takes as long as it needs to execute its code? Or does it mean that the job can only run for 30 seconds? What happens if the code executed within the Execute() method needs more time to complete?
Whatever might be the answer, the property's name "EndSecond" was not chosen very well.
Refer to this post for more details to re-iterate below is the info extract from the Post
Notice how the schedule is set for the timer job. The SPMinuteSchedule.BeginSecond property and the SPMinuteSchedule.EndSecond property specify a start window of execution. The SharePoint Timer service starts the timer job at a random time between the BeginSecond property and the EndSecond property. This aspect of the timer service is designed for expensive jobs that execute on all servers in the farm. If all the jobs started at the same time, it could place an unwanted heavy load on the farm. The randomization helps spread the load out across the farm.