How to make Oban run more frequently than once in a minute? - cron

I'm using Oban for the ongoing tasks.
# .....
{Oban.Plugins.Cron,
crontab: [
{"* * * * *", MyApp.Workers.W1},
{"* * * * *", MyApp.Workers.W2},
]},
I now need to run W1 and W2 more frequently that every minute - around once in every 10...30 seconds. Since cron doesn't support higher frequency than 1/min, how would I get around this limitation? Preferably without hacks, unless absolutely necessary.
I don't consider switching from Oban to other library.

I don't believe Oban.Plugins.Cron supports granularity finer than one minute, by glancing at the current code.
One way you could do this is by having a process (like a GenServer) in your application that uses Process.send_after/3 or :timer.send_interval/2 to periodically queue the Oban jobs you want. This is essentially what Oban.Plugins.Cron is doing. You'll probably want to pay attention to making sure the jobs are unique (documentation).
Some very simplified code:
defmodule MyApp.Scheduler do
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, :no_args)
end
#impl true
def init(:no_args) do
:timer.send_interval(:timer.seconds(30), {:schedule_job, MyApp.Workers.W1, 30})
:timer.send_interval(:timer.seconds(20), {:schedule_job, MyApp.Workers.W2, 20})
{:ok, :no_state}
end
#impl true
def handle_info({:schedule_job, worker, unique_seconds}, state) do
%{my: :params}
|> Oban.Job.new(worker: worker, unique: [period: unique_seconds])
|> Oban.insert!()
{:noreply, state}
end
end

Related

PuLP solvers do not respond to options fed to them

So I've got a fairly large optimization problem and I'm trying to solve it within a sensible amount of time.
Ive set it up as:
import pulp as pl
my_problem = LpProblem("My problem",LpMinimize)
# write to problem file
my_problem.writeLP("MyProblem.lp")
And then alternatively
solver = CPLEX_CMD(timeLimit=1, gapRel=0.1)
status = my_problem .solve(solver)
solver = pl.apis.CPLEX_CMD(timeLimit=1, gapRel=0.1)
status = my_problem .solve(solver)
path_to_cplex = r'C:\Program Files\IBM\ILOG\CPLEX_Studio1210\cplex\bin\x64_win64\cplex.exe' # and yes this is the actual path on my machine
solver = pl.apis.cplex_api.CPLEX_CMD(timeLimit=1, gapRel=0.1, path=path_to_cplex)
status = my_problem .solve(solver)
solver = pl.apis.cplex_api.CPLEX_CMD(timeLimit=1, gapRel=0.1, path=path_to_cplex)
status = my_problem .solve(solver)
It runs in each case.
However, the solver does not repond to the timeLimit or gapRel instructions.
If I use timelimit it does warn this is depreciated for timeLimit. Same for fracgap: it tells me I should use relGap. So somehow I am talking to the solver.
However, nor matter what values i pick for timeLimit and relGap, it always returns the exact same answer and takes the exact same amount of time (several minutes).
Also, I have tried alternative solvers, and I cannot get any one of them to accept their variants of time limits or optimization gaps.
In each case, the problem solves and returns an status: optimal message. But it just ignores the time limit and gap instructions.
Any ideas?
out of the zoo example:
import pulp
import cplex
bus_problem = pulp.LpProblem("bus", pulp.LpMinimize)
nbBus40 = pulp.LpVariable('nbBus40', lowBound=0, cat='Integer')
nbBus30 = pulp.LpVariable('nbBus30', lowBound=0, cat='Integer')
# Objective function
bus_problem += 500 * nbBus40 + 400 * nbBus30, "cost"
# Constraints
bus_problem += 40 * nbBus40 + 30 * nbBus30 >= 300
solver = pulp.CPLEX_CMD(options=['set timelimit 40'])
bus_problem.solve(solver)
print(pulp.LpStatus[bus_problem.status])
for variable in bus_problem.variables():
print ("{} = {}".format(variable.name, variable.varValue))
Correct way to pass solver option as dictionary
pulp.CPLEX_CMD(options={'timelimit': 40})
#Alex Fleisher has it correct with pulp.CPLEX_CMD(options=['set timelimit 40']). This also works for CBC using the following syntax:
prob.solve(COIN_CMD(options=['sec 60','Presolve More','Multiple 15', 'Node DownFewest','HEUR on', 'Round On','PreProcess Aggregate','PassP 10','PassF 40','Strong 10','Cuts On', 'Gomory On', 'CutD -1', 'Branch On', 'Idiot -1', 'sprint -1','Reduce On','Two On'],msg=True)).
It is important to understand that the parameters, and associated options, are specific to a solver. PuLP seems to be calling CBC via the command line so an investigation of those things is required. Hope that helps

Execute python script every 15 mins starting from 9:30:42 till 15:00:42 every day from Mon-Fri

I need a way to execute my python script every 15 mins starting from 9:30:42 till 15:00:42 every day from Mon-Fri.
I have explored APScheduler with cron syntax but can't figure out how to code the above condition. I tried below but doesn't work (execute is my function name)
sched.add_cron_job(execute, day_of_week='mon-fri', hour='9:30:42-15:00:42', minute='*/15')
Any pointer is deeply appreciated.
As far as I can tell you won't be able to do what you want with a single job.
This is the closest I could get with one:30-59/15 9-14 * * 1-5 which equates to Every 15 minutes, minutes 30 through 59 past the hour, between 09:00 AM and 02:59 PM, Monday through Friday.
Although it isn't exactly what you wanted I hope this helps as a base.
I wrote custom code to solve my problem. Posting here in case it helps someone. Any optimisations suggestions are welcome.
The first infinite loop starts the job when the start time is hit. The 2nd infinite loop wakes up every x minutes to check if next run time has approached. If yes, it executes else goes back to sleep. If the end time for execution has reached, then it breaks out
def execute_schedule_custom():
start_time_of_day = datetime.combine(date.today(), time(9, 30, 42))
next_run_time = start_time_of_day
end_time_of_day = datetime.combine(date.today(), time(15, 0, 42))
interval = 15
sleep_secs = 60 * 5 #sleep for 5 mins
while True:
if datetime.now() >= start_time_of_day:
execute()
next_run_time = start_time_of_day + timedelta(minutes=interval)
break
while True:
if datetime.now() >= end_time_of_day:
break
elif datetime.now() >= next_run_time:
execute()
next_run_time = next_run_time + timedelta(minutes=interval)
t.sleep(sleep_secs)

distributed Tensorflow tracking timestamps for synchronization operations

I am new to TensorFlow. Currently, I am trying to evaluate the performance of distributed TensorFlow using Inception model provided by TensorFlow team.
The thing I want is to generate timestamps for some critical operations in a Parameter Server - Worker architecture, so I can measure the bottleneck (the network lag due to parameter transfer/synchronization or parameter computation cost) on replicas for one iteration (batch).
I came up with the idea of adding a customized dummy py_func operator designated of printing timestamps inside inception_distributed_train.py, with some control dependencies. Here are some pieces of code that I added:
def timer(s):
print ("-------- thread ID ", threading.current_thread().ident, ", ---- Process ID ----- ", getpid(), " ~~~~~~~~~~~~~~~ ", s, datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S.%f'))
return Falsedf
dummy1 = tf.py_func(timer, ["got gradients, before dequeues token "], tf.bool)
dummy2 = tf.py_func(timer, ["finished dequeueing the token "], tf.bool)
I modified
apply_gradients_op = opt.apply_gradients(grads, global_step=global_step)
with tf.control_dependencies([apply_gradients_op]):
train_op = tf.identity(total_loss, name='train_op')
into
with tf.control_dependencies([dummy1]):
apply_gradients_op = opt.apply_gradients(grads, global_step=global_step)
with tf.control_dependencies([apply_gradients_op]):
with tf.control_dependencies([dummy2]):
train_op = tf.identity(total_loss, name='train_op')
hoping to print the timestamps before evaluating the apply_gradient_op and after finishing evaluating the apply_gradient_op by enforcing node dependencies.
I did similar things inside sync_replicas_optimizer.apply_gradients, by adding two dummy print nodes before and after update_op:
dummy1 = py_func(timer, ["---------- before update_op "], tf_bool)
dummy2 = py_func(timer, ["---------- finished update_op "], tf_bool)
# sync_op will be assigned to the same device as the global step.
with ops.device(global_step.device), ops.name_scope(""):
with ops.control_dependencies([dummy1]):
update_op = self._opt.apply_gradients(aggregated_grads_and_vars, global_step)
# Clear all the gradients queues in case there are stale gradients.
clear_queue_ops = []
with ops.control_dependencies([update_op]):
with ops.control_dependencies([dummy2]):
for queue, dev in self._one_element_queue_list:
with ops.device(dev):
stale_grads = queue.dequeue_many(queue.size())
clear_queue_ops.append(stale_grads)
I understand that apply_gradient_op is the train_op returned by sync_replicas_optimizer.apply_gradient. And apply_gradient_op is the op to dequeue a token (global_step) from sync_queue managed by the chief worker using chief_queue_runner, so that replica can exit current batch and start a new batch.
In theory, apply_gradient_op should take some time as replica has to wait before it can dequeue the token (global_step) from sync_queue, but the print result for one replica I got, such as the time differences for executing apply_gradient_op is pretty short (~1/1000 sec) and sometimes the print output is indeterministic (especially for chief worker). Here is a snippet of the output on the workers (I am running 2 workers and 1 PS):
chief worker (worker 0) output
worker 1 output
My questions are:
1) I need to record the time TensorFlow takes to execute an op (such as train_op, apply_gradients_op, compute_gradients_op, etc.)
2) Is this the right direction to go, given my ultimate goal is to record the elapsed time for executing certain operations (such as the difference between the time a replica finishes computing gradients and the time it gets the global_step from sync_token)?
3) If this is not the way it should go, please guide me with some insights about the possible ways I could achieve my ultimate goal.
Thank you so much for reading my long long posts. as I have spent weeks working on this!

NodeJs scheduling jobs on multiple nodes

I have two nodeJs servers running behind a Load Balancer. I have some scheduled jobs that i want to run only once on any of the two instances in a distributed manner.
Which module should i use ? Will node-quartz(https://www.npmjs.com/package/node-quartz) be useful for this ?
Adding redis and using node-redlock seemed like overkill for the little caching job I needed to schedule for once a day on a single server with three Node.js processes behind a load balancer.
I discovered http://kvz.io/blog/2012/12/31/lock-your-cronjobs/ - and that led me to the concept behind Tim Kay's solo.
The concept goes like this - instead of locking on an object (only works in a single process) or using a distributed lock (needed for multiple servers), "lock" by listening on a port. All the processes on the server share the same ports. And if the process fails, it will (of course) release the port.
Note that hard-failing (no catch anywhere surrounding) or releasing the lock in catch are both OK, but neglecting to release the lock when catching exceptions around the critical section will mean that the scheduled job never executes until the locking process gets recycled for some other reason.
I'll update when I've tried to implement this.
Edit
Here's my working example of locking on a port:
multiProc.js
var net = require('net');
var server = net.createServer();
server.on('error', function () { console.log('I am process number two!'); });
server.listen({ port: 3000 },
function () { console.log('I am process number one!');
setTimeout(function () {server.close()}, 3000); });
If I run this twice within 3 seconds, here's the output from the first and second instances
first
I am process number one!
second
I am process number two!
If, on the other hand, more than 3 seconds pass between executing the two instances, both claim to be process number one.
I haven't done this before but I can see myself doing it this way.
Using any scheduler library for Node.js.
In order to achieve your goal, i would use redis for distributed lock. Before running any scheduled jobs, a worker / node will have to get the lock; do the job; and release / ack() when finishing the job (or on error).
A single server can be selected a leader by conducting a election among available instances using Zoologist package
https://www.npmjs.com/package/zoologist
Requires Zookeeper server to conduct the election
I don't know if this might help you, but still posting it here.
Usually node-schedule is used for time based schedules where you have to execute arbitrary code only once. For eg: a database read/write on next month 6:00 PM.
The following post will explain writing the scheduled Jobs which will perform certain action based on our requirement for a particular time / day instance.
For performing the above task we are going to use CRON package of node.
To add a job we need to :
1) Install Cron
npm install cron
2) Require cron 's CronJob to our project.
var CronJob = require('cron').CronJob
3) Create an instance of CronJob
var jobs = new CronJob({
cronTime: ' * * * * * *',
onTick: function () {
//perform Your action
},
start: false,
timeZone: 'Asia/Kolkata'
});
Arguments
cronTime: it takes 6 arguments namely :
1) Second - > 0 - 59
2) Minute - > 0 - 59
3) Hour - > 0 - 23
4) Day of Month - > 1 - 31
5) Months - > 0 - 11
6) Day of Week - > 0 - 6
Note: We Can define cronTime in ranges alse like * for always.
0 - 59 / 5 at every 5 minute.
onTick: The operation to perform.
Start: It takes a boolean and if true then starts the job now.
timeZone: job's timeZone
4) To start Job
jobs.start()
For Example :
var jobs = new CronJob({
cronTime: ' 00 00 0-23 * * *',
onTick: function () {
printMyName();
},
start: false,
timeZone: 'Asia/Kolkata'
});
jobs.start();
var printMyName = function () {
var date = new Date();
console.log("Hi Vipul it is ", today);
};
Hope it helps.

Mnesia pagination with fragmented table

I have a mnesia table configured as follow:
-record(space, {id, t, s, a, l}).
mnesia:create_table(space, [ {disc_only_copies, nodes()},
{frag_properties, [ {n_fragments, 400}, {n_disc_copies, 1}]},
{attributes, record_info(fields, space)}]),
I have at least 4 million records for test purposes on this table.
I have implemented something like this Pagination search in Erlang Mnesia
fetch_paged() ->
MatchSpec = {'_',[],['$_']},
{Record, Continuation} = mnesia:activity(async_dirty, fun mnesia:select/4, [space, [MatchSpec], 10000, read], mnesia_frag).
next_page(Cont) ->
mnesia:activity(async_dirty, fun() -> mnesia:select(Cont) end, mnesia_frag).
When I execute the pagination methods it brings batch between 3000 and 8000 but never 10000.
What I have to do to bring me the batches consistently?
The problem is that you expect mnesia:select/4, which is documented as:
select(Tab, MatchSpec, NObjects, Lock) -> transaction abort | {[Object],Cont} | '$end_of_table'
to get you the NObjects limit, being NObjects in your example 10,000.
But the same documentation also says:
For efficiency the NObjects is a recommendation only and the result may contain anything from an empty list to all available results.
and that's the reason you are not getting consistent batches of 10,000 records, because NObjects is not a limit but a recommendation batch size.
If you want to get your 10,000 records you won't have no other option that writing your own function, but select/4 is written in this way for optimization purposes, so most probably the code you will be written will be slower than the original code.
BTW, you can find the mnesia source code on https://github.com/erlang/otp/tree/master/lib/mnesia/src

Resources