I'm thinking about making a Hubot clock-based notifier (think "It's 5, time to go home!" but less annoying and for a different reason). What's the best way to go about doing something like this in a Hubot script?
[edit] Here's an example using node-cron:
TIMEZONE = "America/New_York"
QUITTING_TIME = '0 0 17 * * 2-6' # M-F 5pm
ROOM = "Dev"
cronJob = require('cron').CronJob
module.exports = (robot) ->
gohome = new cronJob QUITTING_TIME,
->
robot.messageRoom ROOM, "It's 5! Go home!"
null
true
TIMEZONE
dependencies:
"cron": "0.3.3",
"time": "0.8.2"
I would use node-cron. It's pretty flexible and fits your use case well.
https://github.com/ncb000gt/node-cron
I just stumbled upon hubot-cron.
This lets you schedule messages in hubot, but unfortunately doesn't persist jobs through restarts.
Related
I would like to migrate my data into an other database. To do that I use Airflow to run a DAG composed of ETL flows. Each run is for one day of data and I have 3 years to catch-up. The problem is that each run has an unknown execution time and it's important to wait the end of the run before starting the next one.
I try to put a minimal schedule_interval and fix max_active_runs=1, but at the end not all data were loaded. A lot of days have jumped. I notice this Airflow comportment : at the end of the schedule interval, my program is partially executed (so my variable that give the date of the day to catch-up is incremented), then is stopped because of the max_active_runs.
with DAG(
dag_id='catch_up',
default_args={
'owner': 'airflow',
'start_date': datetime.now(),
'depends_on_past': False,
'retries': 1,
},
description=' ',
schedule_interval=*/2 * * * *,
max_active_runs=1,
catchup=False
) as dag:
date = launch_date()
start_etl = PythonOperator(
task_id='flow',
python_callable=launch_flow,
op_args=[date]
)
...
My question is how can I really make wait the end of the execution before starting an other one ?
What you are after is a combination of max_active_runs, depends_on_past & wait_for_downstream (see code base)
depends_on_past - when set to true, task instances will run
sequentially and only if the previous instance has succeeded or has been skipped. The task instance for the start_date is allowed
to run.
wait_for_downstream - when set to true, an instance of task
X will wait for tasks immediately downstream of the previous instance
of task X to finish successfully or be skipped before it runs. This is useful if the
different instances of a task X alter the same asset, and this asset
is used by tasks downstream of task X. Note that depends_on_past
is forced to True wherever wait_for_downstream is used. Also note that
only tasks immediately downstream of the previous task instance are waited
for; the statuses of any tasks further downstream are ignored.
I think having :
default_args = {
'wait_for_downstream': true,
'depends_on_past': true, // (*) see note below
...
}
with DAG(
default_args=default_args,
max_active_runs=1,
...
) as dag:
...
Will give you the logic you are after as only 1 active DagRun can be at any given time while each task must wait to the predecessor one in the previous run. On top of that if a specific task failed it will block everything and no new runs will be created till you address the issue manually.
Noting that since you didn't provide much details it may be that your use case can be solved by setting only max_active_runs=1 and having depends_on_past=True on the last task, but once you understand the parameters it's easier to find the right config for you.
(*) technically there is no need to set depends_on_past=True when setting wait_for_downstream=True because Airflow will override it for you but being explicit can help for readability.
I have a simple node.js parser that has to push data to a remote server during work hours only and sleep for the rest of the time.
Looking at the available modules, schedule and node-cron (https://github.com/ncb000gt/node-cron) seems to do part of my requirement.
I am using the PM2 module to restart the process, when it goes down
Here is what I have so far in coffee script:
runParser = (callback) ->
#...
console.log 'waking up parser...'
parseAll()
return
_jobs = [ {
name: 'Start parser'
cronTime: '00 34 16 * * 1-5'
onTick: runParser
start: true
id: 'parsedbf'
#timeZone: 'Europe/London'
} ]
_cronJobs = {}
schedule = ->
_jobs.map (job) ->
_cronJobs[job.id] = new cronJob(job)
console.log util.format('%s cronjob scheduled at %s on timezone', job.name, job.cronTime)
return
return
run = ->
start = moment('08:30','HH:mm').valueOf()
now = moment().valueOf()
end = moment('18:00','HH:mm').valueOf()
if start < now and now < end
runParser()
else
schedule(console.info 'scheduler started...')
run(console.info 'sync code statrted after a hard reboot...')
my question, how do i change the script so that at 18:30 the parser is just idle?
should i use schedule.js (http://bunkat.github.io/schedule/index.html) how do i modify the code for this?
any advice much appreciated
Is there any reason you can't just run this in cron? You have the question tagged for cron which refers to the unix utility, it was made to do this sort of thing. You could use a combination of cron and forever: one call starts it in the am and another stops the script in the evening but it runs continuously otherwise.
We are currently using watir-webdriver (0.6.2) with firefox to run acceptance tests.
Our tests take a long time to run, and often fail with timeout errors.
We wanted to decrease the timeout time, for them to fail faster.
We tried:
browser = Watir::Browser.new("firefox")
browser.driver.manage.timeouts.implicit_wait=3
However, we are still experiencing 30s timeouts.
We couldn't find any documentation or question regarding this issue. Does anyone know how to configure Watir waiting timeouts properly?
It depends exactly what you mean by 'timeout'. AFAIK there are three different definitions of timeout commonly discussed when talking about Watir-Webdriver:
How long does the browser wait for a page to load?
How long does Watir-Webdriver explicitly wait before considering an element 'not present' or 'not visible' when told to wait via the '.when_present' function
How long does Watir-Webdriver implicitly wait for an object to appear before considering an element 'not present' or 'not visible' (when not waiting via explicitly call see #2)
#1: Page load
Justin Ko is right that you can set page load timeout as described if your goal is to modify that, though it looks like the canonical way to do that is to set the client timeout before creating the browser and passing it to the browser on creation:
client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 180 # seconds – default is 60
b = Watir::Browser.new :firefox, :http_client => client
- Alistair Scott, 'How do I change the page load Timeouts in Watir-Webdriver'
#2: Explicit timeout
But I think #p0deje is right in saying you are experiencing explicit timeouts, though it's not possible to say for sure without seeing your code. In the below I experienced the explicit declaration overriding the implicit (I am unsure if that's intentional):
b = Watir::Browser.new :firefox
b.driver.manage.timeouts.implicit_wait = 3
puts Time.now #=> 2013-11-14 16:24:12 +0000
begin
browser.link(:id => 'someIdThatIsNotThere').when_present.click
rescue => e
puts e #=> timed out after 30 seconds, waiting for {:id=>"someIdThatIsNotThere", :tag_name=>"a"} to become present
end
puts Time.now #=> 2013-11-14 16:24:43 +0000
Watir-Webdriver will wait 30 seconds before failure by default thanks to 'when_present'. Alternatively you can say 'when_present(10)' to alter the default and wait 10 seconds. (Watir-Webdriver > Watir::Wait#when_present.) I can not divine any way to do this globally. Unless you find such a thing - and please tell me if you do - it must be done on each call. :( Edit: Fellow answerer Justin Ko gave me the answer as to how to do what I described above. Edit 2: #jarib added this to Watir, per #justinko in the linked answer: "Update: This monkey patch has been merged into watir-webdriver and so will no longer be needed in watir-webdriver v0.6.5. You will be able to set the timeout using: Watir.default_timeout = 90"
#3 Implicit timeout
The code you provided sets the time Watir-Webdriver will wait for any element to be come present without you explicitly saying so:
b = Watir::Browser.new :firefox
b.driver.manage.timeouts.implicit_wait = 3
puts Time.now #=> 2013-11-14 16:28:33 +0000
begin
browser.link(:id => 'someIdThatIsNotThere').when_present.click
rescue => e
puts e #=> unable to locate element, using {:id=>"someIdThatIsNotThere", :tag_name=>"a"}
end
puts Time.now #=> 2013-11-14 16:28:39 +0000
The implicit_wait is the amount of time selenium-webdriver tries to find an element before timing out. The default is 0 seconds. By changing it to "3", you are actually increasing the amount of time that it will wait.
I am guessing that you actually want to change the timeout for waiting for the page to load (rather than for finding an element). This can be done with:
browser.driver.manage.timeouts.page_load = 3
For example, we can say to only wait 0 seconds when loading Google:
require 'watir-webdriver'
browser = Watir::Browser.new :firefox
browser.driver.manage.timeouts.page_load = 0
browser.goto 'www.google.ca'
#=> Timed out waiting for page load. (Selenium::WebDriver::Error::TimeOutError)
Update: Since Watir 6.5, the default timeout is configurable using
Watir.default_timeout = 3
We experienced the same issue and chose to override Watir methods involving timeouts, namely
Watir::Wait.until { ... }
Watir::Wait.while { ... }
object.when_present.set
object.wait_until_present
object.wait_while_present
Here is the code, you can put it in your spec_helper.rb if using rspec
# method wrapping technique adapted from https://stackoverflow.com/a/4471202/177665
def override_timeout(method_name, new_timeout = 3)
if singleton_methods.include?(method_name)
old_method = singleton_class.instance_method(method_name)
define_singleton_method(method_name) do |timeout = new_timeout, *args, &block|
old_method.bind(self).(timeout, *args, &block)
end
else
old_method = instance_method(method_name)
define_method(method_name) do |timeout = new_timeout, *args, &block|
old_method.bind(self).(timeout, *args, &block)
end
end
end
# override default Watir timeout from 30 seconds to 3 seconds
module Watir
module Wait
override_timeout(:until)
override_timeout(:while)
end
module EventuallyPresent
override_timeout(:when_present, 5) # 5 secs here
override_timeout(:wait_until_present)
override_timeout(:wait_while_present)
end
end
We used answer from https://stackoverflow.com/a/4471202/177665 to get it working.
I need to run code in Node.js every 24 hours. I came across a function called setTimeout. Below is my code snippet
var et = require('elementtree');
var XML = et.XML;
var ElementTree = et.ElementTree;
var element = et.Element;
var subElement = et.SubElement;
var data='<?xml version="1.0"?><entries><entry><TenantId>12345</TenantId><ServiceName>MaaS</ServiceName><ResourceID>enAAAA</ResourceID><UsageID>550e8400-e29b-41d4-a716-446655440000</UsageID><EventType>create</EventType><category term="monitoring.entity.create"/><DataCenter>global</DataCenter><Region>global</Region><StartTime>Sun Apr 29 2012 16:37:32 GMT-0700 (PDT)</StartTime><ResourceName>entity</ResourceName></entry><entry><TenantId>44445</TenantId><ServiceName>MaaS</ServiceName><ResourceID>enAAAA</ResourceID><UsageID>550e8400-e29b-41d4-a716-fffffffff000</UsageID><EventType>update</EventType><category term="monitoring.entity.update"/><DataCenter>global</DataCenter><Region>global</Region><StartTime>Sun Apr 29 2012 16:40:32 GMT-0700 (PDT)</StartTime><ResourceName>entity</ResourceName></entry></entries>'
etree = et.parse(data);
var t = process.hrtime();
// [ 1800216, 927643717 ]
setTimeout(function () {
t = process.hrtime(t);
// [ 1, 6962306 ]
console.log(etree.findall('./entry/TenantId').length); // 2
console.log('benchmark took %d seconds and %d nanoseconds', t[0], t[1]);
//benchmark took 1 seconds and 6962306 nanoseconds
},1000);
I want to run the above code once per hour and parse the data. For my reference I had used one second as the timer value. Any idea how to proceed will be much helpful.
There are basically three ways to go
setInterval()
The setTimeout(f, n) function waits n milliseconds and calls function f.
The setInterval(f, n) function calls f every n milliseconds.
setInterval(function(){
console.log('test');
}, 60 * 60 * 1000);
This prints test every hour. You could just throw your code (except the require statements) into a setInterval(). However, that seems kind of ugly to me. I'd rather go with:
Scheduled Tasks
Most operating systems have a way of sheduling tasks. On Windows this is called "Scheduled Tasks" on Linux look for cron.
Use a libary As I realized while answering, one could even see this as a duplicate of that question.
I'm reading up on Log4perl and want to try and use it for simple log management of my Perl scripts running on a Linux box. I've also read up on newsyslog and logrotate but want to use Log4perl if at all possible.
I'm trying to configure the /etc/log4perl.conf file so that it:
Defines a widget logger (INFO level) that will write all output to /opt/myapp/logs/myapp-<datetime>.log, where <datetime> is a date/time formatted string like 2012-12-20
This myapp-<datetime>.log file needs to be rotated daily (preferably at midnight), where the old file is deleted, and a new file is created with <datetime> + 1. For instance, myapp-2012-12-20.log would be replaced with myapp-2012-12-21.log, etc.
Here's my best attempt which I believe is close, but is still missing some configuration:
#####/etc/log4perl.conf############################################################
log4perl.logger.widget = INFO, MyAppLogAppender
log4perl.appender.MyAppLogAppender = Log::Log4perl::Appender::File
log4perl.appender.MyAppLogAppender.filename = /opt/myapp/logs/myapp-???.log
log4perl.appender.MyAppLogAppender.layout = Log::Log4perl::Layout::SimpleLayout
###################################################################################
How do I configure log4perl.appender.MyAppLogAppender to rotate once a day, delete the old file, and create a new one with a correct timestamp? Thanks in advance.
Here's an example of a Log::Log4perl configuration file, defining a daily rollover at midnight (date pattern yyyy-MM-dd), keeping a maximum of 5 saved logfiles around, at WARN level, and dumping everything to screen:
log4perl.logger = TRACE, Screen, Logfile
log4perl.appender.Logfile = Log::Dispatch::FileRotate
log4perl.appender.Logfile.Threshold = WARN
log4perl.appender.Logfile.filename = test.log
log4perl.appender.Logfile.max = 5
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd
log4perl.appender.Logfile.TZ = PST
log4perl.appender.Logfile.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.Logfile.layout.ConversionPattern = %d %m %n
log4perl.appender.Screen = Log::Log4perl::Appender::Screen
log4perl.appender.Screen.stderr = 0
log4perl.appender.Screen.utf8 = 1
log4perl.appender.Screen.layout = Log::Log4perl::Layout::PatternLayout::Multiline
log4perl.appender.Screen.layout.ConversionPattern = [%p] %m %n
(reference: https://metacpan.org/module/Log::Log4perl::FAQ#How-can-I-roll-over-my-logfiles-automatically-at-midnight-)
There is a gotcha in Log::Dispatch::FileRotate, if your daily job is run later in the day (say 23:00) and takes 2h (so ends at about 01:00), the log rotation will never happens with a "day" pattern like :
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd
A simple workaround is to use an "hourly" pattern like this :
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd-HH
So when the next run starts at 23:00, the log file get rotated as more than one hour has passed.