For LivyOperator we set the following parameters:
polling_interval=60
retries_num_timeout=100
We set it up according to this documentation: https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/stable/_api/airflow/providers/apache/livy/operators/livy/index.html
But, in this configuration after 100 * 60 seconds = 6000 seconds = 1 hour 40 minutes Livy-session is interrupted, operator becomes failed, loading is interrupted. Is there any way to resove such inconsistency on Airflow/Livy side?
I want to send 2k mail in one day. My Segment has more than 50k contacts. I tried to update the limits and max contact in Corn job. But it is not working.
Here are the Cron I setted up :
1 */4 * * * /home/ubuntu/backup.sh > /home/ubuntu/backup.log 2>&1
*/5 * * * * /var/www/html/bin/console mautic:segments:update --force --batch-limit=2000 --max-contacts=2000
*/10 * * * * /var/www/html/bin/console mautic:campaigns:update --force --batch-limit=2000 --max-contacts=2000
28 6 * * * /var/www/html/bin/console mautic:campaigns:trigger --force --batch-limit=2000 --max-events=2000
28 6 * * * /var/www/html/bin/console mautic:emails:send
*/1 * * * * /var/www/html/bin/console mautic:import
Can someone help on this, as I check that the limits and max contact is not been supported by mautic in new versions..
Over here you should use a combination of both cronjob setup and also define that Emails are going out through a que and not immediately.
If you go to the Cog (Settings) on the top right of Mautic, -> Configuration -> Email Settings, on the right panel you will see the Mail Sent Settings. Look for "How should email be handled" and change it from "Send Immediately" to "Queue", then under that there is "Message limit for queue processing" and here you can define how many emails you want to send per the mautic:email:send cron is triggered.
So if you would like to only send out 2Kover an entire 24 hour period you could play around and do something like this:
Message Limit for queue processing = 3
Crontab: */2 * * * * php /var/www/mautic/bin/console mautic:email:send > /dev/null
This will do a blast of 3 emails every 2 minutes for a period of 24 hours.
So 3 * 30 * 24 = 2,160 emails in a period of 24 hours.
I am having issues using rampUser() method in my gatling script. The request is getting stuck after the following entry which had passed half way through.
Version : 3.3
================================================================================
2019-12-18 09:51:44 45s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=2 KO=0 )
> graphql / request_0 (OK=1 KO=0 )
> rest / request_0 (OK=1 KO=0 )
---- xxxSimulation ---------------------------------------------------
[##################################### ] 50%
waiting: 1 / active: 0 / done: 1
================================================================================
I am seeing the following in the log which gets repeated for ever and the log size increases
09:35:46.495 [GatlingSystem-akka.actor.default-dispatcher-2] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
09:35:47.494 [GatlingSystem-akka.actor.default-dispatcher-6] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
The above issue is happening only with rampUser and not happening with
atOnceUsers()
rampUsersPerSec()
rampConcurrentUsers()
constantConcurrentUsers()
constantUsersPerSec()
incrementUsersPerSec()
Is there a way to mimic rampUser() in some other way or is there a solution for this.
My code is very minimal
setUp(
scenarioBuilder.inject(
rampUsers(2).during(1 minutes)
)
).protocols(protocolBuilder)
I am stuck with this for some time and my earlier post with more information can be found here
Can any of the gatling experts help me on this?
Thanks for looking into it.
It seems you have slightly incorrect syntax for a rampUsers. You should try remove a . before during.
I have in my own script this code and it works fine:
setUp(userScenario.inject(
// atOnceUsers(4),
rampUsers(24) during (1 seconds))
).protocols(httpProtocol)
Also, in Gatling documentation example is also without a dot Open model:
scn.inject(
nothingFor(4 seconds), // 1
atOnceUsers(10), // 2
rampUsers(10) during (5 seconds), // HERE
constantUsersPerSec(20) during (15 seconds), // 4
constantUsersPerSec(20) during (15 seconds) randomized, // 5
rampUsersPerSec(10) to 20 during (10 minutes), // 6
rampUsersPerSec(10) to 20 during (10 minutes) randomized, // 7
heavisideUsers(1000) during (20 seconds) // 8
).protocols(httpProtocol)
)
My guess is that syntax can't be parsed, so instead 0 is substituted. (Here is example of rounding. Not applicable, but as reference: gatling-user-injection-constantuserspersec)
Also, you mentioned that others method work, could you paste working code as well?
I have two nodeJs servers running behind a Load Balancer. I have some scheduled jobs that i want to run only once on any of the two instances in a distributed manner.
Which module should i use ? Will node-quartz(https://www.npmjs.com/package/node-quartz) be useful for this ?
Adding redis and using node-redlock seemed like overkill for the little caching job I needed to schedule for once a day on a single server with three Node.js processes behind a load balancer.
I discovered http://kvz.io/blog/2012/12/31/lock-your-cronjobs/ - and that led me to the concept behind Tim Kay's solo.
The concept goes like this - instead of locking on an object (only works in a single process) or using a distributed lock (needed for multiple servers), "lock" by listening on a port. All the processes on the server share the same ports. And if the process fails, it will (of course) release the port.
Note that hard-failing (no catch anywhere surrounding) or releasing the lock in catch are both OK, but neglecting to release the lock when catching exceptions around the critical section will mean that the scheduled job never executes until the locking process gets recycled for some other reason.
I'll update when I've tried to implement this.
Edit
Here's my working example of locking on a port:
multiProc.js
var net = require('net');
var server = net.createServer();
server.on('error', function () { console.log('I am process number two!'); });
server.listen({ port: 3000 },
function () { console.log('I am process number one!');
setTimeout(function () {server.close()}, 3000); });
If I run this twice within 3 seconds, here's the output from the first and second instances
first
I am process number one!
second
I am process number two!
If, on the other hand, more than 3 seconds pass between executing the two instances, both claim to be process number one.
I haven't done this before but I can see myself doing it this way.
Using any scheduler library for Node.js.
In order to achieve your goal, i would use redis for distributed lock. Before running any scheduled jobs, a worker / node will have to get the lock; do the job; and release / ack() when finishing the job (or on error).
A single server can be selected a leader by conducting a election among available instances using Zoologist package
https://www.npmjs.com/package/zoologist
Requires Zookeeper server to conduct the election
I don't know if this might help you, but still posting it here.
Usually node-schedule is used for time based schedules where you have to execute arbitrary code only once. For eg: a database read/write on next month 6:00 PM.
The following post will explain writing the scheduled Jobs which will perform certain action based on our requirement for a particular time / day instance.
For performing the above task we are going to use CRON package of node.
To add a job we need to :
1) Install Cron
npm install cron
2) Require cron 's CronJob to our project.
var CronJob = require('cron').CronJob
3) Create an instance of CronJob
var jobs = new CronJob({
cronTime: ' * * * * * *',
onTick: function () {
//perform Your action
},
start: false,
timeZone: 'Asia/Kolkata'
});
Arguments
cronTime: it takes 6 arguments namely :
1) Second - > 0 - 59
2) Minute - > 0 - 59
3) Hour - > 0 - 23
4) Day of Month - > 1 - 31
5) Months - > 0 - 11
6) Day of Week - > 0 - 6
Note: We Can define cronTime in ranges alse like * for always.
0 - 59 / 5 at every 5 minute.
onTick: The operation to perform.
Start: It takes a boolean and if true then starts the job now.
timeZone: job's timeZone
4) To start Job
jobs.start()
For Example :
var jobs = new CronJob({
cronTime: ' 00 00 0-23 * * *',
onTick: function () {
printMyName();
},
start: false,
timeZone: 'Asia/Kolkata'
});
jobs.start();
var printMyName = function () {
var date = new Date();
console.log("Hi Vipul it is ", today);
};
Hope it helps.
I'm using node-cron to run scheduled jobs.
I want the jobs to run every 45 minutes, but its acting strangely
Here's the pattern I'm using
'00 */45 * * * *'
I started my script at
Tue Jun 17 2014 08:17:39 GMT+0000 (GMT)
Here's are the first couple of times the job was executed
1. Tue Jun 17 2014 08:45:03 GMT+0000 (GMT)
2. Tue Jun 17 2014 09:00:01 GMT+0000 (GMT)
3. Tue Jun 17 2014 09:45:02 GMT+0000 (GMT)
This is definitely not what I expected or want.
All I want is to run the Jobs every 45 minutes.
Can anyone help me with the pattern?
Thanks :)
You're probably looking for
0 */45 * * * *
The ranges are here.
Seconds: 0-59
Minutes: 0-59
Hours: 0-23
Day of Month: 1-31
Months: 0-11
Day of Week: 0-6
I'm more familiar with cron than with node-cron, but I've taken a quick look at the documentation.
If I understand it correctly, node-cron uses a syntax similar to that used by cron, but with an additional "seconds" field. So where a cron job might have:
# min hour mday month wday command
*/15 * * * * some-command
to schedule some-command to run every 15 minutes, node-cron would use a similar syntax to specify the time to run:
'0 */15 * * * *'
(with an additional field to specify seconds), but it executes a specified JavaScript function, not an external command.
In standard cron, there is no syntax to specify running a job every 45 minutes. A specification of 0/45 * * * * would run a job twice each hour, at 0 and 45 minutes after the hour. To run a job every 45 minutes (at 00:00, 00:45, 01:30, 02:15, ..., i.e., 32 times per day) you'd have to schedule it to run every 15 minutes, and then invoke a script that checks the current time to decide whether to do anything.
Or you can write an exhaustive list of all the times you want the job to run:
0 0 * * * some-command
45 0 * * * some_command
30 1 * * * some_command
15 2 * * * some_command
# 28 lines omitted
I'd definitely want to write a script to generate this list.
(This is workable because 24 hours happens to be a multiple of 45 minutes.
You couldn't run something every 35 minutes this way.)
A similar approach should work for node-cron. Schedule the function to run every 15 minutes, and invoke a function that checks the current time to decide whether to run. For example, you can check whether the number of minutes since midnight modulo 45 is zero. (You might want to allow for a small variance in case the scheduling is not exact.)
I don't know JavaScript well enough to suggest the best way to write this function, but it should be reasonably straightforward.
Or write 32 lines to specify all the times you want it to run.
There is no direct way to do this. However, we can get the result by intercepting the schedule using a shell command within the target script.
First, run the script at every 15 minutes:
*/15 * * * * <target_script>
Then, within the target_script, place the following code before actual codes:
#!/bin/sh
# Exit except at 0:45, 1:30, 2:15, 3:00, 3:45 etc
if ! echo "((`date +%-H`*60)+`date +%-M`)/45" | bc -l | grep "\.0*$" &> /dev/null;
then exit 1;
fi
# Your actual code goes here;
I tried this string for a 45-second interval and it works well:
'*/45 * * * * *'
You need to write a script as a wrapper to decide if the actual command shall be executed at every 45 minutes. That's 0, 45, 30 (= 45 + 45 - 60), 15 (= 30 + 45 - 60), 0 (= 15 + 45 - 60). so, the minutes to run the script shall be 0,15,30,45.
The command date +%M may be helpful in the shell script.
you can use node-reel which is more readable, straight forward and awesome 😉.
const reel = require('node-reel')
reel().call(() => {
console.log(Date.now());
}).everyFortyFiveMinutes().run()
https://github.com/shakee93/node-reel
Answer: */45 * * * *
This will run every 45th minute
Have a look here for clear understanding 👇
https://crontab.guru/#/45_*_
use cron npm moduel something like this
var cronJob = require('cron').CronJob;
var job = new cronJob({
cronTime:'0 */45 * * * *',
onTick: function(){
var my_date = new Date();
var tomorrow_date = my_date.getFullYear() + "-" + ('0'+(my_date.getMonth()+1)) + "-" + (my_date.getDate()+1)
var condition = [{},{$set: {'plannedDeliveryDate' :tomorrow_date +'T00:00:00.000Z'}}]
dbQuery.updateMany(orderModel, condition, function(err, result){
if(result.nModified == result.n) console.log(err, result)
})
},
start:true,
timeZone:'Asia/Kolkata'
});
job.start();
You can refer to cronr, it supports all the macro pattern and provides online demo cronr -- online demo
Just recently (27|01|22) used Node Cron in my project
const TaskSchedulerTimeInterval = 45;
// Runs a job for every 45 mins asynchronously
const fetchYoutubeAPIScheduler = () => {
try {
nodeCron.schedule(`*/${TaskSchedulerTimeInterval} * * * *`, async () => {
console.log('Running');
})
} catch (error) {
console.log('error =', error);
}
};
Underlying Meaning (loved the explanations given by others, so referencing them together in one post) :
#sec min hour monthday month weekday
*/15 * * * * *
Above example runs every 15 secs
range of each field:
Sec: 0-59
Min: 0-59
Hour: 0-23
monthday(Day of Month): 1-31
Months: 0-11
weekDay(Day of Week): 0-6