I am new to the whole javascript stack .I have been trying to learn by building a small application based on React-Express-Mongo. My application basically saves some config settings to mongo . And based on these settings the app periodically tries to fetch some values by querying and elasticsearch index.
So far i have the part to save the config settings done.
What I need to do now ,is to extract these setting from my mongo DB and schedule a job which keeps runnning periodically (the period is one of the settings) to poll my elastic index.The thing that i cannot wrap my head around ,is how do I create this scheduled job. All i have been using till now is the Express router to interact with my UI and DB.
I did some research ,would spawning a child process be the ideal way to go ahead with this ?
I would suggest you to go take a look to node-corn. Cron is a popular task scheduler on UNIX and node-cron is its implementation in node.
Basic usage - taken from the doc
var CronJob = require('cron').CronJob;
new CronJob('* * * * * *', function() {
console.log('You will see this message every second');
}, null, true, 'America/Los_Angeles');
Related
I am working on a Bot, using BotFramework-node.js hosted on Azure.
I am using four cronjobs in my bot to run different functions at different points of a day. The only issue is that the jobs are not running automatically. They will run normally if i compile/run the code, but wont if i don't compile/run it within 12 hours i.e, if i run the code manually today morning, the jobs will execute fine for today but tomorrow it won't unless i run it again.
Tried changing the modules. I have tried using node-schedule, node-cron. Now using cron
var Name = new CronJob({
cronTime:'0 15 11 * * 0-6',
onTick: function(){
//function call / another js file call
}
});
Name.start();
I believe this is more of a MongoDB question than a Meteor question, so don't get scared if you know a lot about mongo but nothing about meteor.
Running Meteor in development mode, but connecting it to an external Mongo instance instead of using Meteor's bundled one, results in the same problem. This leads me to believe this is a Mongo problem, not a Meteor problem.
The actual problem
I have a meteor project which continuosly gets data added to the database, and displays them live in the application. It works perfectly in development mode, but has strange behaviour when built and deployed to production. It works as follows:
A tiny script running separately collects broadcast UDP packages and shoves them into a mongo collection
The Meteor application then publishes a subset of this collection so the client can use it
The client subscribes and live-updates its view
The problem here is that the subscription appears to only get data about every 10 seconds, while these UDP packages arrive and gets shoved into the database several times per second. This makes the application behave weird
It is most noticeable on the collection of UDP messages, but not limited to it. It happens with every collection which is subscribed to, even those not populated by the external script
Querying the database directly, either through the mongo shell or through the application, shows that the documents are indeed added and updated as they are supposed to. The publication just fails to notice and appears to default to querying on a 10 second interval
Meteor uses oplog tailing on the MongoDB to find out when documents are added/updated/removed and update the publications based on this
Anyone with a bit more Mongo experience than me who might have a clue about what the problem is?
For reference, this is the dead simple publication function
/**
* Publishes a custom part of the collection. See {#link https://docs.meteor.com/api/collections.html#Mongo-Collection-find} for args
*
* #returns {Mongo.Cursor} A cursor to the collection
*
* #private
*/
function custom(selector = {}, options = {}) {
return udps.find(selector, options);
}
and the code subscribing to it:
Tracker.autorun(() => {
// Params for the subscription
const selector = {
"receivedOn.port": port
};
const options = {
limit,
sort: {"receivedOn.date": -1},
fields: {
"receivedOn.port": 1,
"receivedOn.date": 1
}
};
// Make the subscription
const subscription = Meteor.subscribe("udps", selector, options);
// Get the messages
const messages = udps.find(selector, options).fetch();
doStuffWith(messages); // Not actual code. Just for demonstration
});
Versions:
Development:
node 8.9.3
mongo 3.2.15
Production:
node 8.6.0
mongo 3.4.10
Meteor use two modes of operation to provide real time on top of mongodb that doesn’t have any built-in real time features. poll-and-diff and oplog-tailing
1 - Oplog-tailing
It works by reading the mongo database’s replication log that it uses to synchronize secondary databases (the ‘oplog’). This allows Meteor to deliver realtime updates across multiple hosts and scale horizontally.
It's more complicated, and provides real-time updates across multiple servers.
2 - Poll and diff
The poll-and-diff driver works by repeatedly running your query (polling) and computing the difference between new and old results (diffing). The server will re-run the query every time another client on the same server does a write that could affect the results. It will also re-run periodically to pick up changes from other servers or external processes modifying the database. Thus poll-and-diff can deliver realtime results for clients connected to the same server, but it introduces noticeable lag for external writes.
(the default is 10 seconds, and this is what you are experiencing , see attached image also ).
This may or may not be detrimental to the application UX, depending on the application (eg, bad for chat, fine for todos).
This approach is simple and and delivers easy to understand scaling characteristics. However, it does not scale well with lots of users and lots of data. Because each change causes all results to be refetched, CPU time and network bandwidth scale O(N²) with users. Meteor automatically de-duplicates identical queries, though, so if each user does the same query the results can be shared.
You can tune poll-and-diff by changing values of pollingIntervalMs and pollingThrottleMs.
You have to use disableOplog: true option to opt-out of oplog tailing on a per query basis.
Meteor.publish("udpsPub", function (selector) {
return udps.find(selector, {
disableOplog: true,
pollingThrottleMs: 10000,
pollingIntervalMs: 10000
});
});
Additional links:
https://medium.baqend.com/real-time-databases-explained-why-meteor-rethinkdb-parse-and-firebase-dont-scale-822ff87d2f87
https://blog.meteor.com/tuning-meteor-mongo-livedata-for-scalability-13fe9deb8908
How to use pollingThrottle and pollingInterval?
It's a DDP (Websocket ) heartbeat configuration.
Meteor real time communication and live updates is performed using DDP ( JSON based protocol which Meteor had implemented on top of SockJS ).
Client and server where it can change data and react to its changes.
DDP (Websocket) protocol implements so called PING/PONG messages (Heartbeats) to keep Websockets alive. The server sends a PING message to the client through the Websocket, which then replies with PONG.
By default heartbeatInterval is configure at little more than 17 seconds (17500 milliseconds).
Check here: https://github.com/meteor/meteor/blob/d6f0fdfb35989462dcc66b607aa00579fba387f6/packages/ddp-client/common/livedata_connection.js#L54
You can configure heartbeat time in milliseconds on server by using:
Meteor.server.options.heartbeatInterval = 30000;
Meteor.server.options.heartbeatTimeout = 30000;
Other Link:
https://github.com/meteor/meteor/blob/0963bda60ea5495790f8970cd520314fd9fcee05/packages/ddp/DDP.md#heartbeats
I have an app I want to run in Heroku (the free version) every 4 minutes from like 10am-10pm Monday-Friday. I used to do this using a cron job in Digital Ocean but I'm trying to migrate over to Heroku since it's free. But is this even possible? From everything I can see it looks like you can only run stuff every 10 minutes on Heroku.
It is a simple app that sends email updates every 4 minutes.
Thank you.
Heroku doesn't give you a way to run things every 4 minutes. As you mentioned, the lowest the Scheduler add-on will give you is 10 minutes.
You could setup a process running all the time, and enqueuing/processing your tasks every 4 minutes though.
As mentioned in garson's comment, node-cron can handle this very easily for you.
For example, the following code:
var CronJob = require('cron').CronJob;
new CronJob('* */4 * * * *', function() {
console.log('You will see this message every 4 minute');
}, null, true, 'America/Los_Angeles');
Will show the console message every 4 minutes.
You can run it with node index.js.
So if you put the following line in your Procfile with the name cron and deploy your app, you can do:
heroku ps:scale cron=1
And the process will indefinitely, executing your code every 4 minutes.
I want to develop a node.js program that will be executed at a specific time using a job scheduler (node-schedule).
This program is running in the background using forever (node.js module).
Here's the content of my app.js :
var schedule = require('node-schedule');
~
~
var id = request.body.id;
var scheduled = schedule.scheduledJobs;
if(scheduled[id]!=null){
//Tasks
}
else{
scheduled[id].cancel;
delete scheduled[id];
}
But if app.js is killed by any reason,
the schedule object is removed.
and sometimes app.js is restarted by forever.
how can I handle node-schedule objects ?
I have faced similar problem recently and there are two solutions:
1. Use actual cron
2. Use database
I solved my problem by using database. Each time you are creating some event save it to database. In your app.js file when the application is starting make function reading the database and creating scheduled events accordingly.
The first option is better if you do not have dynamic tasks, you do not create new tasks or if they are always the same.
I'm trying to setup a cron job with Bolt.cm via the Plesk 11.5.30 admin console.
Bolt task scheduler documentation: https://docs.bolt.cm/v20/tasks
Scheduled Tasks
I have setup crontab for the apache user as follows:
Min H DM M DW Command
0 */1 * * * /var/www/vhosts/mysite.com/httpdocs/app/nut cron
0 */1 * * * httpdocs/app/nut cron
The reason for two different commands here is that I've read in another thread troubleshooting Plesk that the command path should be relative from the domain. I assume if the first one fails, the second will still run. I also assume the above will run cron every hour.
Bolt Config
In my config file the cron_hour is set to 3am. However as the listener is set for an hourly event CRON_HOURLY I assume the setting is bypassed/ignored. Either way this setup has been running for over 48 hours and no effect has yet been witnessed.
Bolt Setup
As a test I have added the following into my Bolt extension so that it should simply send me an email when it is run (the following has been cut down to keep it short).
use \Bolt\CronEvents;
class Extension extends \Bolt\BaseExtension
{
function initialize() {
$this->app['dispatcher']->addListener(CronEvents::CRON_HOURLY, array($this, 'cronNotifier'));
}
function cronNotifier() {
// send email script...
}
}
I've tested my email script individual so I know it works. With the cron job setup and assumed to be running this function is never hit. What's more is that in the database the bolt_cron table remains to be empty.
If anyone could help suggest some things I could try to get this running it would be greatly appreciated.
THE FIX
In order to fix this issue I altered my schedule to execute every 5 minutes (quicker debug loop). I then made the following discoveries:
I had to set execute permissions on public for the nut file (so apache user could execute it)
for the server I'm using only the full path version works
Thanks #Gawain for prompting me in the right direction.
Scheduled Tasks
Yes, that looks correct. The handling of directory locations was enhanced recently, but you used to need to call nut from in the Bolt directory.
Bolt Config
cron_hour only applies to daily, and longer periods. There was a bug I fixed recently that was blocking out midnight to 3 am hourly tasks.
OK, there should be traces logged in the Bolt log about running cron events.
If you run this, do you see the hourly job trigger and your email sent?
./app/nut cron --run=cron.Hourly
If that works, then the problem is with the system cron/crontab, if editing directly were you using crontab -e?
Also if you're on an older checkout of master, you can try to prefix the crontab command section with
cd /var/www/vhosts/mysite.com/httpdocs &&
...before each invocation of ./app/nut