I am using Meteorjs
I have one heavy query which I need to run in background
e.g:
Meteor.users.update({}, {$set:{status: 1}});
I need this query to run in background.
For that I used worker threads.
I created a worker (when my server launches in config.js):
Meteor.startup(function () {
workerThread = Npm.require('worker_threads');
worker = new workerThread.Worker('worker.js');
});
Contents of worker.js:
Meteor.startup(function () {
Meteor.users.update({}, {$set:{status: 1}});
});
The above code gives me an error:
ReferenceError: Meteor is not defined
Does anyone have idea how can I run a background task in Meteorjs?
Related
What is Process, Worker, Thread, Task, Pool in Node.js from a programmer point of view?
I went through a lot of material, but difficult to understand quickly for a beginner programmer. Here is a quick summary
A Node.js process is created when run a Node.js program like node app.js (or the child process created through child_process or cluster modules). Each process will have its own memory and resources
Worker is a Node.js built-in module which takes your module (.js) as an input and creates worker object, inside a process, that executes asynchronously.
//app.js
//TODO add modules
const { Worker } = require('worker_threads');
//TODO wrap the below code into your code
const worker = new Worker('./task_processor.js');
const workerMaxLifetime = 10000;
//Send message to worker
worker.postMessage('Message to thread');
//Receive message from worker
worker.on('message', (message) => { console.log(' App:', message); });
//Terminate worker
setTimeout(() => { worker.terminate(); }, workerMaxLifetime);
Task is your module (.js) where you write the code to run as a Thread. Actually, we should call it 'Task Processor'
//task_processor.js
//TODO add modules
const { parentPort } = require('worker_threads');
//TODO wrap the below code into your code
//Receive message from App
parentPort.on('message', (task_input) => {
//Send message to App
parentPort.postMessage(task_input.a + task_input.b);
});
Thread is nothing but the worker in execution.
Pool is a wrapper .js file which create/terminate worker objects and facilitates communication between App and worker. Worker pool is not mandatory though most real world scenarios implements pools where worker thread concept is implemented. Example
Node.js module is a .js file
App: The main(or default) thread in a process is also referred as App
Process vs Worker: Each process will have its own memory and resources, whereas worker uses the same memory and resources of the process from which it is created.
I'm trying to learn cluster module and I come across this piece of code I just cant get my mind around it. First it fork childs with child_process module and there it use cluster.fork().process , I've used both cluster module and child_process in an express web-server separately i know cluster module works a load balancer.
But I cant get the idea of using them together. and there also something else, cluster is listening to those worker process and when ever a disconnect and possibly exit event is emitted to master it reforked a process , but here is the question lets assume email worker crashes and the master is going to fork it again how does it know it should fork email ? I mean shouldn't it pass an id which I cant see in this code.
var cluster = require("cluster");
const numCPUs = require("os").cpus().length;
if (cluster.isMaster) {
// fork child process for notif/sms/email worker
global.smsWorker = require("child_process").fork("./smsWorker");
global.emailWorker = require("child_process").fork("./emailWorker");
global.notifiWorker = require("child_process").fork("./notifWorker");
// fork application workers
for (var i = 0; i < numCPUs; i++) {
var worker = cluster.fork().process;
console.log("worker started. process id %s", worker.pid);
}
// if application worker gets disconnected, start new one.
cluster.on("disconnect", function(worker) {
console.error("Worker disconnect: " + worker.id);
var newWorker = cluster.fork().process;
console.log("Worker started. Process id %s", newWorker.pid);
});
} else {
callback(cluster);
}
but here is the question lets assume email worker crashes and the
master is going to fork it again how does it know it should fork email
? I mean shouldn't it pass an id which I cant see in this code.
The disconnect event it is listening to comes from the cluster-specific code, not a generic process listener. So, that disconnect event only fires when one of the cluster child processes exits. If you have some other child processes processing email, then when one of those crashes, it would not trigger this disconnect event. You would have to monitor that child_process yourself separately from within the code that started it.
You can see where the monitoring is for the cluster.on('disconnect', ...) event here in the cluster source code.
Also, I should mention that the cluster module is when you want pure horizontal scaling where all new processes are sharing the exact same work, each taking new incoming connections in turn. The cluster module is not for firing up a specific worker to carry out a specific task. For that, you would use either the Worker Threads module (to fire up a thread) or the child_process module (to fire up a new child process with a specific purpose)
I have an Electron app that is launched inside of my main nodejs application. The user has to launch my index.js and then, an Electron window spawn. The problem is that I can't comunicate between these two processes.
I am using spawn because fork doesn't work with Electron.
Child.send doesn't work. It does nothing.
In my index.js:
let { spawn } = require("child_process")
let electron = spawn(require("electron"), ["."], {"detached": false, "cwd": "D:\\code\\electron_app", "env": {"some": JSON.stringify(["process", "env"])})
electron.send(JSON.stringify({
message: "some message"
}))
electron.on("close", (code) =>{
process.exit(code)
})
electron.on("exit", (code) => {
process.exit(code)
})
electron.stdout.pipe(process.stdout)
electron.stdio.pipe(process.stdio)
electron.stdin.pipe(process.stdin)
electron.stderr.pipe(process.stderr)
In my main.js of my Electron app:
const {app, BrowserWindow} = require('electron')
let win = null
process.on("message", console.log)
//I haven't put all functions here
This doesn't do anything. The Electron app is launching but the message is not sent. Even no errors. I don't know if there is any other way to do it.
Spawn does not support the IPC channel (which you are attempting to use) that fork does. You will need to find another approach to communicate with the process, probably something over TCP (you could use some RPC implementation or spin up a HTTP server to listen for requests in the target process)
You can use ipc with spawn. fork does this automatically.
See here: https://github.com/juliangruber/electron-stream/blob/4fefd22553b0f2bea5bca3e2d00846b96acdcb74/index.js#L87-L89
spawn(electron, args, {
stdio: [null, null, null, 'ipc']
});
ps.send('foo')
ps.on('message', (msg) => {})
The fact is although NodeJs is spawning the electron process but once new process (electron window process in this case) spawned it's become completely stranger to parent process (NodeJs in this case) until it finishes (success or error) and return something to parent process. So don't relay on IPC.
I'm assuming the fact that as NodeJs is not good for CPU intensive tasks so you want to spawn some CPU intensive task (right now on same server but later on some server less architecture).
Simply just communicate through API end points.
Develop required REST end points in NodeJs and call same in electron window process, if both are on same server, request at localhost:port from child process.
If, above not help, Please share your problem statement on UX level (what/ how you want to show).
i have project in sails.js, i want to write a polling service that check some record in some interval and after that send email. my example code is:
module.exports.bootstrap = function(cb) {
cb();
var refresh = function() {
setTimeout(doWork, //someInterval);
};
var doWork = function() {
if (//check some condition) {
sendEmail();
}
refresh();
};
doWork();
}
i use pm2 libary and start my project with cluster mode. example code is:
pm2 start app.js -i 4
this command run app.js in cluster mode with 4 process.
the problem is my polling service run in all process because i run my polling service in config/bootstrap.js file and this is very bad.
my question is how can i run my service once in all process?
You can check if process is master and then run script only on that case.
var cluster = require('cluster');
if(cluster.isMaster) // rest of your service...
But for me... This is strange logic... You should queue your tasks to shared db, and when task is pooled remove it from it etc.
I'm currently trying to use Node.js Kue for processing jobs in a queue, but I believe I'm not doing it right.
Indeed the way I'm working now, I have two different services (which in this case I'm running with Docker Compose): one Web API built with Express with sends jobs to the queue and one processing module. The issue here is with the processing module.
I've coded it as follows:
var kue = require('kue');
var config = require('./config');
var queue = kue.createQueue({
prefix: config.redis.queuePrefix,
redis: {
port: config.redis.port,
host: config.redis.host
}
});
queue.process('jobType', function (job, done) {
// do processing here...
});
When we run this with Node, it sits there waiting for things to be placed on the queue to do the processing.
There are two issues however:
It needs that Redis be available before running this module. If we run this without Redis already available, it crashes because the host is not accessible and ends the process.
If Redis suddenly becomes unavailable, the processing module also crashes because it cannot stablish the connection and the process is killed.
How can I avoid these problems?
My guess is that I should somehow make the code "wait" for Redis, but I have no idea on how to do this.
How can this be done in this case?
You can use promise to wait until redis is loaded. Then run your module.
loadRedis().then(() => {
//load your module
})
Or you can use generator to "stop" until redis is loaded.
function*(){
const redisLoaded = yield loadRedis();
//load your module
}