I have a process that needs to be run periodically - on a Rails app, it would be a worker process. Is there an equivalent for node.js on Heroku?
I'm currently using node-cron to run the periodic process on the same server as my web application. Issues here are:
With only 1 web process, it won't run when the server idles
It will block incoming connections while running
When scaling, the process doesn't need to be run on multiple servers
If it is the case that Heroku simply does not yet handle this, I'm interested in seeing other Node PAAS providers solution here.
you must have missed the new Heroku Cedar app that does all of what you want - read more at
http://devcenter.heroku.com/articles/node-js
Related
I'm trying to understand difference between Node Worker Threads vs Heroku Workers.
We have a single Dyno for our main API running Express.
Would it make sense to have a separate worker Dyno for our intensive tasks such as processing a large file.
worker: npm run worker
Some files we process are up to 20mb and some processes take longer than the 30s limit to run so kills the connection before it comes back.
Then could I add Node Worker Threads in the worker app to create child processes to handle the requests or is the Heroku worker enough on its own?
After digging much deeper into this and successfully implementing workers to solve the original issue, here is a summary for anyone who comes across the same scenario.
Node worker threads and Heroku workers are similar in that they intend to run code on separate threads in Node that do not block the main thread. How you use and implement them differs and depends on use case.
Node worker threads
These are the new way to create clustered environments on NODE. You can follow the NODE docs to create workers or use something like microjob to make it much easier to setup and run separate NODE threads for specific tasks.
https://github.com/wilk/microjob
This works great and will be much more efficient as they will run on separate worker threads preventing I/O blocking.
Using worker threads on Heroku on a Web process did not solve my problem as the Web process still times out after a query hits 30s.
Important difference: Heroku Workers Do not!
Heroku Workers
These are separate virtual Dyno containers on Heroku within a single App. They are separate processes that run without all the overhead the Web process runs, such as http.
Workers do not listen to HTTP requests. If you are using Express with NODE you need a web process to handle incoming http requests and then a Worker to handle the jobs.
The challenge was working out how to communicate between the web and worker processes. This is done using Redis and Bull Query together to store data and send messages between the processes.
Finally, Throng makes it easier to create a clustered environment using a Procfile, so it is ideal for use with Heroku!
Here is a perfect example that implements all of the above in a starter project that Heroku has made available.
https://devcenter.heroku.com/articles/node-redis-workers
It may make more sense for you to keep a single dyno and scale it up, which means multiple instances will be running in parallel.
See https://devcenter.heroku.com/articles/scaling
I am running a nodejs server on heroku, using the throng package to spin up workers to process api requests. I am also using more than on dyno on heroku. For cron and job processing, I use bull queue to distribute the load across servers, but I came across something I am not sure how to do in this distributed environment.
I want to have only one server execute code immediately on startup. In this case, I want to open up a change stream listener for mongodb. But I only want to do this on one worker on one server, not every server.
I am not sure how to do this running in heroku, any suggestions?
I have a node.js web application that runs on my amazon aws server using nginx and pm2. The application processes files for the user, which is done using a job system and child processes. In short, when the application starts via pm2, i create a child process for each cpu core of the server. Each child process (worker) then completes jobs from the job queue.
My question is, could i replicate this in docker or would i need to modify it somehow. One assumption i had was that i would need to create a container for the database, one container for the application, and then multiple worker containers to do the processing, so that if one crashes i just spin up another worker.
I have been doing research online, including a udemy course to get my head around this stuff, but i haven't come across an example or something i can relate to my problem/question.
Any help, reading material or suggestions would be greatly appreciated.
Containers run at the same performance level as the host OS. There is no process performance hit. I created a whitepaper with Docker and HPE on this.
You wouldn't use pm2 or nodemon, which are meant to start multiple processes of your node app and restart them if they fail. That's the job of Docker now.
If in Swarm, you'd just increase the replica count of your service to be similar to the number of CPU/threads you'd want to run at the same time in the swarm.
I don't mention the nodemon/pm2 thing for Swarm in my node-docker-good-defaults so I'll at that as an issue to update it for.
I have created a Heroku free account and deployed a Node JS application through a web dyno.
Besides that, I would like to know whether a Heroku free account would allow me to schedule another dyno to execute another Node JS project which simply writes Hello World in the console every 2 hours. Is that possible? Where can I find documentation on how to do schedule a worker and run a Node JS application?
Heroku Scheduler is what you are looking for.
This will spin up one-off dynos at whatever schedule you define, to run whatever task you define for it.
Provided you do not exceed your account's free dyno-hours allocation between your web dyno and your one-off dynos, you will be able to do this completely for free.
Environment: Windows Server 2008
I have a Node.js and a Sails.js apps. For example, my Node.js app does something like this:
http.createServer(function(req,res){...}).listen(8000);
Both are really simple apps, but I need them available at all times. I had problems creating a Windows Service, so i created a task for each app in the Task Scheduler.
They seem to be working fine except for when the apps haven't been used for over an hour or so (not sure on the exact timing). After some time when I go to localhost:8000, my Node.js app (same with my Sails.js app) responds only after about 10-20 seconds, and in less than a second in the following requests.
I am thinking about writing another task :) - a warm-up script that will send periodic requests to keep the apps running. But there's gotta be a better way! Is there a server timeout setting in Node.js/Sails.js that can be disabled/modified?