Running several node.js scripts on Heroku - node.js

I have several .js scripts, is there a way to have them run all at the same time on the same Heroku app?
My folder looks like this:
**MAIN_FOLDER**
Procfile
script1.js
script2.js
script3.js
script4.js
script5.js
script6.js
script7.js
script8.js
script9.js
My Procfile:
worker: node script1.js
worker: node script2.js
...
worker: node script8.js
worker: node script9.js
And my package.json:
"scripts": {
"script1": "node script1.js",
"script2": "node script2.js",
"script3": "node script3.js",
"script4": "node script4.js",
"script5": "node script5.js",
"script6": "node script6.js",
"script7": "node script7.js",
"script8": "node script8.js",
"script9": "node script9.js",
"start": "npm-run-all --parallel script1 script2 script3 script4 script5 script6 script7 script8 script9"
}

There are several issues here. First of all, do you want to run each of your node scripts on a separate Heroku dyno, or is it sufficient for you to run your node scripts in parallel on one dyno?
If one dyno is enough for you, then I think you should change your Procfile to read:
worker: npm start
(I am assuming that npm-run-all that you are using in your "npm start" script appears in your package.json dependencies, otherwise that won't work).
Also, it seems you don't want a web dyno in your app (i.e. you aren't processing incoming HTTP/S traffic). If so, you need to explicitly scale your app's formation with something like heroku scale web=0 worker=1.
Note that you should only do that if you actually need your worker dyno to be "always on". If however all you need is for your scripts to do some work and then to exit, you should use one-off dynos instead. In that case, heroku scale your worker to 0, and then you can launch it as a one-off dyno from your command line with heroku run worker.
If however your intent is for your node scripts to run in parallel on different dyno instances, then first of all you need to be aware of Dyno scaling limits. Note that "applications using a free dyno type are limited to a maximum of two concurrent running dynos", so that would seem to prevent you from running 9 dynos in parallel with free dynos. You could get slightly more free concurrent dynos by using one-off dynos, but still not enough to run 9 concurrent node scripts.
If you are using paid dynos, then one way to do what you want would be to modify your Procfile as follows:
worker1: node script1.js
worker2: node script2.js
...
worker8: node script8.js
worker9: node script9.js
Then, use heroku scale to scale each of your worker* Process Types to 1.
Note however that you should ONLY be doing that if you need your 9 node scripts to be "always on". If your scripts simply need to do some work and then exit, you should use one-off dynos instead. Otherwise you will be paying for ALOT of unused resources!! In that case, you could use heroku scale to scale all your worker* process types to 0, and then launch your 9 scripts from your command line to run on separate one-off dynos in parallel with something like:
heroku run worker1 &
heroku run worker2 &
...
heroku run worker9 &

Related

What are the differences between the processes run by npm command:

Recently I made a simple API server using node.js+express. And the script below is a part of my package.json file I use to run with npm commands.
"scripts": {
...
"release": "cross-env NODE_ENV=production MODE=release node server/app.js",
}
After I starts the server with npm run release, I can see multiple processes such as below are running on my Linux server.
/bin/sh /api/node_modules/.bin/cross-env NODE_ENV=development MODE=test node server/app.js
node /api/node_modules/.bin/../cross-env/bin/cross-env.js NODE_ENV=development MODE=test node server/app.js
node server/app.js
I read a related documentation here, but I don't understand what actually happens in background.
What is the order of creating processes? npm => /bin/sh => node /api/.. => node server/app.js ?
What does each process do? All three processes are necessary to run my server?
If I wants to kill the server with pid, which process id should I use?
What is the order of creating processes? npm => /bin/sh => node /api/.. => node server/app.js ?
What does each process do? All three processes are necessary to run my server?
Well, the flow is like this:
NPM is spawned (you run it) inside your shell, npm itself runs with NPX in order to set the local path.
Your npm script spawns a process from a package called cross-env for cross-OS environment variable setting.
That process in turn spawns Node.js (after setting the environment variables)
That's why you see 3 processes. After your server itself run - only the actual server process is needed to run the server.
If I wants to kill the server with pid, which process id should I use?
This one: node server/app.js - since that's your actual server, the others are just "utility processes" (one for the npm script you ran and the other for the environment variables").
It's worth mentioning that in general - servers are run inside containers or other orchestrators/managers that have built-in logic for restarting/killing the process. Typically the orchestrator sends SIGTERM to the process.

Integrating PM2 on Google Cloud app engine

I am trying to integrate PM2 to Google Cloud App Engine but I just couldn't work it around. I am using PM2 for my site's staging site and I am very impressed with it. I use Digital Ocean droplet for staging. I realized that Google Cloud App Engine is not that flexible.
This is my package.json:
"main": "server.js",
"scripts": {
"start": "NODE_ENV=production npm run server:prod",
"server:prod": "node server.js",
"server:stage": "NODE_ENV=stage pm2 start server.js --exp-backoff-restart-delay=100 -i max",
"dev": "nodemon server.js",
"gcp-deploy-stage": "gcloud app deploy app.backend.stage.yaml --project=xxxxx",
"gcp-deploy-prod": "gcloud app deploy app.backend.prod.yaml --project=xxxx -v=alpha-16"
},
When I set production script start as staging like this:
"server:prod":"pm2 start server.js --exp-backoff-restart-delay=100 -i max"
and deploy this Google Cloud App Engine normally crashes because there is no global PM2 installed via NPM to start PM2.
Is there anybody went through this and made it work? Or any piece of code or any documentation that could lead me to the right solution?
Or the only option is migrating this to Google Cloud Compute Engine?
Thank you for reading this and your help.
If you want to use any module, you're going to have to include it in your package.json. Have you tried running npm install --save-dev pm2, and then redeploying your site? My guess is, that's going to get you where you want to go.
All of that aside - this probably isn't a good idea :) pm2 does a lot to manage processes on the machine, specifically dealing with crashes. App Engine Flexible does a lot of this at the infrastructure layer, automatically looking at instance health. It uses docker under the hood, which has it's own restart strategy. And then on top of that, if the the docker retry strategy doesn't work, the Google Load Balancer kicks in and will start a new compute instance for you. It's entirely possible doing process level monitoring and restarting like this will work, I just want to make sure you understand everything else that's happening under the hood.
Curiosity killed the cat - why did you end up going with App Engine Flexible over App Engine Standard?

Running forever script from Google Cloud Platform App Engine startup script

I have edited the startup-script variable for one of my instances running on the Google Cloud Platform App Engine. I'd like it to call a forever script to make sure my node app is running. So I added:
cd /opt/bitnami/apps/myapp
forever start --workingDir /opt/bitnami/apps/myapp/ --sourceDir /opt
/bitnami/apps/myapp/ app.js
after the #!/bin/bash line (also tried without the cd as it's not really necessary based on my command). But once the vm is started, running a forever list doesn't list my forever task as having ever started. If I copy and paste that forever command into a gcloud terminal and run, the task shows up fine and my app starts no problem.
Am I not calling this correctly somehow within the bash script?
The simple answer is that GAE does this by default. No need for forever or PM2. There are certain health checks that GAE does on the Docker container holding your app, and if they do not pass the instance is automatically restarted
If you want granular control over these checks (called Legacy Health Checks) you can add this to your app.yaml file:
health_check:
enable_health_check: True
check_interval_sec: 5
timeout_sec: 4
unhealthy_threshold: 2
healthy_threshold: 2
There is also updated mechanisms (called Updated Health Checks) that are still in beta, but can be used instead
The proper way to start your nodejs app on appengine is to specify the "scripts" field in your package.json, as the documentation
Below is an example borrowed from this sample
"scripts": {
"start": "node ./bin/www",
"test": "cd ..; npm run t -- appengine/analytics/test/*.test.js"
},
If you however, are only interested in running a node script, and not interested in the features that come with Google app engine, then you may simply run it on a Google Compute Engine instance.

Run node server and webpack together using package.json

I have completed todo app by learning from this video:
Super MEAN Stack Tutorial: Angular, Node/Express, Webpack, MongoDB, SASS, Babel/ES6, Bootstrap
In that video at time 19:18 at this url it is taught that I should use the below two commands in seperate git-bash instances if I want to run it in windows using npm run dev:
node server
webpack-dev-server --progress --colors
But in Linux (or any other OS than windows) you can use this script:
"Scripts": {
"start": "NODE_PATH=$NODE_PATH:./src node server",
"dev": "npm start & webpack-dev-server --progress --colors"
}
So, Is there any way I can do the same in windows?
Also, In that tutorial I can see that port no. 3000 is assigned to node server, but due to using dev dependencies he runs the localhost:8080 in browser. You can see that here. After the tutorial finishes, I followed along and created that app. Now I would like to deploy it. So, I would first like to learn to run test my site in non-dev dependencies mode. i.e. when I type localhost:3000 in browser, my app should run successfully. So, can anybody explain the steps for that?
Update:
I am a newbie in node.js. I watched many videos on node and tried to learn something from that. In all the videos I see that I run node server on port no. 3000 and then I type localhost:3000 in my browser. Now lastly I watched video about mean stack in which he uses webpack. Now, I am confused. I think there are two servers running. first server is webpack's server and second server is node's server. Upto today I typed localhost:3000 in my browser because I mentioned that port 3000 will be used by node in my code. But now in the video he is running localhost:8080 in browser. It means webpack's server is used. Then what happened to node server. Why can't I just run localhost:3000? Also in the video it is explained that webpack is a dev dependency. So, I think after the app is completed and ready to be deployed, my project can be run on the node server (by making some changes to the code, I am not sure). Let's take an example. Now I don't want to deploy the app to a real server. I want the same app to run on my friend's pc. He is not a developer. So, he should not depend on webpack as webpack is a dev dependency. So, he should be able to run the app on node server instead of webpack's server. So, he should type localhost:3000 instead of localhost:8080. That's what I don't understand.
Let's break this down:
If you've defined this script:
"Scripts": {
"start": "NODE_PATH=$NODE_PATH:./src node server",
"dev": "npm start & webpack-dev-server --progress --colors"
}
... then this npm command: npm run dev
... actually invokes these two actions:
a) npm start & # Runs NPM in the background
b) webpack-dev-server --progress --colors # Concurrently runs webpack in the foreground
You can accomplish the same thing in many ways using Windows, starting with a simple .bat file like this:
EXAMPLE: RunDev.bat:
start npm start
webpack-dev-server --progress --colors
=======================================================================
STRONG SUGGESTION:
Please forget about watching videos for a few moments. Try a couple of "hello world" tutorials. More importantly, play with the actual code. Try changing things in the code, and see what happens.
Forget about webpack, at least for the moment.
Think of npm as a "build tool"; not as a way to run your application. At least for a moment.
Focus on "node". Write a "node application".
Part of your "node application" will require "ExpressJS" and "Jade" (now renamed "pug" - I'm still using "Jade"). Use npm to get your ExpressJS and Jade dependencies, but stay focussed on Node.
SUGGESTED TUTORIAL:
A Simple Website in Node.js, Ben Gourley
Be sure to:
a. Download the code
b. Work through the tutorial, using the downloaded code
Please post back (a new post) with any specific questions you might have as you work through the tutorial.

Start server node.js with forever on ubuntu

I been searching alot and no result with same problem as me.
I have a Node.js application and I want to start it with forever start app.js, the process starts but no website found when i try in browser. The process is in the list when I write forever list.
Npm start works fine but I cant use nodejs/node app.js or my_file.js.. It gives no error or something just new command line with no output in terminal.
So anyone know why I cant start the app with nodejs app.js or forever start app.js .. No files works.
Thanks!
In express 4 you should write :
forever ./bin/www
And if you check your package.json file you can see :
"scripts": {
"start": "node ./bin/www"
}
It's the npm start script
Alternatively, you can try using PM2.
It does a great job at keeping your app alive, and has some really useful features such as load balancing, no downtime, and a web interface to monitor your processes.
In addition, I find it dead simple to use.

Resources