I've seen the pm2 logs and report and there I'm getting JavaScript heap out of memory but I've increased the heap limit to 12 GB and also added GENERATE_SOURCEMAP=false into the environment variables. Locally, by just increasing the max-old space I've resolved the issue but on the server, getting the same error.
After consuming ~2.5GB of memory, my node app is getting restarted
Node: 12.22.12
RAM: 8GB
Added NODE_OPTIONS and GENERATE_SOURCEMAP=false
in Pm2 config file, there is a max memory reload settings
// ecosystem.config.js
module.exports = {
apps: [{
name: 'api',
script: 'api.js',
max_memory_restart: '10000M' // change this
}]
}
I was facing node app restart because of the pm2... Earlier, I had set max_memory_restart: '2G' so pm2 cached that value and when I changed it into the '15G' pm2 didn't get that.
Simply do pm2 kill and deploy your app.
Related
I have a node app running on Ubuntu 18.04. It was started using PM2 like so pm2 start ./bin/web_server.js
Under some circumstances the process runs out of memory throwing this error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa18150 node::Abort() [node /path/to/app/bin/web_server.js]
2: 0xa1855c node::OnFatalError(char const*, char const*) [node /path/to/app/bin/web_server.js]
...
...
So I want to increase the heap size, which seems to be pretty straitforward for a node process. All one has to do is run it with the proper parameter set. e.g. node --max-old-space-size=2048 ./bin/web_server.js
Since I am already running my process using PM2, I want to pass the max-old-space-size parameter to node using PM2's ecosystem.config.js file. So I proceeded to add this config file into the APP's root directory. The contents are here:
module.exports = {
apps : [{
name: 'IntuListAPI',
script: './bin/web_server.js',
//watch: true,
//watch_delay: 1000,
max_memory_restart: '20G',
node_args: [
"--max-old-space-size=6144"
]
}]
};
Now when I run pm2 restart web_server I expect PM2 to pick up the new config and restart my process with it. But it does not seem to work and I can't figure out why. Since it was originally started without ecosystem.config.js, is it now ignoring it?
To be sure, the process restarts just fine, it just doesn't restart with the new heap size.
After some further research, I learned that in order to accomplish this I had to delete my APP from pm2 and then restart it using the ecosystem.config.js file. These two commands is all I needed:
pm2 delete web_server
pm2 start ecosystem.config.js
The last command will start up ./bin/web_server.js and apply all the specified parameters.
Previously I have had success implementing PM2, but not currently.
My node app does run just fine if I start it manually, but nothing I do gets it to run via PM2, even though PM2 appears to be starting it up. Here's what I mean:
If I run pm2 start server/index.js, the response in the terminal reads:
$ pm2 start server/index.js
[PM2] Spawning PM2 daemon with pm2_home=c:\pm2_system\.pm2
[PM2] PM2 Successfully daemonized
[PM2] Starting D:\Program Files\nodeApps\service-management-api\server\index.js in fork_mode (1 instance)
[PM2] Done.*
Then the terminal prints out a table with App info. It doesn't look pretty pasted here so I'll list it out:
App Name: index
id: 0
version: 1.0.0
mode: fork
pid: 8984
status: online
restart: 0
update 0s
cpu: 0%
mem: 26.0 MB
user: ME
watching: disabled
It appears that the node process should be running. But if I immediately enter pm2 list it shows no processes running. If I enter pm2 stop index, it says:
$ pm2 stop index
[PM2] Spawning PM2 daemon with pm2_home=c:\pm2_system\.pm2
[PM2] PM2 Successfully daemonized
[PM2][ERROR] Process index not found
Alternatively, if I try using ecosystem.config.js in the commands, I get similar results. Here are the commands tried:
pm2 reload ecosystem.config.js
pm2 start ecosystem.config.js
Example result of running those commmands:
$ pm2 start ecosystem.config.js
[PM2] Spawning PM2 daemon with pm2_home=c:\pm2_system\.pm2
[PM2] PM2 Successfully daemonized
[PM2][WARN] Applications sm_api not running, starting...
[PM2] App [sm_api] launched (2 instances)
And CLI prints table showing two instances with status "online" and watching "enabled". And yet, app isn't running (when tested from browser) and "pm2 show " returns:
[PM2] Spawning PM2 daemon with pm2_home=c:\pm2_system\.pm2
[PM2] PM2 Successfully daemonized
[PM2][WARN] <app name> doesn't exist
Any clues what's gone awry with my pm2?
Heres my ecosystem.config.js file:
module.exports = {
apps : [{
name: 'sm_api',
script: 'server/index.js',
"log_date_format" : "YYYY-MM-DD HH:mm Z",
// Options reference: https://pm2.io/doc/en/runtime/reference/ecosystem-file/
args: 'one two',
instances: 'max',
error_file : "C:\\pm2_system\\.pm2\\logs\\sm-api-error",
out_file: "C:\\pm2_system\\.pm2\\logs\\sm-api-out",
autorestart: true,
watch: "../",
max_restarts: 10,
max_memory_restart: '1G',
env: {
NODE_ENV: 'development'
},
env_production: {
NODE_ENV: 'production'
},
exec_mode: 'cluster'
}],
};
Running in Windows Server 2012 environment. Note that I intend to add pm2-windows-service package after I get pm2 working.
Just a note for those who are saying to run this in no-daemon mode, (pm2 start --no-daemon), you should use this mode if you're trying to diagnose the reason why when you run with the daemon, your script fails.
Almost invariably, this is because you're telling PM2 to watch a dist folder or similar while it's building, so PM2 keeps restarting until it hits its limit (because files are being regenerated at pretty rapid speed) and then outputs the "errored" status.
Running in no-daemon is absolutely not recommended in production. Ensure that PM2 is running as a daemon so it can itself restart, and so it can restart your processes as a process itself. If you kill your terminal sessions or they are automatically timed out on your host, you will quickly find your service dies when that happens.
So, in short.. do it the right way and figure out what the problem is, rather than being lazy and turning PM2 into a glorified wrapper for the node binary.
Ok, I got the answer after posting an issue to the pm2 github issues page.
Sharing it here in case anyone else finds themselves in this situation:
https://github.com/Unitech/pm2/issues/4113
(Basically pm2 3.2.5 introduced a bug that causes this issue in Windows. My dev install was 3.2.4. The issue was resolved by reverting to 3.2.4 in production. Simple process, see instructions at link above.)
As a workaround I used the following:
pm2 start --no-daemon app.js
In my case (pm2 v3.2.2):
pm2 stop all //stop all
rm ./pm2/*.pid //delete all
pm2 start app.config.js
I have a node server running on pm2 which depends on some external services.
When those servers go down I pm2 starts restarting my app, but this will keep going until it clogs up my cpu and ram on the server, restarting as much as 50 times a minute.
Is there a way to limit the numbers of restarts on pm2? There is a way to restart the server when the server reaches a certain RAM memory level, so I would hope this feature I am asking for exists.
You can use combination of max_restarts and min_uptime to restrict the app to restart consecutively.
number of consecutive unstable restarts (less than 1sec interval or custom time via min_uptime) before your app is considered errored and stop being restarted
More information about max_restarts and min_uptime is available here
Use PM2 ecosystem.config.js file like this:
module.exports = {
apps : [{
name: "app",
script: "./app.js",
merge_logs: true,
max_restarts": 50, //Here you can define your max restarts
instances: "max",
max_memory_restart: "200M",
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
}
}]
}
Start your server by following command:
pm2 start ecosystem.config.js //uses variables from `env`
pm2 start ecosystem.config.js --env production //uses variables from `env_production`
For more details see below link:
PM2 Runtime | Guide | Ecosystem File
Options for config file if you have: ecosystem.config.js
{
watch_delay: 5000,
exp_backoff_restart_delay: 100,
restart_delay: 1000,
max_restarts: 2,
min_uptime: 5000,
autorestart: false,
}
Otherwise, give the same command line, as below:
pm2 start app.js --restart-delay=3000
pm2 start app.js --no-autorestart
They themselves know the issue, so see the 1st link below:
https://pm2.keymetrics.io/docs/usage/restart-strategies/
https://pm2.keymetrics.io/docs/usage/process-management/
Mine got fixed only after watch_delay: 5000, and nothing else was required.
I am using aws ubuntu instance for my application and using node js to sync data from third party apis.
I have 6 node js scripts that calls every minute to sync data by making http request for many users.
For process management I am using PM2 module of node js.This pm2 creates logs for each running file ,this files size increasing tremendously and instance space occupied by log files.
Is there any way to disable pm2 log creation?
We can user pm2 application declaration for our purpose and we can use:
Disabling logs
You can pass /dev/null to error_file or out_file to disable logs saving.
We can also set max_memory_restart parameter in process.json apps to set autorestart process on certain memory occupied by memory.example below for process.json:
{
"apps" : [{
"script": "worker.js",
"watch": true,
"max_memory_restart": "50M",
"error_file": "/dev/null"
}]
}
There is node.js app built with Typescript, so it needs to be first "compiled" to JS before it gets run. I'm planning to use NAR (https://github.com/h2non/nar) to build ready-to-deploy package to avoid fiddling with npm install and compiling it on production. I also use PM2 as process manager for node apps.
However as far as I know PM2 can only deploy from git (fetching sources and calling npm install etc. later on), but I couldnt find a way to easily deploy application that is already pre-built.
This is my deploy.yml file contained within archive that I extract with nar extract <package>:
apps:
- script: dist/app.js
merge_logs: true
name: server
instances: 1 # 0 => max, depending on CPU cores
exec_mode: cluster
node_args: --harmony --harmony_destructuring --harmony_default_parameters
log_file: deploy/logs/server.log
pid_file: deploy/pids/server.pid
source_map_support: true
env:
NODE_ENV: production
It works fine when run for the first time, but then when I try to redeploy it (replacing application content with new version) and call pm2 reload all I get errored processes saying they either cannot load ProcessManager from PM2 or cannot find my .env file (which is in place).
As soon as I kill PM2 daemon with pm2 kill and start apps again with pm2 start all deploy.yml it clicks. But this is probably not how PM2 should be used, right?
Do you have any experience with such setup and had similar issues? Or maybe can you point me to another way of running my deployment?