I am continually running a few server scripts (on different ports) with nodejs using forever.
There is a considerable amount of traffic on some of these servers. The console.log commands I have for tracking connection anomalies result in bloated log files that I don't need all of the time - only for debugging. I have been manually stopping the scripts late at night, truncating the files, and restarting them. This won't do for long term, so we decided to find a solution.
Someone else on my system deleted the log files I had set up for each of the servers without my knowledge. Calling forever list on the command line shows that the server scripts are still running but now I can't tail the log files to see how the nodes are doing.
Node downtime should be kept to a bare minimum, so I'm hesitant to stop the servers during daylight hours for longer than a few minutes. Initial testing from the client side seems to indicate that the scripts are doing fine, but I can't be 100% sure there are no errors due to failed attempts at logging to a nonexistent file.
I have a few questions actually:
Is it ok to keep forever running like this?
If not, is there a proper way to disable logging? The github repository seems to indicate that forever will still log to a default file, which I don't want. Otherwise I may just write a cronjob that periodically stops scripts, truncates logs, then restarts the scripts.
What happens if I just create the logfile again with something like touch logfile_name.log while the script is still running - will this make forever freak out or is this a plausible solution?
Thanks for your time.
according to https://github.com/foreverjs/forever, try to pass -s to silent all log.
forever -s start YOURSCRIPT
Surely, before doing this, try to update forever to the latest:
sudo curl -L https://npmjs.com/install.sh | sudo sh
sudo npm update -g.
1) Just build in a periodic function or admin option to clear the forever logs. From the manual forever cleanlogs
2) At least for linux. Send each log file to /dev/null. Each log type is specified by options -l -o and -r. The -a option for append log, will stop it complaining about the log already existing.
forever start -a -l /dev/null -o /dev/null -r /dev/null your-server.js
Perhaps employ your own logging system, I use log4js, it doesn't complain if I delete the log file while the node process is still running.
There's a nifty tool that can help you that called logrotate. Have a look here
Especially the copytruncate option, it is very useful in your case.
Related
I was wondering, is there any way I can check out the logs of the my nodejs server when I've forever'd it using forever start server.js to look at what's being logged, check any live errors and such.
I looked at their documentation but couldn't find anything related to this. I want to be able to look at the console.
As forever start an application on background it gets hard to access old logs.
If you can use a library for maintaining logs, I will suggest simple-node-logger . Using this library you can save all your logs on a file and can access that file any time using:
tail -f logsFile.txt
Using this command this will keep printing new changes over logsFile.txt as logger keep updating logs.
This will help you to maintain all the logs for you.
First find out where your logs are stored using:
sudo forever list
for each process you can see something like this as logfile:
/root/.forever/_Jht.log
use 'tail -f' to monitor its changes:
sudo tail -f /root/.forever/_Jht.log
You can also set your desired path for log files, see this:
https://stackoverflow.com/a/21021251/1119611
So, I can get the directory of dump.rdb's location to change by using the dir option in redis.conf when I start it normally (just calling redis-server). If I want redis-server to run all of the time (I do) without needing a terminal window always open, I think I need to daemonize it. However, it doesn't seem this ever persists to the disk automatically and whenever the redis-server process ends (I've been ending it in testing by just running redis-cli shutdown or sometimes just killing the process with kill PID) and starts back up, all database changes are lost, which seems pretty bad if a crash or unexpected shutdown were to happen in the future. In the code that runs the processing of data (either python with redis-py or java with jedis), I can explicitly run bgsave(), but that saves dump.rdb in the directory that the code was run in and not where the dir option specifies in redis.conf
So, is there either another way to run redis-server without requiring a whole terminal window to stay open that allows what I want to do or is there a way to get the data to persist on disk in the proper directory when it's run as redis-server --daemonize yes or similar?
You could put it on linux "background" using nohup. It does not need a terminal window to stay up and running. I don't know the daemonize option to give you an advice about that, but, see if it works for you:
nohup redis-server &> redis.log&
or
Set daemonize yes in the conf file and run:
redis-server path/to/redis.conf
I'm running node app under forever in a docker container, how do I get all console.log output to show up on the console? I'm a beginner to docker so I assume the best way to log with docker is to simply log to stdout. If there's a better more proper way to do this I'm open to alternative solutions.
forever, like standard node, simply logs to stdout as well so you shouldn't have to do anything special. Of course, if you start your container with docker run -d you'll have to run docker logs -f myNodeContainer to actually see the output live.
I just came across this question because I also installed forever for the first time, ran it with just forever server.js and...nothing.
No node console messages, just the warnings about no --minUptime and no --spinSleepTime. I stopped the forever processes using forever stopall and went looking for the logs. Sure enough, in the forever log was all the stdout from my server.
I assumed I was missing an option to print to stdout, but apparently not. After reading this SO question, I launch a fresh terminal, run it again and it works - it now logs my server messages to the console.
Since then I have also experienced lots of strange issues with forever. Like it restarting the script continuously, even though the script fails to start within 1000ms (there was a syntax error in one of the files) - I stop it, run forever again (same options, same source files) and it does what it's supposed to (only runs the script once).
Not really an answer (sorry), but this is just in case anyone comes seeking information.
What are the steps required for setting up a Node.js application on Webfaction shared hosting account?
Introduction
The result of the following instructions is a Node.js application that is running continously, produces logfiles and restarts itself even after the reboot of the server machine. The instructions are especially targeted for the shared host accounts on Webfaction but can be used for general purpose also.
Install Node.js
Eventhough Node.js download page offers Linux binaries, it would be more robust to install Node.js from source. Download the most recent source codes and extract them to ~/src/.
(log in to your webfaction console)
> mkdir -p ~/src/
> cd src/
> wget http://nodejs.org/dist/v0.10.18/node-v0.10.18.tar.gz
> tar -xzf node-v0.10.18.tar.gz
The next step is to compile and install the code. For this Python is used. Because the default python version is 2.4 (at least on my Webfaction server web223) and because the Node.js installation requires one of the versions 2.6+, you must temporarily set newer version to be the default. See the following snippet (also see this article for details). Note --prefix=$HOME which is required due Webfaction's environment restrictions (you have access only to your home directory).
> cd node-v0.10.18/
> alias python=python2.7
> python configure --prefix=$HOME
> make
> make install
Node.js installed. Verify the success by
> node -v
v0.10.18
That also installed node package manager npm.
> npm -v
1.3.8
Install Forever
To keep run Node.js application running as long as possible and logging the output for maintenance, you need Forever. For convenience, install it globally (for you) using flag -g.
> cd ~
> npm install -g forever
> forever --version
v0.10.8
You can later update forever by
> npm update -g forever
Start the Application
First, create a custom application via Webfaction Control Panel (Custom app (listening on port)). Name it for example foonode. You may also create a domain and website for the app.
Second, make a note about the socket port number that was given for the app by Webfaction. Let the example be 48888.
Third, copy your application code to the directory of the app i.e. ~/webapps/foonode/. After that the directory contents would for example be the following.
> cd ~/webapps/foonode/
> ls
node_modules/
index.js
Index.js would be something like the snippet below. The key point is using the given port number for the app.
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello Universe.\n');
}).listen(48888, '127.0.0.1');
console.log('Listening at http://127.0.0.1:48888/');
Now you could run the app by node index.js. However there is a problem: when you exit the shell, also the app exits. Therefore you should run the Node.js as a daemon. That is something where Forever does a lot for you. Start node with forever:
> cd ~/webapps/foonode/
> forever start index.js
Now the app continues running after you logout and, as a very nice thing to have, is quickly restarted if it happens to crash. This is a lot but still not enough. What if the server reboots? Where is the output from the app going? What if the script crashes continously? What if you have multiple scripts with the name index.js? Keep reading.
Use Absolute Paths with Forever
Forever identifies the process by the filename of the script fed to forever start <filename_of_script>. A problem arises when you have multiple applications with the same filename for the script.
For example consider you have two applications /home/foouser/webapps/foonode/index.js and /home/foouser/webapps/barnode/index.js. If you now start both with forever start index.js within the directories of the applications and then run forever stop index.js only once, the result is that both applications become stopped. This happens because they both had the same identity index.js
The solution is to use the absolute filepaths when specifying the script.
> forever start /home/foouser/webapps/foonode/index.js
And to stop execution:
> forever stop /home/foouser/webapps/foonode/index.js
This ensures that only the intended application becomes stopped. This habit of using absolute paths has also positive effects to forever list. See this issue for details.
Logging
To store the application output to somewhere, specify the following arguments.
-l <logfile> = stream all output to this file. Relative to ~/.forever/
-o <logfile> = stream script's stdout to this file. Relative to current dir.
-e <logfile> = stream script's stderr to this file. Relative to current dir.
-a = append to the files instead of overwriting them.
It seems convenient to have a subdirectory for the logs, e.g. logs/. With the arguments, absolute path and the subdirectory logs/ the command becomes the following:
> cd ~/webapps/foonode/
> mkdir -p logs/
> forever start -a -l forever.log -o logs/out.log -e logs/err.log /home/foouser/webapps/foonode/index.js
Detecting and Restarting a Spinning Process
Forever has the parameters --minUptime and --spinSleepTime which are not so well documented currently. The meaning of the parameters is to control situation where the script crashes and almost immediately crashes again after restart. As forever tries to restart the script as soon as possible after the crash, this can lead to a busy loop which may eat lots of resources of the server.
--minUptime specifies the number of milliseconds the script has to be up and running without crashes to be restarted immediately. If the uptime of the crashed script is less than minUptime, then the script is considered spinning i.e. problematic. If uptime is greater then the script is considered non-spinning.
If a spinning script crashes i.e. the uptime of the script is less than --minUptime then forever waits --spinSleepTime number of milliseconds before restarting the script. Otherwise the script is restarted as soon as possible. This prevents the resource-eating loop. See this answer for futher discussion.
I personally use 5000 for --minUptime to make sure that Node is fully started before declaring it non-spinning. 2000 would be a good one for --spinSleepTime to avoid the loop but still trying to start the script quickly after the problematic situation resolves.
> forever start -a -l forever.log -o logs/out.log -e logs/err.log --minUptime=5000 --spinSleepTime=2000 /home/foouser/webapps/foonode/index.js
Manage the Application
As the commands grow, memorizing and writing them becomes more and more cumbersome. Therefore it is convenient to copy them into a Makefile. The makefile also becomes handy later on when you need to specify a command to run after the server reboot.
> cd ~/webapps/foonode/
> cat Makefile
foreverstart:
# Run server forever (until reboot)
mkdir -p logs
forever start -a -l forever.log -o logs/out.log -e logs/err.log --minUptime 5000 --spinSleepTime 2000 /home/foouser/webapps/foonode/index.js
foreverstop:
forever stop /home/foouser/webapps/foonode/index.js
Keeping the Application Running
Forever does not cover the case where the server reboots. For that you need to specify a cronjob with #reboot rule that start the forever after reboot.
> export EDITOR=nano # Editor to use to edit crontab. A matter of taste.
> crontab -e
Add the following line and save.
#reboot make -C ~/webapps/foonode/ -f ~/webapps/foonode/Makefile foreverstart
The line ensures that foonode is started immediately after the server reboot. The flag -C specifies the directory to run the makefile from and -f the actual location of the makefile. See this answer for details about using #reboot.
Conclusion
Now you have a node process running truly forever or at least as long as there is maintenance guys feeding the server with electricity. Lots of things done but maybe more will come. Things you may like to do in the future which were not covered here includes the following.
Watching for file changes and automatically restarting when detected (See --watch in forever --help)
Using Grunt.js instead of Make (See bustardcelly/grunt-forever)
Backupping the application once in a while.
See also
Setting up Redis on Webfaction
Any ideas, comments or corrections?
Well, you probably already followed all the steps in the other answer (wow! such detail! great), but it is now a lot easier: they now have a one-click installer for node.js.
Another option is to install nvm (Node Version Manager) and type:
nvm install node
It's not only valid for Webfaction.
We are finishing development of a project, the client is already using it but occasionally some errors occur - crashing the server.
I know I could register a service as 'upstart' script on linux, in order to have my node service restart when it crashes.
But our server is running other stuff, so we can't restart it.
Well, actually, while writing, I realize I have two questions then:
Will 'upstart' work without having to reboot? Something is just whispering yes to me :)
If not, what other option would I have to 'respawn' my node server when it crashes?
Yes, upstart will restart your process without a reboot.
Also, you should look into forever.
PM2 is a Production process manager for Node.js app.
If your focus for automatic restart is an always running application, I suggest to use a process manager. Process manager, in general, handles the node process(es if cluster enabled), and is responsible for the process/es execution. PM leans on the operative system: your node app and the OS are not so strinctly chained because the pm is in the middle.Final trick: put the process manager on upstart. Here is a complete performance improvement path to follow.
Using a shared server and not having root privileges, I can't download or install any of the previously mentioned libraries. What I am able to do is use a simple infinite bash loop to solve my issue. First, I created the file ./startup.sh in the base directory ($ vim startup.sh):
#!/bin/bash
while:
do
node ./dist/sophisticatedPrimate/server/main.js
done
Then I run it with:
$ bash startup.sh
and it works fine. There is a downside to this, which is that is doesn't have a graceful way to end the loop (at least not once I exit the server). What I ended up doing is simply finding the process with:
$ ps aux | grep startup.sh
Then killing it with
$ kill <process id>
example
$ kill 555555