Oh god why is this so hard. I've now spent 3 days trying to get this seemingly simple crap to work.
I need it to:
- npm install on CI server (works)
- run tests (works)
- build angular frontend (works)
- ship code to server via rsync (works)
- ssh into server (works)
- - and npm install (doesn't work. dies because of npm warnings, I think)
- - restart pm2 process (doesn't work as there's no elegant way to say start or restart)
At the deploy step, I have this script in the codeship UI
rsync -avz --exclude 'node_modules' ~/clone/
root#xxx.xxx.xxx.xxx:/root/my-project/
ssh root#xxx.xxx.xxx.xxx cd /root/my-project && bash ./postDeploy.sh
Then the postDeploy.sh script is this:
#!/bin/sh
export PATH=$PATH:/usr/local/bin
npm install --silent &> /dev/null
/usr/local/bin/pm2 stop --silent keystone &> /dev/null
/usr/local/bin/pm2 start keystone.js 2> /dev/null
I'm trying to swallow errors with this trick. &> /dev/null
There are a few vulnerabilities in the project that are unfortunately deep inside a core module and not fixable by me so I need npm to just be quiet in this case.
Then there's the PM2 thing which is slightly annoying. I need to issue a stop command, but if the service is not running it will fail, so again I need to swallow errors. The start command is probably fine.
I think maybe what's happening now is that because I swallow all output codeship's script runner assumes it fails?
I have tried to use the half-baked debug tool, but it magically asks me for a password when I try to login in... Eh?
Also #codeship it would be amazing if 80% of the helpful articles that google has indexed didn't lead to dead pages on your site...
I have tried to use the half-baked debug tool, but it magically asks me for a password when I try to login in... Eh?
I'd say this was a correct instinct. There are too many possible scenarios for why you're coming across these unintended behaviors and nothing short of running the build live with a ssh debug session will likely get to the bottom of that.
Please see our documentation section for troubleshooting password prompts for ssh debug sessions.
If an ssh debug session doesn't solve your situation, then please reach out to us at support#codeship.com with your build url and we'll take a closer look.
Related
Couldn't seem to find a direct answer around here.
I'm not sure if I should run ./myBinary as a Cron process or if I should run "go run myapp.go"
What's an effective way to make sure that it is always running?
Sorry I'm used to Apache and Nginx.
Also what are best practices for deploying a Go app? I want everything (preferably) all served on the same server. Just like how my development environment is like.
I read something else that used S3, but, I really don't want to use S3.
Use the capabilities your init process provides. You're likely running system with either Systemd or Upstart. They've both got really easy descriptions of services and can ensure your app runs with the right privileges, is restarted when anything goes down, and that the output is are handled correctly.
For quick Upstart description look here, your service description is likely to be just:
start on runlevel [2345]
stop on runlevel [!2345]
setuid the_username_your_app_runs_as
exec /path/to/your/app --options
For quick Systemd description look here, your service is likely to be just:
[Unit]
Description=Your service
[Service]
User=the_username_your_app_runs_as
ExecStart=/path/to/your/app --options
[Install]
WantedBy=multi-user.target
You can put it in an inifiny loop, such as:
#! /bin/sh
while true; do
go run myapp.go
sleep 2 # Just in case
done
Hence, once the app dies due some reason, it will be run again.
You can put it in a script and run it in background using:
$ nohup ./my-script.sh >/dev/null 2>&1 &
You may want to go for virtual terminal utility like screen here. Example:
screen -S myapp # create screen with name myapp
cd ... # to your app directory
go run myapp.go # or go install and then ./myappfrom go bin dir
Ctrl-a+d # to go out of screen
If you want to return to the screen:
screen -r myapp
EDIT: this solution will persist the process when you go out of terminal, but won't restart it when it'll crash.
I'm deploying a node web application as an upstart service using grunt and monitoring it using monit. However:
My upstart and monit configuration duplicate each other a little bit
Upstart doesn't do variable expansion inside env stanzas
I can't find a way to configure monit dynamically (ala upstart .override files)
My question
This means I'm looking for a grunt plugin or other tool that I can use to generate the uptstart .conf and monit conf.d/ files. Can you please help me find one (or suggest a better way of robustly running my node web app)?
A rather crude solution?
To be honest an underscore template of the upstart and monit files would probably be sufficient, and that's what I'll wrap up into a grunt plugin if there isn't a ready-made solution, but this feels like a problem that other people must have run into as well so I imagine there's a solution out there, I just can't find it.
Detail
A bit more detail to illustrate the three problems. My upstart conf file looks like this:
setuid node
setgid node
# ...
script
mkdir -p /home/node/.my-app
echo $$ > /home/node/.my-app/upstart.pid
/usr/local/bin/node /var/node/my-app/server.js >> /var/node/my-app/logs/console.log 2>&1
end script
# ...
And my monit configuration looks like this:
check process node with pidfile /home/node/.my-app/upstart.pid
start program = "/sbin/start my-app" with timeout 60 seconds
stop program = "/sbin/stop my-app"
if failed host localhost port 17394 protocol http
and request "/monit.html"
then restart
if 3 restarts within 5 cycles then timeout
As you can see, the PID file path and config directory is duplicated across the two (problem 1) and I'd love to parameterise the host, port and request URL in the monit file (problem 3).
For problem 2 (no variable expansion in upstart's env stanza, bug report here) there are a couple of workarounds that work for variables used inside *script blocks, which are interpreted as bash scripts, but they don't seem to work in the conf file itself. I think this makes it impossible to specify the user id the app should run as in a configuration file?
Those techniques I just mentioned:
Method 1: Don't use env - echo the variables in the pre-script to a file and then source it later
Method 2: Duplicate the variable expansion in all the script bodies where it is needed
Method 3: Store the variables in a file and cat them in using command substitution
...or suggest a better way of robustly running my node web app
Use PM2 by Unitech instead.
I run all my node apps using PM2. It works perfectly and is easy to configure. It has built-in functionality for autogenerating of startup script. It gives you logging, monitoring and easy maintenance.
Here is a good article showing off the highlights.
Install
npm install -g pm2
Start app
pm2 start app.js
Show running apps
pm2 list
Make a startup script
pm2 startup [ubuntu|centos|systemd]
More details in the readme on their github page
I have some cronjobs that I always used and worked fine. But now, trying to move everything to Docker containers, I'm running into these errors:
/usr/bin/service: 127: /usr/bin/service: stop: not found
/usr/bin/service: 128: exec: start: not found
They occur when executing things like "service restart nginx" these cronjobs. Note that the same commands work fine outside the cronjobs.
The PATH is correctly set in /etc/crontab . Adding it to the individual cronfiles in /etc/cron.d doesn't work either. I also tried changing SHELL=/bin/sh to SHELL=/bin/bash (even though it's insecure, but wanted to try) in /etc/crontab, didn't work.
Any ideas?
I solved it changing the command from
"service start mysql"
to
"/sbin/start mysql &"
Good luck
Enric
I am not sure, but I think that the problem is in paths. I believe you want to run /usr/sbin/service and not /usr/bin/service. You may try to specify full path for service instead of just service.
What are the steps required for setting up a Node.js application on Webfaction shared hosting account?
Introduction
The result of the following instructions is a Node.js application that is running continously, produces logfiles and restarts itself even after the reboot of the server machine. The instructions are especially targeted for the shared host accounts on Webfaction but can be used for general purpose also.
Install Node.js
Eventhough Node.js download page offers Linux binaries, it would be more robust to install Node.js from source. Download the most recent source codes and extract them to ~/src/.
(log in to your webfaction console)
> mkdir -p ~/src/
> cd src/
> wget http://nodejs.org/dist/v0.10.18/node-v0.10.18.tar.gz
> tar -xzf node-v0.10.18.tar.gz
The next step is to compile and install the code. For this Python is used. Because the default python version is 2.4 (at least on my Webfaction server web223) and because the Node.js installation requires one of the versions 2.6+, you must temporarily set newer version to be the default. See the following snippet (also see this article for details). Note --prefix=$HOME which is required due Webfaction's environment restrictions (you have access only to your home directory).
> cd node-v0.10.18/
> alias python=python2.7
> python configure --prefix=$HOME
> make
> make install
Node.js installed. Verify the success by
> node -v
v0.10.18
That also installed node package manager npm.
> npm -v
1.3.8
Install Forever
To keep run Node.js application running as long as possible and logging the output for maintenance, you need Forever. For convenience, install it globally (for you) using flag -g.
> cd ~
> npm install -g forever
> forever --version
v0.10.8
You can later update forever by
> npm update -g forever
Start the Application
First, create a custom application via Webfaction Control Panel (Custom app (listening on port)). Name it for example foonode. You may also create a domain and website for the app.
Second, make a note about the socket port number that was given for the app by Webfaction. Let the example be 48888.
Third, copy your application code to the directory of the app i.e. ~/webapps/foonode/. After that the directory contents would for example be the following.
> cd ~/webapps/foonode/
> ls
node_modules/
index.js
Index.js would be something like the snippet below. The key point is using the given port number for the app.
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello Universe.\n');
}).listen(48888, '127.0.0.1');
console.log('Listening at http://127.0.0.1:48888/');
Now you could run the app by node index.js. However there is a problem: when you exit the shell, also the app exits. Therefore you should run the Node.js as a daemon. That is something where Forever does a lot for you. Start node with forever:
> cd ~/webapps/foonode/
> forever start index.js
Now the app continues running after you logout and, as a very nice thing to have, is quickly restarted if it happens to crash. This is a lot but still not enough. What if the server reboots? Where is the output from the app going? What if the script crashes continously? What if you have multiple scripts with the name index.js? Keep reading.
Use Absolute Paths with Forever
Forever identifies the process by the filename of the script fed to forever start <filename_of_script>. A problem arises when you have multiple applications with the same filename for the script.
For example consider you have two applications /home/foouser/webapps/foonode/index.js and /home/foouser/webapps/barnode/index.js. If you now start both with forever start index.js within the directories of the applications and then run forever stop index.js only once, the result is that both applications become stopped. This happens because they both had the same identity index.js
The solution is to use the absolute filepaths when specifying the script.
> forever start /home/foouser/webapps/foonode/index.js
And to stop execution:
> forever stop /home/foouser/webapps/foonode/index.js
This ensures that only the intended application becomes stopped. This habit of using absolute paths has also positive effects to forever list. See this issue for details.
Logging
To store the application output to somewhere, specify the following arguments.
-l <logfile> = stream all output to this file. Relative to ~/.forever/
-o <logfile> = stream script's stdout to this file. Relative to current dir.
-e <logfile> = stream script's stderr to this file. Relative to current dir.
-a = append to the files instead of overwriting them.
It seems convenient to have a subdirectory for the logs, e.g. logs/. With the arguments, absolute path and the subdirectory logs/ the command becomes the following:
> cd ~/webapps/foonode/
> mkdir -p logs/
> forever start -a -l forever.log -o logs/out.log -e logs/err.log /home/foouser/webapps/foonode/index.js
Detecting and Restarting a Spinning Process
Forever has the parameters --minUptime and --spinSleepTime which are not so well documented currently. The meaning of the parameters is to control situation where the script crashes and almost immediately crashes again after restart. As forever tries to restart the script as soon as possible after the crash, this can lead to a busy loop which may eat lots of resources of the server.
--minUptime specifies the number of milliseconds the script has to be up and running without crashes to be restarted immediately. If the uptime of the crashed script is less than minUptime, then the script is considered spinning i.e. problematic. If uptime is greater then the script is considered non-spinning.
If a spinning script crashes i.e. the uptime of the script is less than --minUptime then forever waits --spinSleepTime number of milliseconds before restarting the script. Otherwise the script is restarted as soon as possible. This prevents the resource-eating loop. See this answer for futher discussion.
I personally use 5000 for --minUptime to make sure that Node is fully started before declaring it non-spinning. 2000 would be a good one for --spinSleepTime to avoid the loop but still trying to start the script quickly after the problematic situation resolves.
> forever start -a -l forever.log -o logs/out.log -e logs/err.log --minUptime=5000 --spinSleepTime=2000 /home/foouser/webapps/foonode/index.js
Manage the Application
As the commands grow, memorizing and writing them becomes more and more cumbersome. Therefore it is convenient to copy them into a Makefile. The makefile also becomes handy later on when you need to specify a command to run after the server reboot.
> cd ~/webapps/foonode/
> cat Makefile
foreverstart:
# Run server forever (until reboot)
mkdir -p logs
forever start -a -l forever.log -o logs/out.log -e logs/err.log --minUptime 5000 --spinSleepTime 2000 /home/foouser/webapps/foonode/index.js
foreverstop:
forever stop /home/foouser/webapps/foonode/index.js
Keeping the Application Running
Forever does not cover the case where the server reboots. For that you need to specify a cronjob with #reboot rule that start the forever after reboot.
> export EDITOR=nano # Editor to use to edit crontab. A matter of taste.
> crontab -e
Add the following line and save.
#reboot make -C ~/webapps/foonode/ -f ~/webapps/foonode/Makefile foreverstart
The line ensures that foonode is started immediately after the server reboot. The flag -C specifies the directory to run the makefile from and -f the actual location of the makefile. See this answer for details about using #reboot.
Conclusion
Now you have a node process running truly forever or at least as long as there is maintenance guys feeding the server with electricity. Lots of things done but maybe more will come. Things you may like to do in the future which were not covered here includes the following.
Watching for file changes and automatically restarting when detected (See --watch in forever --help)
Using Grunt.js instead of Make (See bustardcelly/grunt-forever)
Backupping the application once in a while.
See also
Setting up Redis on Webfaction
Any ideas, comments or corrections?
Well, you probably already followed all the steps in the other answer (wow! such detail! great), but it is now a lot easier: they now have a one-click installer for node.js.
Another option is to install nvm (Node Version Manager) and type:
nvm install node
It's not only valid for Webfaction.
I am continually running a few server scripts (on different ports) with nodejs using forever.
There is a considerable amount of traffic on some of these servers. The console.log commands I have for tracking connection anomalies result in bloated log files that I don't need all of the time - only for debugging. I have been manually stopping the scripts late at night, truncating the files, and restarting them. This won't do for long term, so we decided to find a solution.
Someone else on my system deleted the log files I had set up for each of the servers without my knowledge. Calling forever list on the command line shows that the server scripts are still running but now I can't tail the log files to see how the nodes are doing.
Node downtime should be kept to a bare minimum, so I'm hesitant to stop the servers during daylight hours for longer than a few minutes. Initial testing from the client side seems to indicate that the scripts are doing fine, but I can't be 100% sure there are no errors due to failed attempts at logging to a nonexistent file.
I have a few questions actually:
Is it ok to keep forever running like this?
If not, is there a proper way to disable logging? The github repository seems to indicate that forever will still log to a default file, which I don't want. Otherwise I may just write a cronjob that periodically stops scripts, truncates logs, then restarts the scripts.
What happens if I just create the logfile again with something like touch logfile_name.log while the script is still running - will this make forever freak out or is this a plausible solution?
Thanks for your time.
according to https://github.com/foreverjs/forever, try to pass -s to silent all log.
forever -s start YOURSCRIPT
Surely, before doing this, try to update forever to the latest:
sudo curl -L https://npmjs.com/install.sh | sudo sh
sudo npm update -g.
1) Just build in a periodic function or admin option to clear the forever logs. From the manual forever cleanlogs
2) At least for linux. Send each log file to /dev/null. Each log type is specified by options -l -o and -r. The -a option for append log, will stop it complaining about the log already existing.
forever start -a -l /dev/null -o /dev/null -r /dev/null your-server.js
Perhaps employ your own logging system, I use log4js, it doesn't complain if I delete the log file while the node process is still running.
There's a nifty tool that can help you that called logrotate. Have a look here
Especially the copytruncate option, it is very useful in your case.