AWS Codedeploy timesout on ApplicationStart when deploying a node.js / express server - node.js

I have run into a problem when trying to setup an AWS Codepipline. My ApplicationStart script makes a call to start the express server listening on port 60900, but because the express.listen() holds the command line up while it listens the ApplicationStart script times out and my deployment fails.
I've tried moving it to a background process with an & at the end of the command that starts the server, but I'm still getting the error at the ApplicationStart hook.
When I run the my start_server.sh script manually it almost instantly starts the server up and give me back control of the command line.
appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/mbmbam.app/
hooks:
BeforeInstall:
- location: scripts/stop_server.sh
timeout: 300
runas: root
- location: scripts/remove_previous.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/change_permissions.sh
timeout: 300
runas: root
- location: scripts/install_app.sh
timeout: 300
runas: root
- location: scripts/install_db.sh
timeout: 300
runas: root
ApplicationStart:
- location: scripts/start_server.sh
timeout: 300
runas: ubuntu
scripts/start_server.sh
#!/usr/bin/env bash
NODE_ENV=production npm start --prefix /var/www/mbmbam.app/app
Script assigned to the npm start command
app/start_app.sh
#!/bin/sh
if [ "$NODE_ENV" = "production" ]; then
node server.js &
else
nodemon --ignore './sessions' server.js;
fi
AWS Codedeploy error
LifecycleEvent - ApplicationStart
Script - scripts/start_server.sh
[stdout]
[stdout]> mbmbam-search-app#1.0.0 start /var/www/mbmbam.app/app
[stdout]> ./start_app.sh
[stdout]
Any help would be appreciated. I've been stuck on this for a day or so.

I solved it by changing the start_app.sh to
#!/bin/sh
if [ "$NODE_ENV" = "production" ]; then
node server.js > app.out.log 2> app.err.log < /dev/null &
else
nodemon --ignore './sessions' server.js;
fi
Looks like it AWS even listed it in their troubleshooting steps here:
https://docs.aws.amazon.com/codedeploy/latest/userguide/troubleshooting-deployments.html#troubleshooting-long-running-processes

The issue seems to be node not going cleanly in the backgroud.
Can you try the following way to start node server in 'app/start_app.sh':
$ nohup node server.js > /dev/null 2>&1 &
Also I would suggest to look at making your node process a service so it is started if the server is rebooted:
https://stackoverflow.com/a/29042953/12072431

Related

Config Dockerfile for running cronjob

I'm new in Docker and I'm facing a problem with my custom Dockerfile which needs some help from you guys. It's working fine until I add some code to run the cronjob in the docker container.
This is my Dockerfile file:
FROM php:7.2-fpm-alpine
COPY cronjobs /etc/crontabs/root
// old commands
ENTRYPOINT ["crond", "-f", "-d", "8"]
This is cronjobs file:
* * * * * cd /var/www/html && php artisan schedule:run >> /dev/null 2>&1
This is docker-compose.yml file:
version: '3'
networks:
laravel:
services:
nginx:
image: nginx:stable-alpine
container_name: nginx_ctrade
ports:
- "8081:80"
volumes:
- ./app:/var/www/html
- ./config/nginx/default.conf:/etc/nginx/conf.d/default.conf
- ./config/certs:/etc/nginx/certs
- ./log/nginx:/var/log/nginx
depends_on:
- php
- mysql
networks:
- laravel
working_dir: /var/www/html
php:
build:
context: ./build
dockerfile: php.dockerfile
container_name: php_ctrade
volumes:
- ./app:/var/www/html
- ./config/php/php.ini:/usr/local/etc/php/php.ini
networks:
- laravel
mysql:
image: mysql:latest
container_name: mysql_ctrade
tty: true
volumes:
- ./data:/var/lib/mysql
- ./config/mysql/my.cnf:/etc/mysql/my.cnf
environment:
- MYSQL_ROOT_PASSWORD=secret
- MYSQL_USER=admin
- MYSQL_DATABASE=laravel
- MYSQL_PASSWORD=secret
networks:
- laravel
I re-build the docker images and run it. The cronjob is working ok but when I access the localhost at localhost:8081. It isn't working anymore. The page show 502 Bad Gateway, so I checked the Nginx error log. This is the Nginx error shown me:
2020/04/10 13:33:36 [error] 8#8: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.224.1, server: localhost, request: "GET /trades HTTP/1.1", upstream: "fastcgi://192.168.224.3:9000", host: "localhost:8081", referrer: "http://localhost:8081/home"
All the containers are still running after updated.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2403ece8509 nginx:stable-alpine "nginx -g 'daemon of…" 18 seconds ago Up 17 seconds 0.0.0.0:8081->80/tcp nginx_ctrade
69032097b7e4 ctrade_php "docker-php-entrypoi…" 19 seconds ago Up 18 seconds 9000/tcp php_ctrade
592b483305d5 mysql:latest "docker-entrypoint.s…" 3 hours ago Up 18 seconds 3306/tcp, 33060/tcp mysql_ctrade
Is there someone get this issue before? Any help would be appreciated! Thanks so much!
According to the documentation, running two (or more) services inside of a Docker container breaks it's philosophy of single responsability.
It is generally recommended that you separate areas of concern by
using one service per container. That service may fork into multiple
processes (for example, Apache web server starts multiple worker
processes). It’s ok to have multiple processes, but to get the most
benefit out of Docker, avoid one container being responsible for
multiple aspects of your overall application. [...]
If you choose to follow this recommendation, you will end up with two options:
Option 1. Create a separated container that will handle the scheduling tasks.
Example:
# File: Dockerfile
FROM php:7.4.8-fpm-alpine
COPY ./cron.d/tasks /cron-tasks
RUN touch /var/log/cron.log
RUN chown www-data:www-data /var/log/cron.log
RUN /usr/bin/crontab -u www-data /cron-tasks
CMD ["crond", "-f", "-l", "8"]
# File: cron.d/tasks
* * * * * echo "Cron is working :D" >> /var/log/cron.log 2>&1
# File: docker-compose.yml
services:
[...]
scheduling:
build:
context: ./build
dockerfile: cron.dockerfile
[...]
Option 2. Use own host's crontab to execute the scheduled tasks on containers (as defended in this post).
Example:
# File on host: /etc/cron.d/my-laravel-apps
* * * * * root docker exec -t laravel-container-A php artisan schedule:run >> /dev/null 2>&1
* * * * * root docker exec -t laravel-container-B php artisan schedule:run >> /dev/null 2>&1
* * * * * root docker exec -t laravel-container-C php artisan schedule:run >> /dev/null 2>&1
PS: In your case, replace <laravel-container-*> by php_ctrade.
Option 3: Use supervisord
On the other hand, if you really want just one container at all, you may still use supervisord as your main process and configure it to initialize (and supervise) both php-fpm and crontab applications.
Note that this is a moderately heavy-weight approach and requires you
to package supervisord and its configuration in your image (or base
your image on one that includes supervisord), along with the different
applications it manages.
You will find an example of how to do it here.
References:
https://docs.docker.com/config/containers/multi-service_container/
Recommended reading:
https://devops.stackexchange.com/questions/447/why-it-is-recommended-to-run-only-one-process-in-a-container

using pm2 with ansible

I am trying to start a node program using pm2 via ansible. The problem is that the pm2 start command is not idempotent under ansible. It gives error when run again.
This is my ansible play
- name: start the application
become_user: ubuntu
command: pm2 start app.js -i max
tags:
- app
Now if i run this the first time then it runs properly but when i run this again then i get the error telling me that the script is already running.
What would be the correct way to get around this error and handle pm2 properly via ansible.
Before starting the script you should delete previous, like this:
- name: delete existing pm2 processes if running
command: "pm2 delete {{ server_id }}"
ignore_errors: True
become: yes
become_user: rw_user
- name: start pm2 process
command: 'pm2 start -x -i 4 --name "{{server_id}}" server.js'
become: yes
become_user: rw_user
environment:
NODE_ENV: "{{server_env}}"
I would use
pm2 reload app.js -i max
It will allow you to reload configuration ;-)
I ended up on this page looking for a solution to start PM2 multiple times when I rerun my playbook. I also wanted PM2 to reload the server when it was already running and pickup the new code I might have deployed. It turns out that PM2 has such an interface:
- name: Start/reload server
command: '{{path_to_deployed_pm2}} startOrReload pm2.ecosystem.config.js'
The startOrReload command requires a so-called "ecosystem" file to be present. See the documentation for more details: Ecosystem File.
This is a minimal pm2.ecosystem.config.js that is working for me:
module.exports = {
apps : [{
script: 'app.js',
name: "My app"
}],
};
Here we can use the "register" module to perform a conditional restart/start.
register the output of following command:
shell: pm2 list | grep <app_name> | awk '{print $2}'
register: APP_STATUS
become: yes
and the use APP_STATUS.stdout to make a conditional start and restart tasks. This way we don't need a pm2 delete step.

Node.js app can't access any env variables when pm2 is started from a script but can when launched from ssh

I am trying to launch a node.js app on a production EC2 server with pm2 process manager.
When I ssh into the instance and run pm2 start app.js, PM2 starts just fine and has access to all environment variables. Everything good.
However, I want to run pm2 start app.js from a Codedeploy hook script called applicationstart.sh, the app fails with an errored status becasue it is missing all environment variables.
Here is where the script is added so it is launched with each deployment and calls pm2 start: appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/teller-install
hooks:
AfterInstall:
- location: scripts/afterinstall.sh
timeout: 1000
runas: root
ApplicationStart:
- location: scripts/applicationstart.sh
timeout: 300
runas: ubuntu
Here is the applicationstart script:
#!/bin/bash
echo "Running Hook: applicationstart.sh"
cd /home/ubuntu/teller-install/Server/
pm2 start app.js
exit 0
I am logged in as ubuntu when I run the script from ssh and I set the script in the appconfig.yml to run as ubuntu as well.
Why would there be any difference between starting pm2 from terminal and starting it from a launch hook script?
Here is running directly from ssh:
I can provide any information necessary in dire need of solution. Thanks!
I had to add source /etc/profile before I call pm2 start app.js to load in environment variables.
The script now looks like
#!/bin/bash
echo "Running Hook: applicationstart.sh"
cd /home/ubuntu/teller-install/Server/
source /etc/profile
pm2 start app.js
exit 0

Why does aws code deploy throw "No passwd entry for user 'ec2-user'" error inspite of running everything as root?

Here are the error messages:
Here are the concerned files:
stop.sh
#!/bin/bash
pkill -f node
appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/
permissions:
- object: /var/www/
owner: root
mode: 777
hooks:
BeforeInstall:
- location: scripts/install.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/post_install.sh
timeout: 300
runas: root
ApplicationStart:
- location: scripts/run.sh
timeout: 300
runas: root
ApplicationStop:
- location: scripts/stop.sh
timeout: 300
runas: root
ValidateService:
- location: scripts/validate.sh
timeout: 300
runas: root
Here are the OS detail:
ubuntu#ip-172-31-2-33:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
I tried running all hooks as ec2-user 'ubuntu' as well, results are the same.
ec2-user is just default instance user. In all linux, you cannot execute any system command without first issue sudo
Fix :
#!/bin/bash
sudo pkill -f node
If you have successfully deployed the sam application in the instance before, the application stop script is actually running from last successful revision, could you check the application stop script in the last successful revision and it contains everything that you expected?
If the ApplicationStop is not right inside the last successful revision, you might want to set the --ignore-application-stop-failures option. You

CodeDeploy PM2 Command Not Found

I'm trying to use AWS CodeDeploy to deploy my application. Everything seems to be working fine but I'm getting the following error.
[stderr]/opt/codedeploy-agent/deployment-root/f1ea67bd-97bo-08q1-b3g4-7b14becf91bf/d-WJL0QLF9H/deployment-archive/scripts/start_server.sh:
line 3: pm2: command not found
Below is my start_server.sh file.
#!/bin/bash
pm2 start ~/server.js -i 0 --name "admin" &
I have tried using SSH to connect to my server as user ubuntu and running that bash file and it works perfectly with no errors. So I know that PM2 is installed and working correctly on that user.
Below is also my appspec.yml file.
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu
hooks:
ApplicationStart:
- location: scripts/start_server.sh
timeout: 300
runas: ubuntu
ApplicationStop:
- location: scripts/stop_server.sh
timeout: 300
runas: ubuntu
Also not sure if this will help but here is my stop_server.sh file.
#!/bin/bash
npm install pm2 -g
pm2 stop admin || true
pm2 delete admin || true
Any ideas?
Perform the below steps:
which node
sudo ln -s /home/ubuntu/.nvm/versions/node/v12.13.1/bin/node
(output of above step) /usr/bin/node
which pm2
sudo ln -s /home/ubuntu/.nvm/versions/node/v12.13.1/bin/pm2
(output of above step) /usr/bin/pm2
in start_server.sh and stop_server.sh use it as below (run start.sh as ubuntu):
sudo /usr/bin/pm2 status
Hope this will help you!!
All of the lifecycle events happens in the order if they have scripts to run:
ApplicationStop
DownloadBundle (reserved for CodeDeploy)
BeforeInstall
Install (reserved for CodeDeploy)
AfterInstall
ApplicationStart
ValidateService
If your deployment has reached to ApplicationStart step, which means your ApplicationStop lifecycle event is already succeeded. Can you make sure the "pm2 stop admin" is succeeded(means pm2 is installed).
Usually in cases like that is to use full path to pm2.
#!/bin/bash
/usr/local/bin/pm2 start ~/server.js -i 0 --name "admin" &
If you run
npm install pm2 -g
at ApplicationStop step, then it won't be run until the second time you deploy as ApplicationStop is run on the previous deployment archive bundle.
I just ran into this problem again.
I was able to solve it by ensuring that the following code is running at the beginning of all of my CodeDeploy script files.
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

Resources