Deploying complex node.js project with elastic beanstalk - node.js

I wonder what is the best practice of deploying complex node.js with elastic beanstalk without relying on availability of external npm repository (and dealing with credentials and high availability of privately managed git repositories for internally developed packages).
It looks like there one school of thought which preaches to actually check in node_modules into the source tree for projects that are actually being deployed.
source 1: http://www.futurealoof.com/posts/nodemodules-in-git.html
source 2: http://eng.yammer.com/managing-node-js-dependencies-and-deployments-at-yammer/
So sounds like checking them in is the right approach, but then there is a problem of different binary formats for some compiled packages (developing on mac and deploying to linux)
I've tried doing as yammer guys suggested (checkin modules except the bin folders), but even then local "npm rebuild" command fails (it tries to chmod something in a bin folder that doesn't exist in express.js module) so I haven't even gotten to try to see what beanstalk default deployment environment will do with such a repository. I assume it runs "npm install" (which will do nothing), but will it run "npm rebuild"?
So, again, what is the best practice to deploy a a complex project with multiple dependencies? It must be a solved problem by now in the node/beanstalk world, isn't it?
Thanks

Here's my configuration that does what you're talking about. Save it in the .ebextensions folder and you'll be set. The only difference between mine and the superior answer in https://stackoverflow.com/a/23242623/34340 is the NPM_CONFIG_UNSAFE_PERM=true line, which I learned from https://forums.aws.amazon.com/thread.jspa?messageID=534612
packages:
yum:
git: []
gcc: []
make: []
openssl-devel: []
libxml2: []
libxml2-devel: []
files:
"/opt/elasticbeanstalk/env.vars" :
mode: "000775"
owner: root
group: users
content: |
export HOME=/home/ec2-user # ADDED EXPORT COMMAND
export NPM_CONFIG_LOGLEVEL=error
export NPM_CONFIG_UNSAFE_PERM=true
export NODE_PATH=`ls -td /opt/elasticbeanstalk/node-install/node-* | head -1`/bin
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
mode: "000775"
owner: root
group: users
content: |
#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
ln -s /var/node_modules /tmp/deployment/application/
fi
OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && $NODE_PATH/npm install 2>&1) || error_exit "Failed to run npm install. $OUT" $?
echo $OUT
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
mode: "000666"
owner: root
group: users
content: |
#no need to run npm install during configdeploy

Related

Use pre-installed Terraform plugins instead of downloading them with terraform init

While running terraform init when using Terraform 0.11.3 we are getting the following error:
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
Error installing provider "template": Get
https://releases.hashicorp.com/terraform-provider-template/: read tcp
172.25.77.25:53742->151.101.13.183:443: read: connection reset by peer.
Terraform analyses the configuration and state and automatically
downloads plugins for the providers used. However, when attempting to
download this plugin an unexpected error occured.
This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is
blocked by a firewall.
If automatic installation is not possible or desirable in your
environment, you may alternatively manually install plugins by
downloading a suitable distribution package and placing the plugin's
executable file in the following directory:
terraform.d/plugins/linux_amd64
I realized it's because of connectivity issues with https://releases.hashicorp.com domain. For some obvious reasons, we will have to adjust with this connectivity issue as there are some SSL and firewall issues between the control server and Hashicorp's servers.
Is there any way we could bypass this by downloading the plugins from Hashicorp's servers and copying them onto the control server? Or any other alternative to avoid trying to download things from Hashicorp's servers?
You can use pre-installed plugins by either putting the plugins in the same directory as the terraform binary or by setting the -plugin-dir flag.
It's also possible to build a bundle of every provider you need automatically using the terraform-bundle tool.
I run Terraform in our CI pipeline in a Docker container so have a Dockerfile that looks something like this:
FROM golang:alpine AS terraform-bundler-build
RUN apk --no-cache add git unzip && \
go get -d -v github.com/hashicorp/terraform && \
go install ./src/github.com/hashicorp/terraform/tools/terraform-bundle
COPY terraform-bundle.hcl .
RUN terraform-bundle package terraform-bundle.hcl && \
mkdir -p terraform-bundle && \
unzip -d terraform-bundle terraform_*.zip
####################
FROM python:alpine
RUN apk add --no-cache git make && \
pip install awscli
COPY --from=terraform-bundler-build /go/terraform-bundle/* /usr/local/bin/
Note that the finished container image also adds git, make and the AWS CLI as I also require those tools in the CI jobs that uses this container.
The terraform-bundle.hcl then looks something like this (taken from the terraform-bundle README):
terraform {
# Version of Terraform to include in the bundle. An exact version number
# is required.
version = "0.10.0"
}
# Define which provider plugins are to be included
providers {
# Include the newest "aws" provider version in the 1.0 series.
aws = ["~> 1.0"]
# Include both the newest 1.0 and 2.0 versions of the "google" provider.
# Each item in these lists allows a distinct version to be added. If the
# two expressions match different versions then _both_ are included in
# the bundle archive.
google = ["~> 1.0", "~> 2.0"]
# Include a custom plugin to the bundle. Will search for the plugin in the
# plugins directory, and package it with the bundle archive. Plugin must have
# a name of the form: terraform-provider-*, and must be build with the operating
# system and architecture that terraform enterprise is running, e.g. linux and amd64
customplugin = ["0.1"]
}
config plugin_cache_dir in .terraformrc
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
then move the pre-installed provider into the plugin_cache_dir,
terraform will not download the provider anymore
btw, use the ~/.terraform.d/plugin directory doesn't work
/.terraform.d/plugin/linux_amd64$ terraform -v
Terraform v0.12.15
The proper way to handle this since terraform 0.14, as also discussed on the terraform-bundle page mentioned in the currently accepted answer, is to use terraform providers mirror as described on https://www.terraform.io/cli/commands/providers/mirror. This command creates all the necessary index files etc so the folder can be used for plugins. Eg:
$ cd your-tf-root-module
$ terraform providers mirror path/to/tf-plugins
...
$ terraform init --plugin-dir path/to/tf-plugins
...
You can cd to each of your root modules (ie those that have terraform state) and run the mirror command; multiple versions of a plugin may be installed there, and that's ok. When you run the terraform init command, it will fetch the proper one. Same as without the --plugin-dir arg.
So the only difference is that the internet is not used to acquire the plugins, terraform init gets them from the plugin folder.
This is very useful also for creating a cache that can then be used by terraform in ci/cd. Eg in circleci you would have a manual job that calls mirror and does a save-cache; and your automated terraform init job would restore-cache, and use --plugin-dir arg; then the automated terraform apply job would behave as usual.
Starting 0.13.2 version of Terraform release, one could download plugins from a local webserver/http server via network mirror protocol.
For more details, check this link
It expects a .terraformrc file in $HOME path, pointing to the provider path of the plugins like below. If the file is in different directory, you could provide the path with TERRAFORM_CONFIG env var.
provider_installation {
network_mirror {
url = "https://terraform-plugins.example.net/providers/"
}
}
Then, you define providers in a custom tf like below.
providers.tf::
terraform {
required_providers {
azurerm = {
source = "registry.terraform.io/example/azurerm"
}
openstack = {
source = "registry.terraform.io/example/openstack"
}
null = {
source = "registry.terraform.io/example/null"
}
random = {
source = "registry.terraform.io/example/random"
}
local = {
source = "registry.terraform.io/example/local"
}
}
}
However, you have to upload the plugin file in .zip format along with index.json and the <version>.json files for terraform to discover the version of plugin to download.
Example index.json containing the version of plugin::
{
"versions": {
"2.3.0": {}
}
}
Again, 2.3.0.json contains hashes of the plugin file. In this case it's <version>.json
{
"archives": {
"linux_amd64": {
"hashes": [
"h1:nFL6uiwsQFLiP8QCr35sPfWe9LpXI3/c7gP9tYnih+k="
],
"url": "terraform-provider-random_2.3.0_linux_amd64.zip"
}
}
}
How do you get details of index.json and <version>.json files?
By running terraform providers on the directory containing tf files. Note, the machine running this command, needs to connect to public terraform registry. Terraform will download the information of these files. If you have different terraform configuration files, it makes sense to automate these steps otherwise, you could manually do :)
Upon, terraform init, terraform downloads the plugins from above web server rather from terraform registry. Make sure you don't use plugin-dir argument with terraform init as it will override all the changes you made.
Updated Dockerfile for #ydaetskcoR 's solution, because currently terraform-bundle doesn't work with 0.12.x (the problem was fixed at 0.12.2, but appeared on 0.12.18)
FROM hashicorp/terraform:0.12.18 as terraform-provider
COPY provider.tf .
RUN terraform init && \
mv .terraform/plugins/linux_amd64/terraform-provider* /bin/
FROM hashicorp/terraform:0.12.18
# Install terraform pre-installed plugins
COPY --from=terraform-provider /bin/terraform-provider* /bin/
And here is the content of provider.tf
provider "template" { version = "~>2.1.2" }
provider "aws" { version = "~>2.15.0" }
...
This took me awhile, had the same problem. I ended up having to download from source and use the image that this spits out. Its nasty, but it does what i need it do to to work with the Google provider.
FROM golang:alpine AS terraform-bundler-build
ENV TERRAFORM_VERSION=0.12.20
ENV GOOGLE_PROVIDER=3.5.0
RUN apk add --update --no-cache git make tree bash curl
ENV GOPATH=/go
RUN mkdir -p $GOPATH/src/github.com/terraform-providers
RUN cd $GOPATH/src/github.com/terraform-providers && curl -sLO https://github.com/terraform-providers/terraform-provider-google-beta/archive/v$GOOGLE_PROVIDER.tar.gz
RUN cd $GOPATH/src/github.com/terraform-providers && tar xvzf v$GOOGLE_PROVIDER.tar.gz && mv terraform-provider-google-beta-$GOOGLE_PROVIDER terraform-provider-google-beta
RUN cd $GOPATH/src/github.com/terraform-providers/terraform-provider-google-beta && pwd && make build
RUN cd $GOPATH/src/github.com/terraform-providers && curl -sLO https://github.com/terraform-providers/terraform-provider-google/archive/v$GOOGLE_PROVIDER.tar.gz
RUN cd $GOPATH/src/github.com/terraform-providers && tar xvzf v$GOOGLE_PROVIDER.tar.gz && mv terraform-provider-google-$GOOGLE_PROVIDER terraform-provider-google
RUN cd $GOPATH/src/github.com/terraform-providers/terraform-provider-google && pwd && make build
RUN mkdir -p $GOPATH/src/github.com/hashicorp
RUN cd $GOPATH/src/github.com/hashicorp && curl -sLO https://github.com/hashicorp/terraform/archive/v$TERRAFORM_VERSION.tar.gz
RUN cd $GOPATH/src/github.com/hashicorp && tar xvzf v$TERRAFORM_VERSION.tar.gz && mv terraform-$TERRAFORM_VERSION terraform
RUN cd $GOPATH/src/github.com/hashicorp/terraform && go install ./tools/terraform-bundle
ENV TF_DEV=false
ENV TF_RELEASE=true
COPY my-build.sh $GOPATH/src/github.com/hashicorp/terraform/scripts/
RUN cd $GOPATH/src/github.com/hashicorp/terraform && /bin/bash scripts/my-build.sh
ENV HOME=/root
COPY terraformrc $HOME/.terraformrc
RUN mkdir -p $HOME/.terraform.d/plugin-cache
########################################
FROM alpine:3
ENV HOME=/root
RUN ["/bin/sh", "-c", "apk add --update --no-cache bash ca-certificates curl git jq openssh"]
RUN ["bin/sh", "-c", "mkdir -p /src"]
COPY --from=terraform-bundler-build /go/bin/terraform* /bin/
RUN mkdir -p /root/.terraform.d/plugins/linux_amd64
COPY --from=terraform-bundler-build /root/.terraform.d/ $HOME/.terraform.d/
RUN cp /bin/terraform-provider-google $HOME/.terraform.d/plugin-cache/linux_amd64
RUN cp /bin/terraform-provider-google-beta $HOME/.terraform.d/plugin-cache/linux_amd64
COPY terraformrc $HOME/.terraformrc
COPY provider.tf $HOME/
COPY backend.tf $HOME/
# For Testing (This should be echoed or taken care of in the CI pipeline)
#COPY google.json $HOME/.google.json
WORKDIR $HOME
ENTRYPOINT ["/bin/bash"]
.terraformrc:
plugin_cache_dir = "$HOME/.terraform.d/plugins/linux_amd64"
disable_checkpoint = true
provider.tf
# Define which provider plugins are to be included
provider "google" {
credentials = ".google.json"
}
provider "google-beta" {
credentials = ".google.json"
}
my-build.sh
#!/usr/bin/env bash
#
# This script builds the application from source for multiple platforms.
# Get the parent directory of where this script is.
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
DIR="$( cd -P "$( dirname "$SOURCE" )/.." && pwd )"
# Change into that directory
cd "$DIR"
echo "DIR=$DIR"
# Get the git commit
GIT_COMMIT=$(git rev-parse HEAD)
GIT_DIRTY=$(test -n "`git status --porcelain`" && echo "+CHANGES" || true)
# Determine the arch/os combos we're building for
XC_ARCH=${XC_ARCH:-"amd64 arm"}
XC_OS=${XC_OS:-linux}
XC_EXCLUDE_OSARCH="!darwin/arm !darwin/386"
mkdir -p bin/
# If its dev mode, only build for ourself
if [[ -n "${TF_DEV}" ]]; then
XC_OS=$(go env GOOS)
XC_ARCH=$(go env GOARCH)
# Allow LD_FLAGS to be appended during development compilations
LD_FLAGS="-X main.GitCommit=${GIT_COMMIT}${GIT_DIRTY} $LD_FLAGS"
fi
if ! which gox > /dev/null; then
echo "==> Installing gox..."
go get -u github.com/mitchellh/gox
fi
# Instruct gox to build statically linked binaries
export CGO_ENABLED=0
# In release mode we don't want debug information in the binary
if [[ -n "${TF_RELEASE}" ]]; then
LD_FLAGS="-s -w"
fi
# Ensure all remote modules are downloaded and cached before build so that
# the concurrent builds launched by gox won't race to redundantly download them.
go mod download
# Build!
echo "==> Building..."
gox \
-os="${XC_OS}" \
-arch="${XC_ARCH}" \
-osarch="${XC_EXCLUDE_OSARCH}" \
-ldflags "${LD_FLAGS}" \
-output "pkg/{{.OS}}_{{.Arch}}/${PWD##*/}" \
.
## Move all the compiled things to the $GOPATH/bin
GOPATH=${GOPATH:-$(go env GOPATH)}
case $(uname) in
CYGWIN*)
GOPATH="$(cygpath $GOPATH)"
;;
esac
OLDIFS=$IFS
IFS=: MAIN_GOPATH=($GOPATH)
IFS=$OLDIFS
#
# Create GOPATH/bin if it's doesn't exists
if [ ! -d $MAIN_GOPATH/bin ]; then
echo "==> Creating GOPATH/bin directory..."
mkdir -p $MAIN_GOPATH/bin
fi
# Copy our OS/Arch to the bin/ directory
DEV_PLATFORM="./pkg/$(go env GOOS)_$(go env GOARCH)"
if [[ -d "${DEV_PLATFORM}" ]]; then
for F in $(find ${DEV_PLATFORM} -mindepth 1 -maxdepth 1 -type f); do
cp ${F} bin/
cp ${F} ${MAIN_GOPATH}/bin/
ls -alrt ${MAIN_GOPATH}/bin/
echo "MAIN_GOPATH=${MAIN_GOPATH}"
done
fi
bucket.tf
terraform {
backend "gcs" {
bucket = "my-terraform-bucket"
prefix = "terraform/state"
credentials = ".google.json"
}
required_version = "v0.12.20"
}
You can use pre-installed plugins by either putting the plugins binaries in the same directory where Terraform binary is available by setting the "plugins-dir" flag.
By default, all plugins downloaded in .terraform folder. For example, Null resource plugin will be available at below location
.terraform\providers\registry.terraform.io\hashicorp\null\3.0.0.\windows_amd64.
Create new folder like "terraform-plugins" inside Terraform directory and copy all content including registry.terraform.io folder mentioned in above example in created folder.
Now run the terraform init command with plugins-dir flag
terraform init -plugin-dir="/terraform-plugins"
specify complete directory path with plugin-dir flag

Laravel - configuration cache on elastic beanstalk

I have a Laravel application running on an Elastic Beanstalk environment.
Not having access to the database, S3 and SQS variables I did write a config on ebxtensions to copy some environment variables on the .env file during the deploy using the echo On a .sh hook file on post-deploy like this:
echo -e "AWS_BUCKET=$AWS_BUCKET" >> /var/app/current/.env
The .env file is correctly updated however another .sh hook that runs after that is completed that has the code:
php /var/app/current/artisan config:cache
And this saves the cached config file as the .env file was not updated yet.
Right now the config:cache command needs to be run manually after the deploy but I really want to make the process all automatic.
Any ideas why that happen?
Process of EB deploy is very interesting, take a look at /var/log/eb-activity.log
++ /opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir
+ EB_APP_DEPLOY_DIR=/var/app/current
+ '[' -d /var/app/current ']'
+ mv /var/app/current /var/app/current.old
+ mv /var/app/ondeck /var/app/current
+ nohup rm -rf /var/app/current.old
So, your config:cache running in previous environment which deleted after deploy.
You should use this post-hook in .ebextensions/01-post.config:
files:
/opt/elasticbeanstalk/hooks/appdeploy/post/01_create_cache.sh:
mode: "000755"
owner: root
group: root
content: |
php /var/app/current/artisan config:cache >>/var/log/artisan_test.log
But use it carefully! It takes variables only from .env, not from EB VARIABLES!
Right way will be collect all variables to .env and than generate config cache.
files:
/opt/elasticbeanstalk/hooks/appdeploy/post/01_create_cache.sh:
mode: "000755"
owner: root
group: root
content: |
source /opt/elasticbeanstalk/support/envvars && /usr/bin/php /var/www/html/artisan config:cache >>/var/log/artisan_test.log

AWS Linux CodeDeploy Permission Issues (w. Bitbucket, Tomcat, Shell Script)

I'm trying to deploy files using CodeDeploy to my AWS Beanstalk server with Tomcat installed. Everything is well configured except for an exception which occurs when appspec.yml calls my .sh script and mvn install command is executed. I've tried all combinations of permissions I've imagined (as well as every StackOverflow answer I've found), but nothing has worked.
Cannot create resource output directory: /opt/codedeploy-agent/deployment-root/f953d455-9712-454b-84b0-2533cf87f79a/d-3UFCDLD0D/deployment-archive/target/classes
I also expected the files section of appspec.yml to get executed before the .sh script gets executed. It should have been working like this:
appspec.yml moves all files to webapps folder
build.sh gets executed
mvn runs and creates the .war file
build.sh does some cleaning up
appspec.yml (I've tried multiple other)
version: 0.0
os: linux
files:
- source: /
destination: /var/lib/tomcat8/webapps
permissions:
- object: /opt/codedeploy-agent/deployment-root
pattern: "**"
owner: ec2-user
group: root
mode: 755
type:
- directory
- object: /var/lib/tomcat8/webapps
pattern: "**"
owner: ec2-user
group: root
mode: 755
type:
- directory
hooks:
BeforeInstall:
- location: scripts/build.sh
runas: ec2-user
build.sh
export LANG=en_US.UTF-8
SCRIPTPATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
echo "Script path: $SCRIPTPATH"
PROJECT_SOURCE_DIR=$SCRIPTPATH/../
cd $PROJECT_SOURCE_DIR
mvn clean install
cd $PROJECT_SOURCE_DIR/target
ls -a
for file in *.war; do
mv $file /usr/share/tomcat8/webapps/ROOT.war
done;
rm -rf $PROJECT_SOURCE_DIR/target
rm -rf $SCRIPTPATH
It's obvious from the exception that maven tries to create a folder target without having the permissions. So the questions are why on the first place it's trying to execute it in this folder and then how to gain proper access.
The way to solve the problem is to add command to change to proper directory before run "mvn clean install" instead of PROJECT_SOURCE_DIR.
Install is the lifecycle event that AWS CodeDeploy agent copies the revision files from the temporary location to the final destination folder. This event is reserved for the AWS CodeDeploy agent and cannot be used to run scripts. The related doc is here: http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html
The directory that you are getting error is actually under the deployment archive directory as showing here: https://github.com/aws/aws-codedeploy-agent/blob/master/lib/instance_agent/plugins/codedeploy/hook_executor.rb#L174
The reason you got the error is because the build.sh script is running at the current directory which needs root privilege and scripts/build.sh only has ex2-user privilege, which caused the permission issue.

avoid rebuilding node_modules in elastic beanstalk

We have a fairly simple node.js app, but due to AWS Elastic Beanstalk deployment mechanism, it takes about 5 minutes to roll-out a new version (via git aws.push) even after a single file commit.
I.e. the commit itself (and upload) is fast (only 1 file to push), but then Elastic Beanstalk fetches whole package from S3, unzips it and runs npm install, which causes node-gyp to compile some modules. Upon installation/building completion, Elastic Beanstalk wipes /var/app/current and replaces it with the new app version.
Needless to say, constant node_modules rebuilding is not necessary, and rebuilding that takes 30 seconds on my old Macbook Air, takes >5 mins on a ec2.micro instance, not fun.
I see two approaches here:
tweak /opt/containerfiles/ebnode.py and play with node_modules location to avoid its removal and rebuilding upon deployment.
set up a git repo on Elastic Beanstalk EC2 instance and basically re-write deployment procedure ourselves, so /var/app/current receives pushes and runs npm install only when necessary (which makes Elastic Beanstalk to look like OpsWorks..)
Both options lack grace and are prone to issues when Amazon updates their Elastic Beanstalk hooks and architecture.
Maybe somebody has a better idea how to avoid constant rebuilding of node_modules that are already present in the app dir? Thank you.
Thanks Kirill, it was really helpful !
I'm just sharing my config file for people who just look the simple solution to the npm install. This file needs to be placed in the .ebextensions folder of the project, it is lighter since it doesn't include last version of node installation, and ready to use.
It also dynamically checks the node version installed, so no need for it to be included in the env.vars file.
.ebextensions/00_deploy_npm.config
files:
"/opt/elasticbeanstalk/env.vars" :
mode: "000775"
owner: root
group: users
content: |
export NPM_CONFIG_LOGLEVEL=error
export NODE_PATH=`ls -td /opt/elasticbeanstalk/node-install/node-* | head -1`/bin
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
mode: "000775"
owner: root
group: users
content: |
#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
ln -s /var/node_modules /tmp/deployment/application/
fi
OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && $NODE_PATH/npm install 2>&1) || error_exit "Failed to run npm install. $OUT" $?
echo $OUT
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
mode: "000666"
owner: root
group: users
content: |
#no need to run npm install during configdeploy
25/01/13 NOTE: updated scripts to run npm -g version upgrade (only once, on initial instance roll out or rebuild) and to avoid NPM operations during EB configuration change (when app dir is not present, to avoid error and to speed up configuration updates).
Okay, Elastic Beanstalk behaves dodgy with recent node.js builds (including presumably supported v.0.10.10), so I decided to go ahead and tweak EB to do the following:
to install ANY node.js version as per your env.config (including
the most recent ones that are not yet supported by AWS EB)
to avoid rebuilding existing node modules, including in-app
node_modules dir
to install node.js globally (and any desired module as well).
Basically, I use env.config to replace deploy&config hooks with customized ones (see below). Also, in a default EB container setup some env variables are missing ($HOME for example) and node-gyp sometimes fails during rebuild because of it (took me 2 hours of googling and reinstalling libxmljs to resolve this).
Below are the files to be included along with your build. You can inject them via env.config as inline code or via source: URL (as in this example)
env.vars (desired node version & arch are included here and in env.config, see below)
export HOME=/root
export NPM_CONFIG_LOGLEVEL=error
export NODE_VER=0.10.24
export ARCH=x86
export PATH="$PATH:/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/:/root/.npm"
40install_node.sh (fetch and ungzip desired node.js version, make global symlinks, update global npm version)
#!/bin/bash
#source env variables including node version
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#UNCOMMENT to update npm, otherwise will be updated on instance init or rebuild
#rm -f /opt/elasticbeanstalk/node-install/npm_updated
#download and extract desired node.js version
OUT=$( [ ! -d "/opt/elasticbeanstalk/node-install" ] && mkdir /opt/elasticbeanstalk/node-install ; cd /opt/elasticbeanstalk/node-install/ && wget -nc http://nodejs.org/dist/v$NODE_VER/node-v$NODE_VER-linux-$ARCH.tar.gz && tar --skip-old-files -xzpf node-v$NODE_VER-linux-$ARCH.tar.gz) || error_exit "Failed to UPDATE node version. $OUT" $?.
echo $OUT
#make sure node binaries can be found globally
if [ ! -L /usr/bin/node ]; then
ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/node /usr/bin/node
fi
if [ ! -L /usr/bin/npm ]; then
ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm /usr/bin/npm
fi
if [ ! -f "/opt/elasticbeanstalk/node-install/npm_updated" ]; then
/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/ && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm update npm -g
touch /opt/elasticbeanstalk/node-install/npm_updated
echo "YAY! Updated global NPM version to `npm -v`"
else
echo "Skipping NPM -g version update. To update, please uncomment 40install_node.sh:12"
fi
50npm.sh (creates /var/node_modules, symlinks it to app dir and runs npm install. You can install any module globally from here, they will land in /root/.npm)
#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
ln -s /var/node_modules /tmp/deployment/application/
fi
OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm install 2>&1) || error_exit "Failed to run npm install. $OUT" $?
echo $OUT
env.config (note node version here too, and to be safe, put desired node version in env config in AWS console as well. I'm not certain which of these settings will take precedence.)
packages:
yum:
git: []
gcc: []
make: []
openssl-devel: []
option_settings:
- option_name: NODE_ENV
value: production
- option_name: RDS_HOSTNAME
value: fill_me_in
- option_name: RDS_PASSWORD
value: fill_me_in
- option_name: RDS_USERNAME
value: fill_me_in
- namespace: aws:elasticbeanstalk:container:nodejs
option_name: NodeVersion
value: 0.10.24
files:
"/opt/elasticbeanstalk/env.vars" :
mode: "000775"
owner: root
group: users
source: https://dl.dropbox.com/....
"/opt/elasticbeanstalk/hooks/configdeploy/pre/40install_node.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
mode: "000666"
owner: root
group: users
content: |
#no need to run npm install during configdeploy
"/opt/elasticbeanstalk/hooks/appdeploy/pre/40install_node.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
There you have it: on t1.micro instance deployment now takes 20-30 secs instead of 10-15 minutes! If you deploy 10 times a day, this tweak will save you 3 (three) weeks in a year.
Hope it helps and special thanks to AWS EB staff for my lost weekend :)
There's npm package that's overwriting default EB behaviour for npm install command by truncating following files:
/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh
/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh
https://www.npmjs.com/package/eb-disable-npm
Might be better than just copying script from SO, since this package is maintained and probably will be updated when EB behaviour will change.
I've found a quick solution to this. I looked through the build scripts that Amazon are using and they only run npm install if package.json is present. So after your initial deploy you can change it to _package.json and npm install won't run anymore! It's not the best solution but it's a quick fix if you need one!
I had 10+ minute builds when I would deploy. The solution was much simpler than others have came up with... Just check node_modules into git! See http://www.futurealoof.com/posts/nodemodules-in-git.html for the reasoning

NPM private git module on Heroku

I am trying to deploy my app to Heroku however I rely on using some private git repos as modules. I do this for code reuse between projects, e.g. I have a custom logger I use in multiple apps.
"logger":"git+ssh://git#bitbucket.org..............#master"
The problem is Heroku obviously does not have ssh access to this code. I can't find anything on this problem. Ideally Heroku have a public key I can can just add to the modules.
Basic auth
GitHub has support for basic auth:
"dependencies" : {
"my-module" : "git+https://my_username:my_password#github.com/my_github_account/my_repo.git"
}
As does BitBucket:
"dependencies" : {
"my-module": "git+https://my_username:my_password#bitbucket.org/my_bitbucket_account/my_repo.git"
}
But having plain passwords in your package.json is probably not desired.
Personal access tokens (GitHub)
To make this answer more up-to-date, I would now suggest using a personal access token on GitHub instead of username/password combo.
You should now use:
"dependencies" : {
"my-module" : "git+https://<username>:<token>#github.com/my_github_account/my_repo.git"
}
For Github you can generate a new token here:
https://github.com/settings/tokens
App passwords (Bitbucket)
App passwords are primarily intended as a way to provide compatibility with apps that don't support two-factor authentication, and you can use them for this purpose as well. First, create an app password, then specify your dependency like this:
"dependencies" : {
"my-module": "git+https://<username>:<app-password>#bitbucket.org/my_bitbucket_account/my_repo.git"
}
[Deprecated] API key for teams (Bitbucket)
For BitBucket you can generate an API Key on the Manage Team page and then use this URL:
"dependencies" : {
"my-module" : "git+https://<teamname>:<api-key>#bitbucket.org/team_name/repo_name.git"
}
Update 2016-03-26
The method described no longer works if you are using npm3, since npm3 fetches all modules described in package.json before running the preinstall script. This has been confirmed as a bug.
The official node.js Heroku buildpack now includes heroku-prebuild and heroku-postbuild, which will be run before and after npm install respectively. You should use these scripts instead of preinstall and postinstall in all cases, to support both npm2 and npm3.
In other words, your package.json should resemble:
"scripts": {
"heroku-prebuild": "bash preinstall.sh",
"heroku-postbuild": "bash postinstall.sh"
}
I've come up with an alternative to Michael's answer, retaining the (IMO) favourable requirement of keeping your credentials out of source control, whilst not requiring a custom buildpack. This was borne out of frustration that the buildpack linked by Michael is rather out of date.
The solution is to setup and tear down the SSH environment in npm's preinstall and postinstall scripts, instead of in the buildpack.
Follow these instructions:
Create two scripts in your repo, let's call them preinstall.sh and postinstall.sh.
Make them executable (chmod +x *.sh).
Add the following to preinstall.sh:
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod 400 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host github.com\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
fi
Add the following to postinstall.sh:
#!/bin/bash
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Cleaning up SSH config" >&1
echo "" >&1
# Now that npm has finished running, we shouldn't need the ssh key/config anymore.
# Remove the files that we created.
rm -f ~/.ssh/config
rm -f ~/.ssh/deploy_key
# Clear that sensitive key data from the environment
export GIT_SSH_KEY=0
fi
Add the following to your package.json:
"scripts": {
"preinstall": "bash preinstall.sh",
"postinstall": "bash postinstall.sh"
}
Generate a private/public key pair using ssh-agent.
Add the public key as a deploy key on Github.
Create a base64 encoded version of your private key, and set it as the Heroku config var GIT_SSH_KEY.
Commit and push your app to Github.
When Heroku builds your app, before npm installs your dependencies, the preinstall.sh script is run. This creates a private key file from the decoded contents of the GIT_SSH_KEY environment variable, and creates an SSH config file to tell SSH to use this file when connecting to github.com. (If you are connecting to Bitbucket instead, then update the Host entry in preinstall.sh to bitbucket.org). npm then installs the modules using this SSH config. After installation, the private key is removed and the config is wiped.
This allows Heroku to pull down your private modules via SSH, while keeping the private key out of the codebase. If your private key becomes compromised, since it is just one half of a deploy key, you can revoke the public key in GitHub and regenerate the keypair.
As an aside, since GitHub deploy keys have read/write permissions, if you are hosting the module in a GitHub organization, you can instead create a read-only team and assign a 'deploy' user to it. The deploy user can then be configured with the public half of the keypair. This adds an extra layer of security to your module.
It's a REALLY bad idea to have plain text passwords in your git repo, using an access token is better, but you will still want to be super careful.
"my_module": "git+https://ACCESS_TOKEN:x-oauth-basic#github.com/me/my_module.git"
I created a custom nodeJS buildpack that will allow you to specify an SSH key that is registered with ssh-agent and used by npm when dynos are first setup. It seamlessly allows you to specify your module as an ssh url in your package.json like shown:
"private_module": "git+ssh://git#github.com:me/my_module.git"
To setup your app to use your private key:
Generate a key: ssh-keygen -t rsa -C "your_email#example.com" (Enter no passphrase. The buildpack does not support keys with passphrases)
Add the public key to github: pbcopy < ~/.ssh/id_rsa.pub (in OS X) and paste the results into the github admin
Add the private key to your heroku app's config: cat id_rsa | base64 | pbcopy, then heroku config:set GIT_SSH_KEY=<paste_here> --app your-app-name
Setup your app to use the buildpack as described in the heroku nodeJS buildpack README included in the project. In summary the simplest way is to set a special config value with heroku config:set to the github url of the repository containing the desired buildpack. I'd recommend forking my version and linking to your own github fork, as I'm not promising to not change my buildpack.
My custom buildpack can be found here: https://github.com/thirdiron/heroku-buildpack-nodejs and it works for my system. Comments and pull requests are more than welcome.
Based on the answer from #fiznool I created a buildpack to solve this problem using a custom ssh key stored as an environment variable. As the buildpack is technology agnostic, it can be used to download dependencies using any tool like composer for php, bundler for ruby, npm for javascript, etc: https://github.com/simon0191/custom-ssh-key-buildpack
Add the buildpack to your app:
$ heroku buildpacks:add --index 1 https://github.com/simon0191/custom-ssh-key-buildpack
Generate a new SSH key without passphrase (lets say you named it deploy_key)
Add the public key to your private repository account. For example:
Github: https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
Bitbucket: https://confluence.atlassian.com/bitbucket/add-an-ssh-key-to-an-account-302811853.html
Encode the private key as a base64 string and add it as the CUSTOM_SSH_KEY environment variable of the heroku app.
Make a comma separated list of the hosts for which the ssh key should be used and add it as the CUSTOM_SSH_KEY_HOSTS environment variable of the heroku app.
# MacOS
$ heroku config:set CUSTOM_SSH_KEY=$(base64 --input ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
# Ubuntu
$ heroku config:set CUSTOM_SSH_KEY=$(base64 ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
Deploy your app and enjoy :)
I was able to setup resolving of Github private repositories in Heroku build via Personal access tokens.
Generate Github access token here: https://github.com/settings/tokens
Set access token as Heroku config var: heroku config:set GITHUB_TOKEN=<paste_here> --app your-app-name or via Heroku Dashboard
Add heroku-prebuild.sh script:
#!/bin/bash
if [ "$GITHUB_TOKEN" != "" ]; then
echo "Detected GITHUB_TOKEN. Setting git config to use the security token" >&1
git config --global url."https://${GITHUB_TOKEN}#github.com/".insteadOf git#github.com:
fi
add the prebuild script to package.json:
"scripts": {
"heroku-prebuild": "bash heroku-prebuild.sh"
}
For local environment we can also use git config ... or we can add the access token to ~/.netrc file:
machine github.com
login PASTE_GITHUB_USERNAME_HERE
password PASTE_GITHUB_TOKEN_HERE
and installing private github repos should work.
npm install OWNER/REPO --save will appear in package.json as: "REPO": "github:OWNER/REPO"
and resolving private repos in Heroku build should also work.
optionally you can setup a postbuild script to unset the GITHUB_TOKEN.
This answer is good https://stackoverflow.com/a/29677091/6135922, but I changed a little bit preinstall script. Hope this will help someone.
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
echo "Preinstall"
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod o-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host bitbucket.org\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" HostName bitbucket.org\n" \
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
echo "eval `ssh-agent -s`"
eval `ssh-agent -s`
echo "ssh-add -l"
ssh-add -l
echo "ssh-add ~/.ssh/deploy_key"
ssh-add ~/.ssh/deploy_key
# uncomment to check that everything works just fine
# ssh -v git#bitbucket.org
fi
You can use in package.json private repository with authentication example below:
https://usernamegit:passwordgit#github.com/reponame/web/tarball/branchname
In short it is not possible. The best solution to this problem I came up with is to use the new git subtree's. At the time of writing they are not in the official git source and so needs to be installed manual but they will be included in v1.7.11. At the moment it is available on homebrew and apt-get. it is then a case of doing
git subtree add -P /node_modules/someprivatemodue git#github.......someprivatemodule {master|tag|commit}
this bulks out the repo size but an update is easy by doing the command above with gitsubtree pull.
I have done this before with modules from github. Npm currently accepts the name of the package or a link to a tar.gz file which contains the package.
For example if you want to use express.js directly from Github (grab the link via the download section) you could do:
"dependencies" : {
"express" : "https://github.com/visionmedia/express/tarball/2.5.9"
}
So you need to find a way to access you repository as a tar.gz file via http(s).

Resources