I have been trying to create CI/CD using code deploy and bitbucket repo.
The pipeline is successful but I am not seeking any codes into ec2. I can only see the node module in ec2.
If anyone came through the same issues or could help me to solve them that would be great.
appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/gt
hooks:
ApplicationStart:
- location: scripts/start_server.sh
runas: root
start_server.sh
sudo apt-get update
# install the application using npm
# we need to traverse to where the application bundle is copied too.
#some comments
#added commets
sudo su
rm -rf /home/ubuntu/gt
mkdir /home/ubuntu/gt
echo installing application with npm
cd /home/ubuntu/gt
sudo apt-get install -y npm
echo installing pm2
npm install pm2 -g
sudo yarn
pm2 delete gt
pm2 start npm --name 'gt' -- start
I am not seeking any codes into ec2
This could be because you are removing all content of your folder:
rm -rf /home/ubuntu/gt
since your ApplicationStart runs after files. So whatever you copy in files, gets deleted in ApplicationStart. For the order of execution, please have a look here.
Related
I'm trying to provision a project locally that's using NodeJs with NPM.
I'm running npm install on my host machine (MacBook Pro Retina, 15-inch, Mid 2015) using nvm with node version 10.19:
added 2335 packages from 985 contributors and audited 916010 packages in 61.736s
When I run the same setup in Docker, the result is much slower. This is my docker-compose.yml file:
version: '3.4'
services:
node:
image: node:10.19-alpine
container_name: node
volumes:
- .:/app/
- npm-cache:/root/.npm
working_dir: /app
command: ["tail", "-f", "/dev/null"]
volumes:
npm-cache:
external: false
Then I execute:
docker-compose up -d node; docker exec -t node npm install
And the result is:
added 2265 packages from 975 contributors and audited 916010 packages in 259.895s
(I'm assuming the number of resulting packages is different due to a different platform).
I thought the speedy installation was achieved by having a local cache (that's why there is an extra volume for caching in the docker-compose) but then I ran:
$ npm cache clean --force && rm -rf ~/.npm && rm -rf node_modules
and the result for installation on the host machine is still consistently ~60 seconds.
When it comes to resources allocated to the Docker VM, it shouldn't be a problem, here's my Docker VM configuration:
I don't know where else to look, any help would be greatly appreciated.
Thanks
This slowdown is caused by sharing files between a container and your host machine.
In order to cope with it, you can give a try to docker-sync.
This tool supports different strategies for automatical syncing between a host machine and containers (including rsync).
However, beware that it has own issues like occasional sync freezing.
Here is how I got around the issue.
Create a base docker image using a similar Dockerfile.
FROM node:latest
RUN mkdir -p /node/app
COPY ./package.json /node/app/package.json
WORKDIR "/node/app"
RUN yarn install --network-timeout 100000
Then in your container, make this script the entry point:
#!/bin/bash
mkdir -p /node/app
cp /srv/package.json /node/app
cd /node/app
yarn install
sleep 1
rm -f /srv/node_modules
ln -s /node/app/node_modules /srv/node_modules
cd /srv
sleep 1
yarn serve
This installs npm modules in another directory that is not synced between container and host, and links the directory for the app. This seems to work just fine until this is properly resolved by Docker.
I try to deploy a Laravel repo on AWS Elastic Beanstalk. For this I've created a PHP environment on a t2.medium instance and initialized a deployment with AWS CodePipeline.
I need to run npm and gulp on production for creating my static assets, but unfortunately I'm not able to run NPM because of an old NodeJS version installed. It's 0.10.46-1nodesource.el7.centos.
At the moment I'm using two config scripts to install NodeJS 10.X, run the migrations and gulp:
First:
commands:
01getNodeRepo:
command: "curl --silent --location https://rpm.nodesource.com/setup_10.x | bash -"
02installNode:
command: "yum install -y nodejs"
03updateNpm:
command: "npm install npm -g"
04enableSudo:
command: "echo Defaults:root \\!requiretty >> /etc/sudoers"
Second:
container_commands:
01artisanMigrate:
command: "php artisan migrate --force"
02showNodeVersion:
command: "node -v"
03showNpmVersion:
command: "npm -v"
04npmInstall:
command: "sudo npm install"
05gulp:
command: "sudo ./node_modules/.bin/gulp --production"
Here is an extract from my log files which shows that AWS EB found two nodesource repos and will be using the old version:
[2019-01-01T19:32:23.008Z] INFO [1535] - [Application update code-pipeline-xx-xxx#17/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/xxxx/Command 02installNode] : Starting activity...
[2019-01-01T19:32:25.314Z] INFO [1535] - [Application update code-pipeline-xx-xxx#17/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/xxxx/Command 02installNode] : Completed activity. Result:
Loaded plugins: priorities, update-motd, upgrade-helper
Repository nodesource is listed more than once in the configuration
Repository nodesource-source is listed more than once in the configuration
Resolving Dependencies
--> Running transaction check
---> Package nodejs.x86_64 0:0.10.46-1nodesource.el7.centos will be installed
--> Finished Dependency Resolution
Any ideas how I can force the environment to use the new version? Thanks!
Got it. This was the solution: https://github.com/nodesource/distributions/issues/421#issuecomment-318560799
rm -f /etc/yum.repos.d/nodesource-el.repo
yum clean all
yum -y remove nodejs
yum -y install nodejs
Isit possible to configure EBS to install my NodeJS application using yarn package manager instead of NPM?
I've figured out a way, but it is a little hacky.
Create a .ebextensions/yarn.config file. (The name does not have to be 'yarn'.)
Put this content into the file:
files:
# Runs right before `npm install` in '.../50npm.sh'
"/opt/elasticbeanstalk/hooks/appdeploy/pre/49yarn.sh" :
mode: "000775"
owner: root
group: users
content: |
#!/bin/bash
app="$(/opt/elasticbeanstalk/bin/get-config container -k app_staging_dir)";
# install node
curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -;
# install yarn
curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo;
yum -y install yarn;
# install node_modules with yarn
cd "${app}";
yarn --production;
This ebextension creates a file which does 3 things:
Installs node.
Installs yarn.
Installs node_modules with yarn.
In order to make Elastic Beanstalk run yarn install before it runs npm install, the file is created under /opt/elasticbeanstalk/hooks/appdeploy/pre. This turns the file into a pre-deployment hook, which means that Elastic Beanstalk will run it during the first phase of deployment. By default, there is another file in this directory called 50npm.sh, which runs npm install. Since Elastic Beanstalk runs the files in this directory alphabetically, 49yarn.sh (our file) will run before 50npm.sh (the default file), resulting in yarn install running before npm install.
One potential problem is that the environment variables set in the Elastic Beanstalk UI (under Configuration > Software Configuration) are not available at this point of the deployment phase. This is a big problem if you have an npm auth token there which you use to install private npm modules.
Another potential problem is that this installs node manually, so the "Node version" you specify in the Elastic Beanstalk UI (under Configuration > Software Configuration) will have no effect on the version of node your application uses; you need to specify it in this ebextension. Elastic Beanstalk's 50npm.sh both installs node and runs npm install. Since we have to run yarn install before that file runs, we also have to install node manually. Then, when Elastic Beanstalk goes to install node, it detects that node is already installed but does not verify that it is the correct version, so it skips the node installation.
For reference, the yarn installation instructions came from here: https://yarnpkg.com/docs/install#linux-tab
I did this following instructions on https://yarnpkg.com/lang/en/docs/install/
commands:
01_install_yarn:
command: "sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo && curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash - && sudo yum install yarn -y"
This way that i came up with lets you still control the node version via the Elastic Beanstalks Dashboard.
Thanks for this question! couldn't have come to this solution without it :D
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh":
mode: "000755"
owner: root
group: users
content: |
#!/usr/bin/env bash
#
# Prevent installing or rebuilding like Elastic Beanstalk tries to do by
# default.
#
# Note that this *overwrites* Elastic Beanstalk's default 50npm.sh script
# (https://gist.github.com/wearhere/de51bb799f5099cec0ed28b9d0eb3663).
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh":
mode: "000755"
owner: root
group: users
content: |
#!/usr/bin/env bash
#
# Prevent installing or rebuilding like Elastic Beanstalk tries to do by
# default.
#
# Note that this *overwrites* Elastic Beanstalk's default 50npm.sh script.
# But their default script actually doesn't work at all, since the app
# staging dir, where they try to run `npm install`, doesn't exist during
# config deploys, so ebnode.py just aborts:
# https://gist.github.com/wearhere/de51bb799f5099cec0ed28b9d0eb3663#file-ebnode-py-L140
"/opt/elasticbeanstalk/hooks/appdeploy/pre/49yarn.sh" :
mode: "000775"
owner: root
group: users
content: |
tmp="$(mktemp || bail)";
app="$(/opt/elasticbeanstalk/bin/get-config container -k app_staging_dir)";
version="$(/opt/elasticbeanstalk/bin/get-config optionsettings -n aws:elasticbeanstalk:container:nodejs -o NodeVersion)";
echo $version
major="$(cut -d'.' -f1 <<<${version})"
yum -y install python26 python26-libs
wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo;
wget "https://rpm.nodesource.com/pub_${major}.x/el/7/x86_64/nodejs-${version}-1nodesource.x86_64.rpm" -O "${tmp}";
rpm -i --nosignature --force "${tmp}";
rm -f "${tmp}";
yum -y install yarn;
cd "${app}";
yarn --production;
Had to revisit this as we couldn't figure out why we were stuck on node 8 even though we set it to node 12 in the EB UI. Seems that if you install a global node it overrides the version setting. Instead of installing a global node, this uses the Elastic Beanstalk node install and adds it to the path. You have to add the PATH in again at the start of your yarn install script but it seems to be the least invasive way to use yarn.
content: |
#!/usr/bin/env bash
set -euxo pipefail
EB_NODE_VERSION=$(/opt/elasticbeanstalk/bin/get-config optionsettings -n aws:elasticbeanstalk:container:nodejs -o NodeVersion)
echo "EB node version: $(EB_NODE_VERSION)"
# Make sure Node binaries can be found (required to run npm).
# And this lets us invoke npm more simply too.
export PATH=/opt/elasticbeanstalk/node-install/node-v$EB_NODE_VERSION-linux-x64/bin:$PATH
if yarn -v; then
echo 'Yarn already installed.'
else
echo 'Installing yarn...'
npm install yarn -g
fi
An easy way to prevent EB from running npm install is to create an empty node_modules folder in a prebuild hook:
Edit your_project/.platform/hooks/prebuild/prevent-npm.sh:
#!/bin/bash
# EB build scripts will not install using npm if node_modules folder exists
mkdir node_modules
This way, EB will still install Node.js, so you don't have to install that yourself. But it will skip npm install when node_modules exists.
Then inside the predeploy hook, you can run yarn. If you are running Node.js 16 or newer, you can use corepack yarn, and it will just work (you don't have to install yarn from yum, npm or anything else.)
Edit your_project/.platform/hooks/predeploy/yarn.sh:
#!/bin/bash
corepack yarn
Be sure to also create symlinks under confighooks in your project, or else it will fail to build when changing config (e.g. updating environment values):
mkdir -p .platform/confighooks/{prebuild,predeploy}
ln -s ../../hooks/predeploy/yarn.sh .platform/confighooks/predeploy/yarn.sh
ln -s ../../hooks/prebuild/prevent-npm.sh .platform/confighooks/prebuild/prevent-npm.sh
Since get-config is no longer present in the new Amazon Linux 2 platform, we had to figure another clean way to do this, and came up with the following :
container_commands:
01_npm_install_yarn:
command: "npm install -g yarn"
10_yarn_install:
command: 'PATH="$PATH:$(dirname $(readlink $(which node)))" yarn install'
You may want to put the PATH= logic in a script and call it before every yarn command, to have clean command: instructions in your extentions.
Also, note that if you install yarn using the yum package manager, you completely break the NodeJS version management provided by Beanstalk (since it the black magic running behind make some symlinks in /bin and /usr/bin).
I am having some trouble mounting a directory on my machine into my Docker container. I would like to mount a directory containing files necessary to run a node server. So far, I have successfully been able to run and access my server in browser using the Dockerfile below:
# Use an ubuntu base image
FROM ubuntu:14.04
# Install Node.js and npm (this will install the latest version for ubuntu)
RUN apt-get update
RUN apt-get -y install curl
RUN curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
RUN apt-get -y install nodejs
RUN apt-get -y install git
# Bundle app source (note: all of my configuration files/folders are in the current directory along with the Dockerfile)
COPY . /src
Install app dependencies
#WORKDIR /src
RUN npm install
RUN npm install -g bower
RUN bower install --allow-root
RUN npm install -g grunt
RUN npm install -g grunt-cli
#What port to expose?
EXPOSE 1234
#Run grunt on container start
CMD grunt
And these commands:
docker build -t test/test .
docker run -p 1234:1234 -d test/test
However, I figured that I would like the configuration files and whatnot to persist, and thought to do this by mounting the directory (with the files and Dockerfile) as a volume. I used other solutions on StackOverflow to get this command:
docker run -p 1234:1234 -v //c/Users/username/directory:/src -d test/test
My node server seems to start up fine (no errors in the log), but it takes significantly longer to do so, and when I try to access my webpage in browser I just get a blank page.
Am I doing something incorrectly?
EDIT: I have gotten my server to run--seems to have been a weird error in my configuration. However, it still takes a long time (around a minute or two) for my server to start when I mount a volume from my host machine. Does anyone have some insight as to why this is?
So I have another follow-up question regarding installing a node.js-based framework under Docker on CoreOS, per this post.
So, because npm is finicky about installing from package.json via root, I've had to create a nonroot sudo user in order to install the package. This is what my Dockerfile currently looks like inside of our repo, building off of an ubuntu image:
# Install dependencies and nodejs
RUN apt-get update
RUN apt-get install -y python-software-properties python g++ make
RUN add-apt-repository ppa:chris-lea/node.js
RUN apt-get update
RUN apt-get install -y nodejs
# Install git
RUN apt-get install -y git
# Bundle app source
ADD . /src
# Create a nonroot user, and switch to it
RUN /usr/sbin/useradd --create-home --home-dir /usr/local/nonroot --shell /bin/bash nonroot
RUN /usr/sbin/adduser nonroot sudo
RUN chown -R nonroot /usr/local/
RUN chown -R nonroot /usr/lib/
RUN chown -R nonroot /usr/bin/
RUN chown -R nonroot /src
USER nonroot
# Install app source
RUN cd /src; npm install
I know that this is an inelegant way to do things, but I'm otherwise at a loss as to how to complete an npm install here. When I try the above, I get errors on all of the packages as they try to install:
Error: Attempt to unlock javascript-brunch#1.7.1, which hasn't been locked
at unlock (/usr/lib/node_modules/npm/lib/cache.js:1304:11)
at cb (/usr/lib/node_modules/npm/lib/cache.js:646:5)
at /usr/lib/node_modules/npm/lib/cache.js:655:20
at /usr/lib/node_modules/npm/lib/cache.js:1282:20
at afterMkdir (/usr/lib/node_modules/npm/lib/cache.js:1013:14)
at /usr/lib/node_modules/npm/node_modules/mkdirp/index.js:37:53
at Object.oncomplete (fs.js:107:15)
If you need help, you may report this *entire* log,
including the npm and node versions, at:
<http://github.com/npm/npm/issues>
...
Any thoughts as to how I should modify my Dockerfile? I can only assume that this is some permissioning issue having to do with the way that I've provisioned the nonroot user above that might be particular to the Docker framework; I have no problem doing this kind of thing on just a vanilla ubuntu install, albeit not from script.
so it turns out that this may be an issue with docker.
was able to get around this by switching from USER nonroot to RUN /bin/su nonroot instead, afterwards everything worked fine.
I was getting the similar mistakes when trying to create any yeoman project and finnally found the solution :)
I was getting that error because the owner of .npm folder in my home directory was "root" user so I used
sudo chown [username] .npm
and now I can use Yeoman and npm without errors :)
Hope it helps!
I have a couple of suggestions for modifications/suggestions:
Cache your node modules earlier in the Dockerfile using something like this (put this right after the apt-get:
ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /src && cp -a /tmp/node_modules /src
This way if your application code changes you aren't rebuilding all your node modules each time. Put this before your ADD . /src. More details/examples are available in my blog post article.
You shouldn't need to worry about running things as root in your Dockerfile... that is the default. Perhaps your problem isn't related to root but to the content inside your host's directory. Do you maybe need to clean out a lockfile from your code's dir?