Grunt.js - Removing/Cleaning folder on remote server - node.js

In my project, I have two servers: Development and Production. I am managing static files (CSS/JS, etc) with Git, and DB deployment with Grunt. But after deploying the database, I need to remove Cache folder from my Production server. How can I do it with Grunt?
And, by the way, can I manage my files without Git using only Grunt?
Thanks in advance.

As I thought, this was really easy:
For this king of task, all you need is grunt-shell and grunt-ssh packages. I faced only one problem with this - SSH was refusing connections because of ssh-agent was not active at the moment. Here is the sample code for pulling the git commits to remote server and deploying the database:
shell: {
git: {
command: ['eval `ssh-agent -s`', 'ssh-add ~/.ssh/yourKey.pem', 'grunt sshexec:gitpull'].join(' && ')
},
db: {
command: ['eval `ssh-agent -s`', 'ssh-add ~/.ssh/yourKey.pem', 'grunt db_push', 'grunt sshexec:clear'].join(' && ')
}
},
sshexec: {
gitpull: {
command: ['cd /var/www/', 'sudo -u yourSudoUser git pull --no-edit'].join("&&"),
options: {
host: 'youHost.com',
username: 'username',
agent: process.env.SSH_AUTH_SOCK
}
},
clearCache: {
command: ['cd /var/www/core', 'sudo rm -rf cache'].join("&&"),
options: {
host: 'yourHost.com',
username: 'username',
agent: process.env.SSH_AUTH_SOCK
}
}
}
--no-edit - if not set, git providing a window from GNU nano, where you must edit your commit message. This window cannot be closed, because Nano shortcuts will not work in current session.
'eval ssh-agent -s', 'ssh-add ~/.ssh/yourKey.pem' - starts SSH-agent and adding you keyPair. NB! Notice, that grunt sshexec:gitpull executing within the shell task, after ssh-agent starts. Otherwise you will not reach ssh-agent when executing sshexec in a separate task.
'grunt db_push' - task for grunt-deployments module.
One more thing: Consider updating Grunt and npm to the latest versions with npm update npm -g and npm install grunt#0.4.4 -g. After update this tasks went really smooth.

Related

npm: not found on jenkins agent, but available through ssh

I am trying to set up a jenkins pipeline that utilizes multiple agents. The agents are ubuntu instances living in a cloud tenancy (openstack). When trying to run some npm commands on some of the instances, I am getting the error npm: not found. I've read multiple other threads, but I am struggling to understand why npm might not be found. I set these instances up myself, and I know I installed all requirements, including node and npm.
Let's say I have 2 nodes - agent1 at IP1, and agent2 at IP2. They both have a user login with username cooluser1. When I do an ssh cooluser1#IP1 or ssh cooluser1#IP2, in either case, running npm -v gives me a proper node version (6.14.13). However, in my pipeline, npm is not found in the IP2 instance. Here's my pipline script:
pipeline {
agent {
node {
label 'agent1'
}
}
stages {
stage('Build'){
steps {
sh 'hostname -I'
sh 'echo "$USER"'
sh 'echo "$PATH"'
sh 'npm -v'
}
}
stage ('Run Tests'){
parallel {
stage('Running tests in parallel') {
agent {
node {
label 'agent2'
}
}
steps {
sh 'hostname -I'
sh 'echo "$USER"'
sh 'echo "$PATH"'
sh 'npm -v'
}
}
stage {
// more stuff running on another agent (agent3)
}
}
}
}
}
As you can see, in both the main agent agent1, and in the parallel stages, I run the same code, which checks the host IP, the username, the path, and the npm version. The IPs are as expected - IP1 and IP2. The $USER in both cases is indeed cooluser1. The path looks something like this:
// agent1
+ echo
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
// agent2
+ echo
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
A bit strange, but identical in both cases.
However, when I get to npm --v, for agent1, I get a version number, and any npm commands I want to run are workoing. but in agent2, I get npm: not found, and the pipeline fails if I try to use any npm commands. The full error is here:
+ npm -v
/home/vine/workspace/tend-multibranch_jenkins-testing#tmp/durable-d2a0251e/script.sh: 1: /home/vine/workspace/tend-multibranch_jenkins-testing#tmp/durable-d2a0251e/script.sh: npm: not found
But I clearly saw with ssh cooluser1#IP2 that npm is available in that machine to that user.
What might be going wrong here?
I will propose to you to install nodejs plugin, configure any nodejs version you want in 'manage jenkins' -> 'global tools configurations' and set nodejs in pipeline:
pipeline {
agent any
tools {
nodejs 'NodeJS_14.17.1'
}
stages {
stage ('nodejs test') {
steps {
sh 'npm -v'
}
}
}
}

Node.js Child Processes failing GitHub ssh authentication

Goal
Hello, I am creating a Node.js application to update my code when a push is made automatically.
Problem
Everything works in it, except, the actual git pull. The repo is private and needs to use ssh, however when I use the same command in terminal it works. I have keychaining on so it doesn't ask for my passphrase. Any ideas and how to fix this?
Relevant code
const exec = require("child_process").exec;
exec('cd ' + repo + ' && git pull origin deployment', (egitpull,stdoutgitpull,stderrgitpull)=>{
if(egitpull) return console.error(`git pull exec error:${egitpull}`)
console.log(`git pull stdout: ${stdoutgitpull}`);
console.log(`git pull stderr: ${stderrgitpull}`);
Command run:
cd /mp/ && git pull origin deployment
Out of child process vrs. in child process
Edit:
Removing the passphrase from the key entirely does seem to solve the issue, but I would much prefer having it in there for security reasons.
If you are using an ssh-agent for your ssh keys, try forwarding the SSH_AUTH_SOCK and SSH_AGENT_PID env variables to the child process, like this:
exec('cd ' + repo + ' && git pull origin deployment',
{
env: {
SSH_AUTH_SOCK: process.env.SSH_AUTH_SOCK,
SSH_AGENT_PID: process.env.SSH_AGENT_PID
}
},
(egitpull,stdoutgitpull,stderrgitpull) => {
// ...
});

Jenkins - env: ‘node’: No such file or directory

I have a jenkins server that is configured using
https://github.com/shierro/jenkins-docker-examples/tree/master/05-aws-ecs
I am running a blue ocean pipeline using a simple Jenkinsfile and the jenkins NodeJS plugin
pipeline {
agent any
tools {
nodejs 'node10'
}
stages {
stage ('Checkout Code') {
steps {
checkout scm
}
}
stage ('Install dependencies') {
steps {
sh "echo $PATH"
sh "npm install"
}
}
}
}
I made sure to add the node10 global tool as well w/c is used above
When the pipeline gets to the script sh "npm install" i am running through this error
this is the output of the command echo $PATH
so i think it's not a path issue
Also, it also wasn't able to add the global package
More info that might help:
Docker Jenkins server: FROM jenkins/jenkins:2.131-alpine
Blue ocean version: 1.7.0
NodeJS Plugin: 1.2.6
Multiple server restarts already
Any ideas why the jenkins server does not know where node is?
Big thanks in advance!
Thanks to #JoergS for some insight! The culprit in this case is: using alpine image as the docker base. So switching from jenkins/jenkins:2.131-alpine to jenkins/jenkins:2.131 solved the NodeJS plugin issue.
I have faced the same issue with jenkinsci/blueocean. I have resolved this by installing nodejs with below command(inside docker) not as jenkins plugin
apk add nodejs
I have faced the same issue with jenkinsci/blueocean. No jenkins nodejs plugin needed.
pipeline {
agent any
stages {
stage ('Checkout Code') {
steps {
checkout scm
}
}
stage ('Install dependencies') {
steps {
sh "apk add nodejs"
sh "echo $PATH"
sh "npm install"
}
}
}
}
Make a symbolic link like this:
sudo ln -s /var/lib/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/node/bin/node /usr/bin/node
I want to highlight Mitch Downey's comment, it can't be just a comment because after spending 4 hours with no solution this comment helped me to resolve the solution
My issue ended up being with the jenkinsci/blueocean image. I was able
to just replace that image with jenkins/jenkins:lts and the NodeJS
plugin began working as expected

Hexo deploy on github

I tried to deploy the Hexo on my GithubPage.
The generate process looks fine, but error happens when I deploy it on my GithubPage.
Here's the deployment part in _config.yml:
# Deployment
## Docs: https://hexo.io/docs/deployment.html
deploy:
type: git
repo: https://github.com/ZhangYuef/ZhangYuef.github.io.git
# branch: Hexo
Generate
Deployment
So what's going on there?
Thx for help! :)
The context you provided in the question is not sufficient...
But according to invalid chars on the screenshot, I suppose that your Chinese file path may be the cause.
References:
Node JS Error: ENOENT
Why does ENOENT mean "No such file or directory"?
try to update the _config.yml like this:
deploy:
type: git
repository: https://github.com/fakeYanss/fakeYanss.github.io.git
branch: master
yaml is very very very strict, and indent is important!
Not sure what reason causing this error.
Check your environment whether these things have been set up.
I think it might be your config type is wrong.
npm install hexo-deployer-git --save
git repository settings like
deploy:
- type: git
repo: git#github.com:xxx.git
branch: master
- type: git
repo: git#github.com:xxx.git
branch: src
extend_dirs: /
ignore_hidden: false
ignore_pattern:
public: .
By this way, you can not only deploy your blog, but also backup your blog files, which you can use the command git pull to get the blog files on another machine.
- set up your ssh
ssh-keygen -t rsa -C "yourEmail#icloud.com"
ssh-agent -s
chmod id_rsa 600
ssh-add id_rsa
(you need to add the id_rsa.pub to the github's deployer key)
ssh -T git#github.com
sometimes it maybe you have several gits, make deployer confused.
Try delete .git directory and make sure there is no any git in other directories.
encoding. It could be the encoding is different. In my case, I make all the files belong to UTF-8.
By the way, it could be your files' error.try npm install hexo-server --save and hexo server to detect whether the website can be deployed.
(http://localhost:4000/xx)

avoid rebuilding node_modules in elastic beanstalk

We have a fairly simple node.js app, but due to AWS Elastic Beanstalk deployment mechanism, it takes about 5 minutes to roll-out a new version (via git aws.push) even after a single file commit.
I.e. the commit itself (and upload) is fast (only 1 file to push), but then Elastic Beanstalk fetches whole package from S3, unzips it and runs npm install, which causes node-gyp to compile some modules. Upon installation/building completion, Elastic Beanstalk wipes /var/app/current and replaces it with the new app version.
Needless to say, constant node_modules rebuilding is not necessary, and rebuilding that takes 30 seconds on my old Macbook Air, takes >5 mins on a ec2.micro instance, not fun.
I see two approaches here:
tweak /opt/containerfiles/ebnode.py and play with node_modules location to avoid its removal and rebuilding upon deployment.
set up a git repo on Elastic Beanstalk EC2 instance and basically re-write deployment procedure ourselves, so /var/app/current receives pushes and runs npm install only when necessary (which makes Elastic Beanstalk to look like OpsWorks..)
Both options lack grace and are prone to issues when Amazon updates their Elastic Beanstalk hooks and architecture.
Maybe somebody has a better idea how to avoid constant rebuilding of node_modules that are already present in the app dir? Thank you.
Thanks Kirill, it was really helpful !
I'm just sharing my config file for people who just look the simple solution to the npm install. This file needs to be placed in the .ebextensions folder of the project, it is lighter since it doesn't include last version of node installation, and ready to use.
It also dynamically checks the node version installed, so no need for it to be included in the env.vars file.
.ebextensions/00_deploy_npm.config
files:
"/opt/elasticbeanstalk/env.vars" :
mode: "000775"
owner: root
group: users
content: |
export NPM_CONFIG_LOGLEVEL=error
export NODE_PATH=`ls -td /opt/elasticbeanstalk/node-install/node-* | head -1`/bin
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
mode: "000775"
owner: root
group: users
content: |
#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
ln -s /var/node_modules /tmp/deployment/application/
fi
OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && $NODE_PATH/npm install 2>&1) || error_exit "Failed to run npm install. $OUT" $?
echo $OUT
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
mode: "000666"
owner: root
group: users
content: |
#no need to run npm install during configdeploy
25/01/13 NOTE: updated scripts to run npm -g version upgrade (only once, on initial instance roll out or rebuild) and to avoid NPM operations during EB configuration change (when app dir is not present, to avoid error and to speed up configuration updates).
Okay, Elastic Beanstalk behaves dodgy with recent node.js builds (including presumably supported v.0.10.10), so I decided to go ahead and tweak EB to do the following:
to install ANY node.js version as per your env.config (including
the most recent ones that are not yet supported by AWS EB)
to avoid rebuilding existing node modules, including in-app
node_modules dir
to install node.js globally (and any desired module as well).
Basically, I use env.config to replace deploy&config hooks with customized ones (see below). Also, in a default EB container setup some env variables are missing ($HOME for example) and node-gyp sometimes fails during rebuild because of it (took me 2 hours of googling and reinstalling libxmljs to resolve this).
Below are the files to be included along with your build. You can inject them via env.config as inline code or via source: URL (as in this example)
env.vars (desired node version & arch are included here and in env.config, see below)
export HOME=/root
export NPM_CONFIG_LOGLEVEL=error
export NODE_VER=0.10.24
export ARCH=x86
export PATH="$PATH:/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/:/root/.npm"
40install_node.sh (fetch and ungzip desired node.js version, make global symlinks, update global npm version)
#!/bin/bash
#source env variables including node version
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#UNCOMMENT to update npm, otherwise will be updated on instance init or rebuild
#rm -f /opt/elasticbeanstalk/node-install/npm_updated
#download and extract desired node.js version
OUT=$( [ ! -d "/opt/elasticbeanstalk/node-install" ] && mkdir /opt/elasticbeanstalk/node-install ; cd /opt/elasticbeanstalk/node-install/ && wget -nc http://nodejs.org/dist/v$NODE_VER/node-v$NODE_VER-linux-$ARCH.tar.gz && tar --skip-old-files -xzpf node-v$NODE_VER-linux-$ARCH.tar.gz) || error_exit "Failed to UPDATE node version. $OUT" $?.
echo $OUT
#make sure node binaries can be found globally
if [ ! -L /usr/bin/node ]; then
ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/node /usr/bin/node
fi
if [ ! -L /usr/bin/npm ]; then
ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm /usr/bin/npm
fi
if [ ! -f "/opt/elasticbeanstalk/node-install/npm_updated" ]; then
/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/ && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm update npm -g
touch /opt/elasticbeanstalk/node-install/npm_updated
echo "YAY! Updated global NPM version to `npm -v`"
else
echo "Skipping NPM -g version update. To update, please uncomment 40install_node.sh:12"
fi
50npm.sh (creates /var/node_modules, symlinks it to app dir and runs npm install. You can install any module globally from here, they will land in /root/.npm)
#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
eventHelper.py --msg "$1" --severity ERROR
exit $2
}
#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
ln -s /var/node_modules /tmp/deployment/application/
fi
OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm install 2>&1) || error_exit "Failed to run npm install. $OUT" $?
echo $OUT
env.config (note node version here too, and to be safe, put desired node version in env config in AWS console as well. I'm not certain which of these settings will take precedence.)
packages:
yum:
git: []
gcc: []
make: []
openssl-devel: []
option_settings:
- option_name: NODE_ENV
value: production
- option_name: RDS_HOSTNAME
value: fill_me_in
- option_name: RDS_PASSWORD
value: fill_me_in
- option_name: RDS_USERNAME
value: fill_me_in
- namespace: aws:elasticbeanstalk:container:nodejs
option_name: NodeVersion
value: 0.10.24
files:
"/opt/elasticbeanstalk/env.vars" :
mode: "000775"
owner: root
group: users
source: https://dl.dropbox.com/....
"/opt/elasticbeanstalk/hooks/configdeploy/pre/40install_node.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
"/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
"/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
mode: "000666"
owner: root
group: users
content: |
#no need to run npm install during configdeploy
"/opt/elasticbeanstalk/hooks/appdeploy/pre/40install_node.sh" :
mode: "000775"
owner: root
group: users
source: https://raw.github.com/....
There you have it: on t1.micro instance deployment now takes 20-30 secs instead of 10-15 minutes! If you deploy 10 times a day, this tweak will save you 3 (three) weeks in a year.
Hope it helps and special thanks to AWS EB staff for my lost weekend :)
There's npm package that's overwriting default EB behaviour for npm install command by truncating following files:
/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh
/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh
https://www.npmjs.com/package/eb-disable-npm
Might be better than just copying script from SO, since this package is maintained and probably will be updated when EB behaviour will change.
I've found a quick solution to this. I looked through the build scripts that Amazon are using and they only run npm install if package.json is present. So after your initial deploy you can change it to _package.json and npm install won't run anymore! It's not the best solution but it's a quick fix if you need one!
I had 10+ minute builds when I would deploy. The solution was much simpler than others have came up with... Just check node_modules into git! See http://www.futurealoof.com/posts/nodemodules-in-git.html for the reasoning

Resources