Installing Node.js modules from file system on Azure Websites - node.js

I have a Node.js app running as an Azure Website and doing my deployments using Git. The app requires a module that's private (i.e. written by me, no relevance to anyone else and thus not available in NPM) and common to my other projects. Let's say the module is located in ./lib/mymodule and it has its own dependencies in its package.json file. The problem is I cannot figure out how to have Azure install the dependencies of my own module. I would like to avoid having to add my node_modules directory under version control (currently .gitignor'd) or having to add the module's dependencies as my app's dependencies (which is ugly and a bit inconvenient).
It's trivial to get this working on my local dev environment. The first thing I did was simply include my own module as a local dependency in my package.json as described in https://docs.npmjs.com/files/package.json (see section "Local Paths"). A simple npm install would now install my own module nice and easy. Since I know that Azure runs npm install every time I deploy a new version, I figured this is all I need. Turns out I was wrong: local paths in package.json dependencies section were only introduced in NPM 2.0.0 and Azure is running NPM 1.4.x, so the deployment obviously failed. Ok then, can I run a different version of NPM in Azure? In theory, yes, but practice seems to be something else. I tried out https://github.com/glennblock/azure-node-runtime-selector but to no avail. Here's some of the deployment output:
remote: Handling node.js deployment.
remote: KuduSync.NET from: 'D:\home\site\repository' to: 'D:\home\site\wwwroot'
remote: ............
remote: node version: 0.11.14
remote: ..............
remote: 2.1.18
remote: npm version: 1.4.6
remote: An error has occured during web site deployment.
remote: npm failed
remote:
remote: Error - Changes committed to remote repository but deployment to website failed.
Apparently it fails to install any NPM version >= 2.0.0.
Ok then, this is not so bad, I figured. I can work with an older version of NPM, I just need to run another npm install to install my own module, as NPM documentation hints. So I took the local dependency away from my package.json and simply run npm install ./lib/mymodule. Nice and easy... on my local. I dug up instructions how to run custom commands on deployment in Azure: just run azure site deploymentscript --node and edit deploy.sh it creates. Just to ensure the custom deployment script works at all, I made one deployment without editing anything. Worked ok. Then I added a line in deploy.sh (around line 111):
# 3. Install npm packages
if [ -e "$DEPLOYMENT_TARGET/package.json" ]; then
cd "$DEPLOYMENT_TARGET"
eval $NPM_CMD install --production
eval $NPM_CMD install ./lib/mymodule #This line added by me
exitWithMessageOnError "npm failed"
cd - > /dev/null
fi
Nope. This is what I get:
remote: Installing npm packages...
remote: npm ERR! addLocal Could not install ./lib/mymodule
remote: An error has occurred during web site deployment.
remote: npm ERR! Error: ENOENT, stat 'd:\home\site\wwwroot\lib\mymodule'
remote: npm failed
...snip...
remote:
remote: npm ERR! System Windows_NT 6.2.9200
remote: npm ERR! command "d:\\Program Files (x86)\\nodejs\\0.10.28\\node.exe" "D:\\Program Files(x86)\\npm\\1.4.9\\node_modules\\npm\\bin\\npm-cli.js" "install" "./lib/mymodule"
remote: npm ERR! cwd d:\home\site\wwwroot
remote: npm ERR! node -v v0.10.28
remote: npm ERR! npm -v 1.4.9
remote: npm ERR! path d:\home\site\wwwroot\lib\mymodule
remote: npm ERR! code ENOENT
remote: npm ERR! errno 34
remote: npm
remote:
remote: Error - Changes committed to remote repository but deployment to website failed.
This is where I got stuck. I tried playing around with the path, guessing that Windows is mangling it somehow, but haven't found a working solution yet. The weird thing is that ./lib/mymodule would seem to resolve to d:\home\site\wwwroot\lib\mymodule which look just fine to me and DOES exist, at least after deployment when I FTP'd in to take a look.
I would appreciate any and all help in either getting NPM 2.x.x working or my making custom deployment go through. Or maybe there's an alternative to managing my local modules I haven't thought of? And if all else fails, maybe I'll just commit my node_modules or manage my dependencies manually. I've already used way too much time to resolve what first seemed like a trivial issue. :)

Azure WebSites team will be shipping support for npm 2.1.17 out-of-the box by end of the month (Jan'15). Hope this will help you unblock you.

Related

Build fails with "host key verification failed" while installing node_modules

UPDATE Clearing my build cache and deploying with no package-lock.json at all fixed.
Deploying my Node app to Heroku has started failing with Host key verification failed at installation of one of my dependencies, a public repository:
-----> Monorepo app detected
Copied /server to root of app successfully
-----> Node.js app detected
-----> Creating runtime environment
NPM_CONFIG_LOGLEVEL=error
NODE_ENV=DEVELOPMENT
NODE_MODULES_CACHE=true
NODE_VERBOSE=false
-----> Installing binaries
engines.node (package.json): 10.x
engines.npm (package.json): unspecified (use default)
Resolving node version 10.x...
Downloading and installing node 10.22.1...
Using default npm version: 6.14.6
-----> Restoring cache
- node_modules
-----> Installing dependencies
Installing node modules
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git#github.com/ethereumjs/ethereumjs-abi.git
npm ERR!
npm ERR! Host key verification failed.
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR!
npm ERR! exited with error code: 128
npm ERR! A complete log of this run can be found in:
npm ERR! /tmp/npmcache.RchR7/_logs/2020-10-25T17_11_28_328Z-debug.log
-----> Build failed
There's been no change to my buildpacks, which were working before:
1. https://github.com/lstoll/heroku-buildpack-monorepo
2. https://github.com/heroku/heroku-buildpack-nodejs
App has been running on Heroku for weeks, using the packages that ethereumjs-abi is a dependency of. (It isn't a direct dependency of my app.)
ethereumjs-abi hasn't updated since I started using it
App still runs correctly locally
My own repo is private, but the package repo is public.
I changed my SSH keys recently, but since adding them to GitHub haven't had any other SSH issues. I've now added the new key to Heroku with heroku keys:add and it had no effect.
Previously I was specifying engines: { node: 10.x }, but have now also tried 12.x and 14.x
Pushing from local master to Heroku master
Yeah a few days wasted so far...
Faced with the same issue when replaced npm package with my own fork and provided custom link on github.com.
What helped:
remove package-lock from GIT
clear cache with heroku builds:cache:purge --app $HEROKU_APP_NAME --confirm $HEROKU_APP_NAME

HEROKU : node-pre-gyp install --fallback-to-build

I had this problem since I used #discordjs/opus, I saw a big bunch of similar problems everywhere, and nothing helped me changing this error. It always concern node-pre-gyp. I first tried to put it manually in my package.json, and the build suceeded, but there was no sound siplayed, and a lot of warning messages in the build log... Also some people had issues with bcrypt, but it wasn't the case here...
Thanks for your help!
-----> Creating runtime environment
NPM_CONFIG_LOGLEVEL=error
NODE_ENV=production
NODE_MODULES_CACHE=true
NODE_VERBOSE=false
-----> Installing binaries
engines.node (package.json): unspecified
engines.npm (package.json): unspecified (use default)
Resolving node version 12.x...
Downloading and installing node 12.18.4...
Using default npm version: 6.14.6
-----> Restoring cache
- node_modules is checked into source control and cannot be cached
-----> Installing dependencies
Prebuild detected (node_modules already exists)
Rebuilding any native modules
> #discordjs/opus#0.3.2 install /tmp/build_2510a47a_/node_modules/#discordjs/opus
> node-pre-gyp install --fallback-to-build
sh: 1: node-pre-gyp: Permission denied
npm ERR! code ELIFECYCLE
npm ERR! errno 126
npm ERR! #discordjs/opus#0.3.2 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 126
npm ERR!
npm ERR! Failed at the #discordjs/opus#0.3.2 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /tmp/npmcache.N30Iz/_logs/2020-09-28T09_36_00_646Z-debug.log
-----> Build failed
We're sorry this build is failing! You can troubleshoot common issues here:
https://devcenter.heroku.com/articles/troubleshooting-node-deploys
Some possible problems:
- node_modules checked into source control
https://devcenter.heroku.com/articles/node-best-practices#only-git-the-important-bits
- Node version not specified in package.json
https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version
Love,
Heroku
! Push rejected, failed to compile Node.js app.
! Push failed`
Well, as JM-AGMS said, the solution was to make visible package lock, type the node engine in the package, and remove the node modules folder in order to have a succeeded build.
However, the error about node pre-gyp is still there.

How to update npm on an OpenShift gear?

How can I update npm on OpenShift?
I'm having problems while deploying because npm started using ^1.2.3 version notations and it's not compatible with the current npm in my application:
remote: npm ERR! Error: No compatible version found: through#'^2.3.4'
remote: npm ERR! Valid install targets:
remote: npm ERR! ["0.0.1","0.0.2","0.0.3","0.0.4","0.1.0","0.1.1","0.1.2","0.1.3","0.1.4","1.0.0","1.1.0","1.1.1","1.1.2","2.0.0","2.1.0","2.2.0","2.2.1","2.2.2","2.2.4","2.2.5","2.2.6","2.2.7","2.3.1","2.3.2","2.3.3","2.3.4"]
Is there a way of fixing this, or I'll have to go back to outdated packages?
OpenShift does not provide root access to developers, but you can still select a custom version of npm by running your own nodejs binary in user space.
Developers can also package up their own custom nodejs cartridge, allowing teams to define and standardize their dependencies in a reusable way.
Here is an answer that helps you run a custom version of Nodejs on OpenShift
You can also try working with user-defined npm globals on OpenShift

Azure Website: Kudu fails to install

I started to run into this problem yesterday. Deploys fail when I run:
npm install kudusync -g
with the following error:
Error: SSL Error: CERT_UNTRUSTED
From some searches it appears that one of Kudu's dependencies is self-signed or unsigned. It seems like I can disable SSL checks for Node -- but that seems like a dangerous option and not recommended. Is anyone aware of any other workarounds?
The output in azure is as follows:
remote: npm http GET https://registry.npmjs.org/kudusync[K
remote: ......[K
remote: npm http GET https://registry.npmjs.org/kudusync[K
remote: .......................................................[K
remote: npm http GET https://registry.npmjs.org/kudusync[K
remote: npm ERR! Error: SSL Error: CERT_UNTRUSTED[K
remote: npm ERR! at ClientRequest. (d:\Program Files (x86)\nodejs\node_modules\npm\node_modules\request\main.js:440:26)[K
remote: An error has occurred during web site deployment.[K
The break most likely refers to this change npm made.
Do you have a custom deployment script, or the standard script? With custom, your options are:
use a newer version of npm
run npm config set strict-ssl false before running npm
if your custom script has a reference to KUDU_SYNC_COMMAND, rename that to KUDU_SYNC_CMD

Fail to deploy node.js application to heroku

I am trying to deploy a simple node.js express-based application to heroku, something which is apparently very basic: https://devcenter.heroku.com/articles/nodejs
Here is my package.json:
{
"name": "cours-lic3-blois",
"version": "0.0.1",
"private": true,
"scripts": {
"start": "node app"
},
"dependencies": {
"express": "*",
"ejs": "*",
"github-flavored-markdown": "*",
"less-middleware": "*"
},
"engines": {
"node": "0.8.8",
"npm": "1.1.65"
}
}
When I git push heroku master I got the following trace:
-----> Heroku receiving push
-----> Node.js app detected
-----> Resolving engine versions
Using Node.js version: 0.8.8
Using npm version: 1.1.65
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
npm ERR! Error: ENOENT, chmod '/tmp/build_1suuxlhd9s8n6/node_modules/express/bin/express'
npm ERR! If you need help, you may report this log at:
npm ERR! <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR! <npm-#googlegroups.com>
npm ERR! System Linux 2.6.32-348-ec2
npm ERR! command "/tmp/node-node-tonf/bin/node" "/tmp/node-npm-NG88/cli.js" "rebuild"
npm ERR! cwd /tmp/build_1suuxlhd9s8n6
npm ERR! node -v v0.8.8
npm ERR! npm -v 1.1.65
npm ERR! path /tmp/build_1suuxlhd9s8n6/node_modules/express/bin/express
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /tmp/build_1suuxlhd9s8n6/npm-debug.log
npm ERR! not ok code 0
! Failed to rebuild dependencies with npm
! Heroku push rejected, failed to compile Node.js app
To git#heroku.com:fast-everglades-2007.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git#heroku.com:fast-everglades-2007.git'
I tried to tweak various versions in my package.json but to no avail. I am developing on windows and it might be possible this ENOENT issue is due to some filemode issue.
I got this fixed by:
Making sure Procfile is committed into git
Removing the node_modules/ folder and committing that into git (git rm -r node_modules/)
Afterwards, I did the git push heroku master then the error disappeared.
I had this problem, and it was because:
I keep node_modules in version control
I had bin in my .gitignore file
npm was attempting to chmod express/bin/express, but due to my .gitignore this file wasn't in git and thus was not being cloned during the deploy, so it failed. I didn't notice it because a local npm install would create the bin/express file as usual.
Removing bin from .gitignore and committing the missing files solved the problem for me.
Heroku team, could you please consider configuring npm to use npm install --no-bin-links --production by default, or creating an environment variable to let users set that flag. This is a serious bug in node installs. Removing the bin directory from my .gitignore (as suggested below) enabled me to deploy, but it breaks using git effectively in a cross-platform development environment, requiring an explicit npm rebuild on every git pull where node_modules might have changed.
NPM best practice is to avoid having node_modules in .gitignore (see http://www.futurealoof.com/posts/nodemodules-in-git.html ).
I believe --no-bin-links will provide the best of both worlds, enabling deploys of node_modules where the bin directory has been excluded, supporting npm rebuild, but not failing when a bin link is created.
I submitted a pull request here: https://github.com/heroku/heroku-buildpack-nodejs/pull/33
Thanks!
similar issue was fixed by renaming gruntfile.js to Gruntfile.js
I had a similar problem a few days ago, I think it was because I imported a module created by me and that was what gave the error;
I corrected it like this:
I called the module "path"
const path = require ("path")
and then where my module was imported I made a join of the file path
const myModule = require (path.join (__ dirname, 'MyModule.js'))

Resources