Vendor File not found during node-gyp build - node.js

I am willing to install a custom build of libvips with sharp on heroku. I use buildpacks that run without errors, but when it comes to node-gyp compilation, it fails here saying it cannot access /app/vendor/vips/lib/libvips-cpp.so. When I run the build without installing sharp and then connect with heroku run bash, I can see that the file is at this location.
Why node-gyp cannot find it then ?
My buildpacks
=== my-site Buildpack URLs
1. heroku-community/apt
2. https://github.com/lechinoix/heroku-buildpack-vips
3. heroku/nodejs
The error
remote: -----> vips (heroku-20 stack) app detected
remote: -----> Vendoring binaries
remote: Fetching /tmp/codon/tmp/buildpacks/2ef659928923e1b5ae4bf85b886e53d084c6a3d4/build/heroku-20.tar.gz
remote: Changing permissions on libvips folder
remote: -----> Configuring build environment
remote: -----> Building runtime environment
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_VERBOSE=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): unspecified
remote: engines.npm (package.json): unspecified (use default)
remote:
remote: Resolving node version 14.x...
remote: Downloading and installing node 14.18.1...
remote: Using default npm version: 6.14.15
remote:
remote: -----> Restoring cache
remote: Caching has been disabled because NODE_MODULES_CACHE=false
remote:
remote: -----> Installing dependencies
remote: Installing node modules (package.json)
remote:
remote: > sharp#0.28.1 install /tmp/build_1f4c5c03/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: sharp: Detected globally-installed libvips v8.10.6
remote: sharp: Building from source via node-gyp
remote: readelf: Error: '/app/vendor/vips/lib/libvips-cpp.so': No such file
remote: gyp: Call to 'if readelf -Ws "$(PKG_CONFIG_PATH="/tmp/build_1f4c5c03/.apt/usr/lib/x86_64-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/i386-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/pkgconfig::/tmp/build_1f4c5c03/vendor/vips/lib/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/x86_64-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/i386-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/pkgconfig::/usr/local/lib/pkgconfig:/usr/lib/pkgconfig" pkg-config --variable libdir vips-cpp)/libvips-cpp.so" | c++filt | grep -qF __cxx11;then echo "1";else echo "0";fi' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
Thank you so much for your help !

Finally found that heroku is creating the whole /app folder in a directory /tmp/build_1f4c5c03 that also contains the freshly installed vendors. In my .pc files, the package is located in /app so it cannot be found. I used a heroku-prebuild npm script to sed inplace the paths :
TMP_DIR=$(pwd)
for file in $(grep -rl /app/vendor ${TMP_DIR}/vendor/vips/lib/pkgconfig)
do
sed -i "s+/app/vendor+${TMP_DIR}/vendor+g" $file
done
Not perfectly sure it is the best way to do...

Related

Heroku - EmberJS - Deploy fail ( Unable download node )

when I try to push code on Heroku, the build/deploy fail because of node problem.
Console log
cd dungeon-diary
heroku buildpacks:set https://codon-buildpacks.s3.amazonaws.com/buildpacks/heroku/emberjs.tgz
git push heroku master
Output console with error
remote: Building source:
remote:
remote: -----> emberjs app detected
remote: -----> Setting NPM_CONFIG_PRODUCTION to false to install ember-cli toolchain
remote: -----> Fetching buildpack heroku/nodejs-v98
remote: -----> Node.js detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NPM_CONFIG_PRODUCTION=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote: NODE_VERBOSE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): 10.* || >= 12
remote: engines.npm (package.json): 6.x
remote:
remote: Resolving node version 10.* || >= 12 via semver.io...
remote: Downloading and installing node 14.7.0...
remote: Unable to download node 14.7.0; does it exist?**
remote:
remote: -----> Build failed
package.json
"engines": {
"node": "10.* || >= 12"
},
The problem was...
in the log you can see, the console choose node 14.7.0 by itself,
but for Heroku it was unacceptable. So I had to explicitly write which version use ( to avoid 14.7.0 ) and it starts work :)
...weird for me why console choose the unsupported version :)

Failing to install Node.js dependencies on Heroku

I'm getting a memory error when trying to deploy a Node.js app to heroku. It doesn't seem to get past the installation section:
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote: NODE_VERBOSE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): unspecified
remote: engines.npm (package.json): unspecified (use default)
remote:
remote: Resolving node version 12.x...
remote: Downloading and installing node 12.13.1...
remote: Using default npm version: 6.12.1
remote:
remote: -----> Restoring cache
remote: Cached directories were not restored due to a change in version of node, npm, yarn or stack
remote: Module installation may take longer for this build
remote:
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote: npm ERR! Maximum call stack size exceeded
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /tmp/npmcache.TGwfg/_logs/2019-11-23T11_15_42_537Z-debug.log
I've tried turning the module cache off, which did nothing. I've also tried turning more verbose npm logging, but all that tells me is that the installation hasn't finished.
The application isn't particularly big, just an express application. There must be some dependency that has a large stack of pre/post install commands... I guess the only way to find out which one it is by uninstalling one at a time and redeploying :(
But before I go down that route, I thought I'd ask around to see if anyone else has experienced this before. I'm only using a free dyno at the moment to test if Heroku would be suitable.
I had this issue and #Rashomon's solution worked, but I had to type it thusly "engines": {"node": "14.16.1", "npm": "6.14.12"}, note the double quotes on everything.
To check your versions, use npm version and node -v in command line
Turns out that using yarn will help me avoid this problem.
rm package-lock.json
yarn
git add -A
git commit -m "chore: use yarn"
git push heroku master
If I decide to upgrade the service then I ask Heroku, directly, why I experienced this. Until then, this is an OK solution,

Sharp package Heroku deployment issue

I'm deploying my NodeJs app to Heroku. No issues running the app with nodemon locally.
However when I'm trying to push to Heroku Master, seems like "sharp" package is causing an issue:
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote:
remote: > sharp#0.22.0 install
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy &&
prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: info sharp Downloading
https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
remote:
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/install/libvips.js:86
remote: throw new Error(Status
${response.statusCode});
remote: ^
remote:
remote: Error: Status 403
remote: at
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/install/libvips.js:86:17
remote: at f
(/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/once/once.js:25:25)
remote: at ClientRequest.protocol.request.res
(/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/simple-get/index.js:63:5)
remote: at Object.onceWrapper (events.js:277:13)
remote: at ClientRequest.emit (events.js:189:13)
remote: at HTTPParser.parserOnIncomingClient [as
onIncoming] (_http_client.js:556:21)
remote: at HTTPParser.parserOnHeadersComplete
(_http_common.js:109:17)
remote: at TLSSocket.socketOnData (_http_client.js:442:20)
remote: at TLSSocket.emit (events.js:189:13)
remote: at addChunk (_stream_readable.js:284:12)
remote: make: Entering directory
'/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/build'
remote: TOUCH Release/obj.target/libvips-cpp.stamp
remote: CXX(target) Release/obj.target/sharp/src/common.o
remote: ../src/common.cc:25:10: fatal error: vips/vips8: No
such file or directory
remote: #include
remote: ^~~~~~~~~~~~
remote: compilation terminated.
remote: sharp.target.mk:128: recipe for target
'Release/obj.target/sharp/src/common.o' failed
remote: make: *** [Release/obj.target/sharp/src/common.o] Error
1
remote: make: Leaving directory
'/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/build'
Update:
Try removing sharp from package.json and add it back again using npm install sharp works this time... I still don't know why.
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote:
remote: > sharp#0.22.0 install
/tmp/build_5c8d3b01594e0b86f9d3e0d372534414/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy &&
prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: info sharp Downloading
https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
remote: added 57 packages from 92 contributors and audited
20393 packages in 10.683s
remote: found 62 low severity vulnerabilities
remote: run npm audit fix to fix them, or npm audit for
details
remote:
remote: -----> Build
remote:
remote: -----> Caching build
remote: - node_modules
remote:
remote: -----> Pruning devDependencies
remote: removed 606 packages and audited 497 packages in 8.181s
remote: found 0 vulnerabilities
remote:
remote:
remote: -----> Build succeeded!
There are many people with that problem, it's related to libvips as dependency not sharp by itself.
On github in this issue-comment the problem is explained:
The pre-built binaries of sharp are only guaranteed to work with the pre-built binaries of libvips.
If the download of a pre-built binary of libvips fails then sharp has to fall back to attempting to build itself from source.
http://sharp.pixelplumbing.com/en/stable/install/#pre-compiled-libvips-binaries provides information about serving the libvips binaries from a local URL.
The reason why the download is failing might be the cache, or a corrupted download like reported in the above linked issue on github:
I had to manually remove the ~/.npm/_libvips/libvips-8.7.0-linux-x64.tar.gz and try again. Seems to be downloaded a corrupted file.
Here is an interesting discussion concerning the same problem related to gatsby, a solution seems not yet existing even the issue was closed (before any comment was made): https://github.com/gatsbyjs/gatsby/issues/1754
However if the problem is only related to the download for some reason it's possible to deploy libvips locally like explained in the manual:
Pre-compiled libvips binaries
This module will attempt to download a pre-compiled bundle of libvips
and its dependencies on Linux and Windows machines under either of
these conditions:
If a global installation of libvips that meets the minimum version
requirement cannot be found;
If SHARP_IGNORE_GLOBAL_LIBVIPS environment variable is set.
SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install sharp
Should you need to manually download and inspect these files,
you can do so via https://github.com/lovell/sharp-libvips/releases
Should you wish to install these from your own location, set the
sharp_dist_base_url npm config option, e.g.
npm config set sharp_dist_base_url "https://hostname/path/"
npm install sharp
or set the SHARP_DIST_BASE_URL environment variable, e.g.
SHARP_DIST_BASE_URL="https://hostname/path/" npm install sharp
to use https://hostname/path/libvips-x.y.z-platform.tar.gz.
So this is a bit background, I'm aware that this can not really be seen as an answer with solution :/
If someone reaches here and didn't found an answer - my issue was that my project was using an old node version (8.x)
The solution was to install an older version of sharp - The latest one that support 8.x - "sharp": "0.23.4"

Heroku cedar node app - cannot push master after update to 6.11.1

After updating the node server to version 6.11.1 in response to the recent security update email from Heroku, I cannot git push my local master changes any more. The server and app restarted fine after the update.
I have tried deleted and reinstalled all node_modules, committing the changes and the app runs without errors locally but I get the same build failure when trying to deploy the master. The only change I've made to the package.json file is adding the npm version which was previously unspecified.
From the logs the server build fails when installing node modules:
remote: -----> Building dependencies
remote: Installing node modules (package.json)
remote: npm ERR! Cannot read property '0' of undefined
Is force pushing the best/a good solution?
This is the full logs:
Counting objects: 18, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 8.80 KiB | 0 bytes/s, done.
Total 18 (delta 13), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NPM_CONFIG_PRODUCTION=true
remote: NODE_VERBOSE=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): 6.11.1
remote: engines.npm (package.json): 5.3.0
remote:
remote: Downloading and installing node 6.11.1...
remote: Downloading and installing npm 5.3.0 (replacing version 3.10.10)...
remote:
remote: -----> Restoring cache
remote: Loading 2 from cacheDirectories (default):
remote: - node_modules
remote: - bower_components (not cached - skipping)
remote:
remote: -----> Building dependencies
remote: Installing node modules (package.json)
remote: npm ERR! Cannot read property '0' of undefined
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /app/.npm/_logs/2017-07-20T22_54_19_003Z-debug.log
remote:
remote: -----> Build failed
remote:
remote: We're sorry this build is failing! You can troubleshoot common issues here:
remote: https://devcenter.heroku.com/articles/troubleshooting-node-deploys
remote:
remote: If you're stuck, please submit a ticket so we can help:
remote: https://help.heroku.com/
remote:
remote: Love,
remote: Heroku
remote:
remote: ! Push rejected, failed to compile Node.js app.
remote:
remote: ! Push failed
remote: Verifying deploy...
remote:
remote: ! Push rejected to app-name.
remote:
To https://git.heroku.com/app-name.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/app-name.git'
Thanks.
I got the same error message after updating to node 6.11.1 and npm 5.3.0. heroku support suggested that I run these commands:
heroku plugins:install heroku-repo
heroku repo:purge_cache -a APPNAME
git commit --allow-empty -m "Purge cache"
git push heroku master
... where "APPNAME" was the heroku identifier for my app.
That resolved the error and I did not have to revert to an earlier version of npm. I've asked for an explanation but they haven't responded yet (just asked a few minutes ago).
UPDATE 2017-07-27: heroku support responded and said, "We're still working on the root cause for this but it seems to be fairly intermittent." Now that I have installed the plugin (see above), I am running these steps to update my app:
heroku repo:purge_cache -a APPNAME
git push heroku master
That's been working reliably for me.

failure to deploy heroku app

Here is the error log:
remote: Compressing source files... done.
remote: Building source:remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NPM_CONFIG_PRODUCTION=true
remote: NODE_VERBOSE=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): ~6.11.0
remote: engines.npm (package.json): ~5.0.3
remote:
remote: Resolving node version ~6.11.0 via semver.io...
remote: Downloading and installing node 6.11.1...
remote: Resolving npm version ~5.0.3 via semver.io...
remote: Downloading and installing npm 5.0.4 (replacing version
3.10.10)...
remote:
remote: -----> Restoring cache
remote: Skipping cache restore (new-signature)
remote:
remote: -----> Building dependencies
remote: Prebuild detected (node_modules already exists)
remote: Rebuilding any native modules
remote: npm ERR! path
/tmp/build_9c402097d0e7567eccf5e236dc7f0959/node_modules/jade/bin/jade.js
remote: npm ERR! code ENOENT
remote: npm ERR! errno -2
remote: npm ERR! syscall chmod
remote: npm ERR! enoent ENOENT: no such file or directory, chmod
'/tmp/build_9c402097d0e7567eccf5e236dc7f0959/node_modules/jade/bin/jade.js'
remote: npm ERR! enoent This is related to npm not being able to
find a file.
remote: npm ERR! enoent
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /app/.npm/_logs/2017-07-16T02_45_14_132Z-
debug.log
remote:
remote: -----> Build failed
remote:
remote: We're sorry this build is failing! You can troubleshoot
common issues here:
remote:
remote: Some possible problems:
remote:
remote: - node_modules checked into source control
remote:
remote: Love,
remote: Heroku
remote:
remote: ! Push rejected, failed to compile Node.js app.
remote:
remote: ! Push failed
remote: Verifying deploy...
remote:
remote: ! Push rejected to pacific-harbor-25184.
remote:
To https://git.heroku.com/pacific-harbor-25184.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/pacific-harbor-25184.git'
How can I get past this build error when deploying to heroku? (even though it works locally)
This is similar to a recent issue in cloudfoundry (but the same idea applies to heroku as well)
Male sure, in your local project, to ignore node_modules first:
echo 'node_modules' >> .gitignore
git rm -r --cached node_modules
git commit -m "ignore node_modules"
Then try your heroku deploy again.

Resources