Sharp package Heroku deployment issue - node.js

I'm deploying my NodeJs app to Heroku. No issues running the app with nodemon locally.
However when I'm trying to push to Heroku Master, seems like "sharp" package is causing an issue:
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote:
remote: > sharp#0.22.0 install
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy &&
prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: info sharp Downloading
https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
remote:
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/install/libvips.js:86
remote: throw new Error(Status
${response.statusCode});
remote: ^
remote:
remote: Error: Status 403
remote: at
/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/install/libvips.js:86:17
remote: at f
(/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/once/once.js:25:25)
remote: at ClientRequest.protocol.request.res
(/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/simple-get/index.js:63:5)
remote: at Object.onceWrapper (events.js:277:13)
remote: at ClientRequest.emit (events.js:189:13)
remote: at HTTPParser.parserOnIncomingClient [as
onIncoming] (_http_client.js:556:21)
remote: at HTTPParser.parserOnHeadersComplete
(_http_common.js:109:17)
remote: at TLSSocket.socketOnData (_http_client.js:442:20)
remote: at TLSSocket.emit (events.js:189:13)
remote: at addChunk (_stream_readable.js:284:12)
remote: make: Entering directory
'/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/build'
remote: TOUCH Release/obj.target/libvips-cpp.stamp
remote: CXX(target) Release/obj.target/sharp/src/common.o
remote: ../src/common.cc:25:10: fatal error: vips/vips8: No
such file or directory
remote: #include
remote: ^~~~~~~~~~~~
remote: compilation terminated.
remote: sharp.target.mk:128: recipe for target
'Release/obj.target/sharp/src/common.o' failed
remote: make: *** [Release/obj.target/sharp/src/common.o] Error
1
remote: make: Leaving directory
'/tmp/build_4cfeda2379ea362c72ac083b57638cce/node_modules/sharp/build'
Update:
Try removing sharp from package.json and add it back again using npm install sharp works this time... I still don't know why.
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote:
remote: > sharp#0.22.0 install
/tmp/build_5c8d3b01594e0b86f9d3e0d372534414/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy &&
prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: info sharp Downloading
https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
remote: added 57 packages from 92 contributors and audited
20393 packages in 10.683s
remote: found 62 low severity vulnerabilities
remote: run npm audit fix to fix them, or npm audit for
details
remote:
remote: -----> Build
remote:
remote: -----> Caching build
remote: - node_modules
remote:
remote: -----> Pruning devDependencies
remote: removed 606 packages and audited 497 packages in 8.181s
remote: found 0 vulnerabilities
remote:
remote:
remote: -----> Build succeeded!

There are many people with that problem, it's related to libvips as dependency not sharp by itself.
On github in this issue-comment the problem is explained:
The pre-built binaries of sharp are only guaranteed to work with the pre-built binaries of libvips.
If the download of a pre-built binary of libvips fails then sharp has to fall back to attempting to build itself from source.
http://sharp.pixelplumbing.com/en/stable/install/#pre-compiled-libvips-binaries provides information about serving the libvips binaries from a local URL.
The reason why the download is failing might be the cache, or a corrupted download like reported in the above linked issue on github:
I had to manually remove the ~/.npm/_libvips/libvips-8.7.0-linux-x64.tar.gz and try again. Seems to be downloaded a corrupted file.
Here is an interesting discussion concerning the same problem related to gatsby, a solution seems not yet existing even the issue was closed (before any comment was made): https://github.com/gatsbyjs/gatsby/issues/1754
However if the problem is only related to the download for some reason it's possible to deploy libvips locally like explained in the manual:
Pre-compiled libvips binaries
This module will attempt to download a pre-compiled bundle of libvips
and its dependencies on Linux and Windows machines under either of
these conditions:
If a global installation of libvips that meets the minimum version
requirement cannot be found;
If SHARP_IGNORE_GLOBAL_LIBVIPS environment variable is set.
SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install sharp
Should you need to manually download and inspect these files,
you can do so via https://github.com/lovell/sharp-libvips/releases
Should you wish to install these from your own location, set the
sharp_dist_base_url npm config option, e.g.
npm config set sharp_dist_base_url "https://hostname/path/"
npm install sharp
or set the SHARP_DIST_BASE_URL environment variable, e.g.
SHARP_DIST_BASE_URL="https://hostname/path/" npm install sharp
to use https://hostname/path/libvips-x.y.z-platform.tar.gz.
So this is a bit background, I'm aware that this can not really be seen as an answer with solution :/

If someone reaches here and didn't found an answer - my issue was that my project was using an old node version (8.x)
The solution was to install an older version of sharp - The latest one that support 8.x - "sharp": "0.23.4"

Related

The `npm ci` command can only install with an existing package-lock.json or remote npm-shrinkwrap.json with lockfileVersion >= 1

I made one tiny text change to an old(er) Node application which works as intended in my local development system, including the npm ci command, but it fails to deploy to Heroku with this error:
npm ERR! The `npm ci` command can only install with an existing package-lock.json or
remote npm-shrinkwrap.json with lockfileVersion >= 1
but I have a package-lock.json with:
"lockfileVersion": 2,
This application used to deploy just fine but it's been a year of two since the last deploy so I'm guessing this is some underlying issue with the Heroku deployment, perhaps with the updated build-packs. The application is currently using heroku-18 but it's trying to upgrade to heroku-22 in the same deploy.
I've tried different versions of Node and NPM though I can't go above Node 12 without making significant updates to this legacy application...which I'd r e a a a l l y like to avoid.
Any ideas on how to tickle this away?
Full Error
remote: Building source:
remote:
remote: -----> Building on the Heroku-22 stack
remote: -----> Using buildpack: heroku/nodejs
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_VERBOSE=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): 15.3.0
remote: engines.npm (package.json): 7.0.14
remote:
remote: Resolving node version 15.3.0...
remote: Downloading and installing node 15.3.0...
remote: npm 7.0.14 already installed with node
remote:
remote: -----> Restoring cache
remote: Cached directories were not restored due to a change in version of node, npm, yarn or stack
remote: Module installation may take longer for this build
remote:
remote: -----> Prebuild
remote: Running heroku-prebuild
remote:
remote: > snazzywiz#2.1.1 heroku-prebuild
remote: > npm --global install pushstate-server#3.0.1; npm --global install #angular/cli
remote:
remote:
remote: added 44 packages, and audited 44 packages in 3s
remote:
remote: 10 vulnerabilities (5 low, 3 moderate, 2 high)
remote:
remote: To address issues that do not require attention, run:
remote: npm audit fix
remote:
remote: To address all issues, run:
remote: npm audit fix --force
remote:
remote: Run `npm audit` for details.
remote:
remote: added 209 packages, and audited 209 packages in 8s
remote:
remote: 25 packages are looking for funding
remote: run `npm fund` for details
remote:
remote: found 0 vulnerabilities
remote: npm notice
remote: npm notice New major version of npm available! 7.0.14 -> 8.19.2
remote: npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.19.2>
remote: npm notice Run `npm install -g npm#8.19.2` to update!
remote: npm notice
remote:
remote: -----> Installing dependencies
remote: Installing node modules
remote: npm ERR! The `npm ci` command can only install with an existing package-lock.json or
remote: npm ERR! npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm#5 or
remote: npm ERR! later to generate a package-lock.json file, then try again.
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /tmp/npmcache.ULXke/_logs/2022-09-17T19_15_54_600Z-debug.log
remote:
remote: -----> Build failed
Note: There are several SO questions with the same "The npm ci command can only..." error message but triggered from different scenarios but none from just a deploy to Heroku.

Vendor File not found during node-gyp build

I am willing to install a custom build of libvips with sharp on heroku. I use buildpacks that run without errors, but when it comes to node-gyp compilation, it fails here saying it cannot access /app/vendor/vips/lib/libvips-cpp.so. When I run the build without installing sharp and then connect with heroku run bash, I can see that the file is at this location.
Why node-gyp cannot find it then ?
My buildpacks
=== my-site Buildpack URLs
1. heroku-community/apt
2. https://github.com/lechinoix/heroku-buildpack-vips
3. heroku/nodejs
The error
remote: -----> vips (heroku-20 stack) app detected
remote: -----> Vendoring binaries
remote: Fetching /tmp/codon/tmp/buildpacks/2ef659928923e1b5ae4bf85b886e53d084c6a3d4/build/heroku-20.tar.gz
remote: Changing permissions on libvips folder
remote: -----> Configuring build environment
remote: -----> Building runtime environment
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_VERBOSE=false
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): unspecified
remote: engines.npm (package.json): unspecified (use default)
remote:
remote: Resolving node version 14.x...
remote: Downloading and installing node 14.18.1...
remote: Using default npm version: 6.14.15
remote:
remote: -----> Restoring cache
remote: Caching has been disabled because NODE_MODULES_CACHE=false
remote:
remote: -----> Installing dependencies
remote: Installing node modules (package.json)
remote:
remote: > sharp#0.28.1 install /tmp/build_1f4c5c03/node_modules/sharp
remote: > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
remote:
remote: sharp: Detected globally-installed libvips v8.10.6
remote: sharp: Building from source via node-gyp
remote: readelf: Error: '/app/vendor/vips/lib/libvips-cpp.so': No such file
remote: gyp: Call to 'if readelf -Ws "$(PKG_CONFIG_PATH="/tmp/build_1f4c5c03/.apt/usr/lib/x86_64-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/i386-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/pkgconfig::/tmp/build_1f4c5c03/vendor/vips/lib/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/x86_64-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/i386-linux-gnu/pkgconfig:/tmp/build_1f4c5c03/.apt/usr/lib/pkgconfig::/usr/local/lib/pkgconfig:/usr/lib/pkgconfig" pkg-config --variable libdir vips-cpp)/libvips-cpp.so" | c++filt | grep -qF __cxx11;then echo "1";else echo "0";fi' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
Thank you so much for your help !
Finally found that heroku is creating the whole /app folder in a directory /tmp/build_1f4c5c03 that also contains the freshly installed vendors. In my .pc files, the package is located in /app so it cannot be found. I used a heroku-prebuild npm script to sed inplace the paths :
TMP_DIR=$(pwd)
for file in $(grep -rl /app/vendor ${TMP_DIR}/vendor/vips/lib/pkgconfig)
do
sed -i "s+/app/vendor+${TMP_DIR}/vendor+g" $file
done
Not perfectly sure it is the best way to do...

Heroku deployment issue at node installArchSpecificPackage

(Story)
So my MERN app works fine on localhost and all paths and json files have been taken care of... proxy, scrips, NODE_ENV, etc... then this new dumb error showed up. I have been trying to get this to work for a month. My file structure wasn't right before and I still got it to deploy by changing the deployment paths but the server was giving a 404 for the axios calls to MongoDB Atlas using Express.
So I put the whole thing together in the right file structure, set the environment variable for ATLAS_URI and after resolving a ton of errors I am stuck on this last one, hopefully. Please help me get rid of it.
(Problem)
Not able to deploy on heroku. It's working fine locally.
Error Log after doing git push heroku master
Total 4619 (delta 1796), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote: NODE_VERBOSE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): ^8.15.0
remote: engines.npm (package.json): ^6.4.1
remote:
remote: Resolving node version ^8.15.0...
remote: Downloading and installing node 8.17.0
remote: Bootstrapping npm ^6.4.1 (replacing 6.13.4)...
remote: npm ^6.4.1 installed
remote:
remote: -----> Restoring cache
remote: Cached directories were not restored due to a change in version of node, npm, yarn or stack
remote: Module installation may take longer for this build
remote:
remote: -----> Installing dependencies
remote: Prebuild detected (node_modules already exists)
remote: Rebuilding any native modules
remote:
remote: > node#8.15.0 preinstall /tmp/build_a457d1d9/node_modules/node
remote: > node installArchSpecificPackage
remote:
remote: /tmp/build_a457d1d9/node_modules/node/node_modules/.bin/node: 1: /tmp/build_a457d1d9/node_modules/node/node_modules/.bin/node: �����H: not found
remote: /tmp/build_a457d1d9/node_modules/node/node_modules/.bin/node: 2: /tmp/build_a457d1d9/node_modules/node/node_modules/.bin/node: Syntax error: "(" unexpected
remote: module.js:550
remote: throw err;
remote: ^
remote:
remote: Error: Cannot find module 'node-linux-x64/package.json'
remote: at Function.Module._resolveFilename (module.js:548:15)
remote: at Function.resolve (internal/module.js:18:19)
remote: at ChildProcess.<anonymous> (/tmp/build_a457d1d9/node_modules/node-bin-setup/index.js:18:27)
remote: at emitTwo (events.js:126:13)
remote: at ChildProcess.emit (events.js:214:7)
remote: at maybeClose (internal/child_process.js:915:16)
remote: at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
remote: npm ERR! code ELIFECYCLE
remote: npm ERR! errno 1
remote: npm ERR! node#8.15.0 preinstall: node installArchSpecificPackage
remote: npm ERR! Exit status 1
remote: npm ERR!
remote: npm ERR! Failed at the node#8.15.0 preinstall script.
remote: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
I'm new to this. Please tell me exactly what to do without much jargon. Any help is much appreciated. Thanks!
Okay I found out how to do this. Just delete the node_modules folder in the source (server.. not client) directory and try again. It should work like a charm...
Found it here Error while pushing to Heroku: Cannot find module 'node-linux-x64/package.json'
Thanks guys

Failing to install Node.js dependencies on Heroku

I'm getting a memory error when trying to deploy a Node.js app to heroku. It doesn't seem to get past the installation section:
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote: NODE_VERBOSE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): unspecified
remote: engines.npm (package.json): unspecified (use default)
remote:
remote: Resolving node version 12.x...
remote: Downloading and installing node 12.13.1...
remote: Using default npm version: 6.12.1
remote:
remote: -----> Restoring cache
remote: Cached directories were not restored due to a change in version of node, npm, yarn or stack
remote: Module installation may take longer for this build
remote:
remote: -----> Installing dependencies
remote: Installing node modules (package.json + package-lock)
remote: npm ERR! Maximum call stack size exceeded
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /tmp/npmcache.TGwfg/_logs/2019-11-23T11_15_42_537Z-debug.log
I've tried turning the module cache off, which did nothing. I've also tried turning more verbose npm logging, but all that tells me is that the installation hasn't finished.
The application isn't particularly big, just an express application. There must be some dependency that has a large stack of pre/post install commands... I guess the only way to find out which one it is by uninstalling one at a time and redeploying :(
But before I go down that route, I thought I'd ask around to see if anyone else has experienced this before. I'm only using a free dyno at the moment to test if Heroku would be suitable.
I had this issue and #Rashomon's solution worked, but I had to type it thusly "engines": {"node": "14.16.1", "npm": "6.14.12"}, note the double quotes on everything.
To check your versions, use npm version and node -v in command line
Turns out that using yarn will help me avoid this problem.
rm package-lock.json
yarn
git add -A
git commit -m "chore: use yarn"
git push heroku master
If I decide to upgrade the service then I ask Heroku, directly, why I experienced this. Until then, this is an OK solution,

Could not find a required file index.html - Heroku push error

I am pushing the react-node app to Heroku using Heroku CLI.
build is failing and the error is Could not find a required file index.html
I have checked all the files and everything in place
don't know what am I missing?
I am stuck here for one week.
Please help me
I am pushing the react-node app to Heroku using Heroku CLI.
build is failing and the error is Could not find a required file index.html
I have checked all the files and everything in place
don't know what am I missing?
I am stuck here for one week.
Please help me
Package.json
"main": "server.js",
"scripts": {
"start": "node server.js",
"server": "nodemon server.js",
"client": "npm start --prefix client",
"dev": "concurrently \"npm run server\" \"npm run client\"",
"heroku-postbuild": "NPM_CONFIG_PRODUCTION=false npm install --prefix client && npm run build --prefix client"
},
Error :
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 398 bytes | 398.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote: NPM_CONFIG_LOGLEVEL=error
remote: NODE_ENV=production
remote: NODE_MODULES_CACHE=true
remote: NODE_VERBOSE=false
remote:
remote: -----> Installing binaries
remote: engines.node (package.json): 8.9.4
remote: engines.npm (package.json): 6.2.0
remote:
remote: Resolving node version 8.9.4...
remote: Downloading and installing node 8.9.4...
remote: Bootstrapping npm 6.2.0 (replacing 5.6.0)...
remote: npm 6.2.0 installed
remote:
remote: -----> Restoring cache
remote: - node_modules
remote:
remote: -----> Building dependencies
remote: Installing node modules (package.json)
remote: audited 2601 packages in 4.802s
remote: found 0 vulnerabilities
remote:
remote: Running heroku-postbuild
remote:
remote: > devconnector#1.0.0 heroku-postbuild /tmp/build_cb13714e30f186bd579e6f8f21bf9383
remote: > NPM_CONFIG_PRODUCTION=false npm install --prefix client && npm run build --prefix client
remote:
remote:
remote: > jss#9.8.7 postinstall /tmp/build_cb13714e30f186bd579e6f8f21bf9383/client/node_modules/jss
remote: > node -e "console.log('\u001b[35m\u001b[1mLove JSS? You can now support us on open collective:\u001b[22m\u001b[39m\n > \u001b[34mhttps://opencollective.com/jss/donate\u001b[0m')"
remote:
remote: Love JSS? You can now support us on open collective:
remote: > https://opencollective.com/jss/donate
remote:
remote: > uglifyjs-webpack-plugin#0.4.6 postinstall /tmp/build_cb13714e30f186bd579e6f8f21bf9383/client/node_modules/uglifyjs-webpack-plugin
remote: > node lib/post_install.js
remote:
remote: added 1375 packages from 852 contributors and audited 13755 packages in 46.845s
remote: found 0 vulnerabilities
remote:
remote:
remote: > client#0.1.0 build /tmp/build_cb13714e30f186bd579e6f8f21bf9383/client
remote: > react-scripts build
remote:
**remote: Could not find a required file.
remote: Name: index.html**
remote: Searched in: /tmp/build_cb13714e30f186bd579e6f8f21bf9383/client/public
remote: npm ERR! code ELIFECYCLE
remote: npm ERR! errno 1
remote: npm ERR! client#0.1.0 build: `react-scripts build`
remote: npm ERR! Exit status 1
remote: npm ERR!
remote: npm ERR! Failed at the client#0.1.0 build scr`enter code here`ipt.
remote: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /tmp/npmcache.bspBQ/_logs/2018-10-14T10_57_59_194Z-debug.log
remote: npm ERR! code ELIFECYCLE
remote: npm ERR! errno 1
remote: npm ERR! devconnector#1.0.0 heroku-postbuild: `NPM_CONFIG_PRODUCTION=false npm install --prefix client && npm run build --prefix client`
remote: npm ERR! Exit status 1
remote: npm ERR!
remote: npm ERR! Failed at the devconnector#1.0.0 heroku-postbuild script.
remote: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
remote:
remote: npm ERR! A complete log of this run can be found in:
remote: npm ERR! /tmp/npmcache.bspBQ/_logs/2018-10-14T10_57_59_208Z-debug.log
remote:
remote: -----> Build failed
remote:
remote: We're sorry this build is failing! You can troubleshoot common issues here:
remote: https://devcenter.heroku.com/articles/troubleshooting-node-deploys
remote:
remote: If you're stuck, please submit a ticket so we can help:
remote: https://help.heroku.com/
remote:
remote: Love,
remote: Heroku
remote:
remote: ! **Push rejected, failed to compile Node.js app.**
remote:
remote: ! Push failed
remote: Verifying deploy...
remote:
remote: ! Push rejected to murmuring-depths-54253.
I also encountered this issue today and managed to fix it by removing /public folder from .gitignore.
Two things could be cause of this:
Probably you have /public directory specified inside your .gitignore
Case sensitive paths
more info at: https://create-react-app.dev/docs/deployment/#heroku
Possible Solution: Your file structure might be the issue
By chance do you have an 'api' folder next to your 'client' folder?
Heroku doesn't seem to play well with certain file structures (not sure why, I opened a ticket with them to inquire)
I was receiving almost the exact same error because I had most of my 'server' files within an api folder at the root.
After moving those files outside the api folder, removing my lock files, and reinstalling node modules, deployment went off without a hitch.
So, try the following:
Make sure that your file structure isn't conflicting with standard deployments (see screenshot below)
Remove your lock files and node_modules
Reinstall node_modules at root and in client
commit and and attempt another deploy -- let us know if that succeeds
As you can see, the only difference is the location of those folders within the api folder (obviously, import/require pathnames needed to be updated as well)
Failing Folder Structure
Successful Folder Structure

Resources