Question
Is there a way to install node-sqlite3 for multiple platforms I am targeting in my app without running standalone build for just every target platform combination?
Context
In my Node.js app I have a npm dependency node-sqlite3 (GitHub, npm), which contains different binaries (bindings) for different platforms.
My app is targeting different platforms, including Windows, Linux and macOS (both ia32 and x64) and modern Node versions: v6, v7 and v8. The app doesn't have any platform-specific behavior.
If I install the project's dependencies using npm install, node-sqlite3 downloads binaries just for the current platform (let's say win32, x64, Node v7.10).
I also have a Travis CI build configuration, which I use for Continuous Deployment as well as Continuous Integration. I chose Ubuntu Trusty as a host for executing builds.
As a part of a build process the app's dependencies are being installed by npm install. Within deployment process, the built app with it's dependencies is being packaged (archived) and uploaded to a file hosting for further distribution.
Issue
node-sqlite3 is not installed for all target platforms I need, but just for a platform currently being used (for development or executing a build).
Possible solution
I could execute builds and deploy:
with Travis - for Linux and macOS
with AppVeyor - for Windows
But that's looks like a big overhead. As I've already said, the app doesn't have any platform-specific behavior. And I trust node-sqlite3's vendor tested it at all major platforms I am targeting.
Yes, in case with node-sqlite3 you do have such a capability.
It is possible because it's owner mapbox uses node-pre-gyp (GitHub, npm) for distribution of node-sqlite3.
After installing your app's dependencies with npm install execute the following command at the root of your Node project for every target platform combination:
./node_modules/.bin/node-pre-gyp install \
--directory=./node_modules/sqlite3 \
--target_platform={OS} \
--target_arch={OS architecture} \
--target={Node version}
As a result, you will have required bindings in the ./node_modules/sqlite3/lib/binding/ directory.
Options
Here's the options' descriptions from the node-pre-gyp docs.
--directory: run the command in this directory
--target_platform=win32: Pass the target platform and override the host platform. Valid values are linux, darwin, win32, sunos, freebsd, openbsd, and aix.
--target_arch=ia32: Pass the target arch and override the host arch. Valid values are 'ia32','x64', or arm.
--target=0.10.25: Pass the target node or node-webkit version to compile against
If they exist, prebuilt binaries for chosen platform will be downloaded from a file storage (Amazon S3). Otherwise you have to build binaries by yourself.
A list of available binaries of node-sqlite3 is here.
Examples
A couple of examples for certain target platforms:
• Windows x86 and Node 6.10.0:
./node_modules/.bin/node-pre-gyp install --directory=./node_modules/sqlite3 --target_platform=win32 --target_arch=ia32 --target=6.10.0
• macOS x64 and Node 7.10.0:
./node_modules/.bin/node-pre-gyp install --directory=./node_modules/sqlite3 --target_platform=darwin --target_arch=x64 --target=7.10.0
• Linux x64 and Node 8.0.0:
./node_modules/.bin/node-pre-gyp install --directory=./node_modules/sqlite3 --target_platform=linux --target_arch=x64 --target=8.0.0
Related
I'm using Rosetta 2 with Homebrew and have sqlite3 installed.
I added these to my ~/.zshrc so that the node compiler can find the brew installs:
export PATH="/usr/local/opt/sqlite/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"
I'm using installing using npm install sqlite3, building from source with --build-from-source flag, I'm also specifying the homebrew version of sqlite with --sqlite=/usr/local/opt/sqlite/
node-gyp goes to its fallback build node-pre-gyp install --fallback-to-build
After installation, I'm rebuilding it's native dependencies with electron-builder and install-app-deps
It even rebuilds to the platform and arch I'm looking for, darwin and x64. Which is promising.
• electron-builder version=23.0.3
• loaded parent configuration preset=react-cra
• rebuilding native dependencies dependencies=sqlite3#5.0.8 platform=darwin arch=x64
However, when run the app with npm run dev which runs concurrently " cross-env BROWSER=none npm start" "wait-on http://localhost:3000 && electron ."
It still is trying to find the arm64 version of it:
Error: Cannot find module '[..]/node_modules/sqlite3/lib/binding/napi-v6-darwin-unknown-arm64/node_sqlite3.node'
Even though I can confirm, it did build the x64 version because I have one located at:
/napi-v6-darwin-unknown-x64/node_sqlite3.node
Any help to get this working would be greatly appreciated, thanks!
After 4 days of digging, I have finally gotten it to work!!!
For anyone that might stumble on this:
The reason why sqlite3 won't behave is threefold:
When node sqlite3 is installed using npm install sqlite3, it fetches all dependencies and installs it. It then fetches the precompiled binary binding file for the target arch and platform. In my case we would want napi-v6-darwin-unknown-arm64 for ARM64 and darwin for Apple M1. There is no precompiled binary available yet for this Apple ARM64 and even if there is, the next paragraph will detail why it still won't work.
The problem is that it determines the the system's platform and architecture using the binary compiling package node-pre-gyp and this very savior of a Github issue details how node-pre-gyp is not handling ARM architecture detection properly and basically mixing everything up. Because it's not detecting properly, even if we build our own binding with --build-from-source when installing, it still won't work because it is compiling the wrong binding file for the wrong architecture. To make matters worse, if we don't use --build-from-source, it just simply fetches the Intel precompiled binding file. napi-v6-darwin-unknown-x64
Now for some reason, during runtime, it now detects the architecture properly and tries to look for the ARM64 version of the binding file, it won't find it and will throw the feared error 'module not found' for napi-v6-darwin-unknown-arm64. It obviously won't find it and even if it does, it will throw wrong architecture error because we have the Intel version on board napi-v6-darwin-unknown-x64.
So finally after digging at this for 4 days, here's how I got it working:
Uninstall sqlite3: npm uninstall sqlite3
Install a fresh sqlite3, build it from source, specify the target arch and use fallback build with node-pre-gyp just to be safe: npm install sqlite3 --build-from-source --target_arch=arm64 --fallback-to-build
Now the correct binding file is compiled for the correct platform and architecture, on runtime it will find it and will run!
I am trying to use Firebase admin SDK in my server. When I deploy I get the error I am missing file node-v59-linux-x64/grpc_node.node in firebase-admin node_module map. I added "grpc": "1.7.1" in my package, but I still do not get that file after NPM update. I get an older version, node-v57. I also checked this path https://registry.npmjs.org/grpc/-/grpc-1.7.1.tgz, but I could not locate the file. I deleted my node_modules map and ran npm install again, still no node-v59.
How/where can I download that file? Is there any one who can put the file here so I can manually add it?
Error: Cannot find module
'/data/app/node_modules/grpc/src/node/extension_binary/node-v59-linux-x64/grpc_node.node'
This kind of problem is usually caused by installing the library on one system, then deploying and running it on a different system that requires a different binary file.
The simplest solution to this problem is to run npm rebuild after deployment on the system you deployed on.
Alternatively, if npm rebuild is not an option, you can pre-install the binary for the system you are deploying on by running npm install with some extra options. The --target argument allows you to install for a different version of Node. An argument of --target=9.0.0 installs the binary for Node 9 (only the major version has to match). The --target_platform argument allows you to install for a specific operating system: windows, linux, or darwin (Mac). The --target_arch argument lets you install for a different processor architecture: ia32, x64, or arm. And finally, the --target_libc argument lets you select binaries built for a different libc: glibc or musl (for Alpine Linux).
So, in your case, you should be able to get that binary by running
npm install --target=9.0.0 --target_platform=linux --target_arch=x64
I had the same problem. You can download the file here: https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.7.1/node-v59-linux-x64.tar.gz
This helped in my case, based on #murgatroid99’s answer:
npm rebuild --target=8.1.0 --target_platform=linux --target_arch=x64 --target_libc=glibc --update-binary
It downloads the required binary to your node_modules/grpc directory.
I run macOS X on my dev machine and I’m deploying to AWS Lambda; this keeps both runtime versions installed, which means I can develop and test locally and then deploy to Lambda.
I have a node webkit application that uses the mdns module for publishing a Bonjour service from a Mac (with Mavericks). When I run the server code with node server.js everything works OK, but when running the node webkit application that uses the same server code I get this error:
"Uncaught Error: dlopen(/Users/me/myfolder/node_modules/mdns/build/Release/dns_sd_bindings.node, 1): no suitable image found. Did find:
/Users/me/myfolder/node_modules/mdns/build/Release/dns_sd_bindings.node: mach-o, but wrong architecture", source: /Users/me/myfolder/node_modules/mdns/lib/dns_sd.js (35)
Apparently when you install the mdns module with npm it's built for an x86 architecture and I need it for i386, because node-webkit is built for i386 (I found out about this by reading this thread: http://forums.macrumors.com/showthread.php?t=879780). You can verify it by running this in a terminal:
$ lipo -info /Applications/node-webkit.app/Contents/MacOS/node-webkit
Non-fat file: /Applications/node-webkit.app/Contents/MacOS/node-webkit is architecture: i386
I found this link suggesting a solution: https://github.com/rogerwang/node-webkit/issues/296 for another module (node proxy). The suggested instructions are:
I managed to build a 32-bit version of node-proxy as follows:
I installed nw-gyp
I ran nw-gyp configure --target=0.3.6
I edited the generated file nodeproxy.target.mk in the build directory by replacing -arch x86_64by -arch i386
I ran nw-gyp build
But as I'm not used to building node modules manually, while following the instructions it wasn't clear to me in which folder I should run those steps (I assumed it's in the module folder inside node_modules:
a) when I install nw-gyp I don't get the nw-gyp command to use globally (I guess the -g option is missing in the instructions)
b) using instead gyp configure --target=0.3.6 gives me an error saying that there is no option target
c) I tried skipping the configure step (just to try) and the build command breaks with:
Could not automatically locate src directory. This isa temporary
Chromium feature that will be removed. Use--depth as a workaround.
But when trying to use --depth (of course) it requires an argument, I couldn't find out what to put there.
So... how should I build the mdns module for using it with node webkit? (either the 0.8.6 version or the 0.10.0, I can adapt).
I managed to make it work.
As I had already installed the mdns module, I already had the source code of the module in the folder node_modules/mdns inside my project folder.
So these are the steps I followed to build the mdns module for the i386 architecture:
1) Install nw-gyp by running: npm install -g nw-gyp
2) Enter in the node_modules/mdns folder of your node-webkit project
3) Run nw-gyp configure --target=0.8.6 (this target is the version of the node-webkit you have installed)
4) And finally run nw-gyp build
I got a lot of warnings of deprecated functions, but it was built OK and now my node-webkit application can publish the Bonjour service successfully.
Unfortunately this is not the best solution, though, because the next person that installs the project will have to do the same after the regular npm install... But at least it's something to get it working.
I frequently hear from experienced node developers that checking in node_modules is good practice. But most developers write on Mac/Darwin x64, but deploy on Linux x64.
If a node module happens to be written in C, and I install it on OS X, wouldn't I have to rebuild it on Linux?
The answer is: it depends on the package
Most packages do require reinstall, as node gyp the compiler does not cross compile by default - thanks #tkone.
Some packages like node-sass dynamically download prebuilt binaries for the relevant platform (previously node-sass used to include binaries for all platforms, however this has changed recently).
The now current node-sass 3.4.2 doesn't include the binaries in the originally downloaded npm package (as found in the npm package cache under ~/.npm).
What it does is that its install.js script will download the platform specific binary and store it under vendor/{platform}-{arch}-{process.versions.module}.node. Once installed, the install.js script normally isn't invoked again by npm.
So when you check in node_modules, it will contain the binary only for your initial platform.
For the fun of it I moved away the downloaded binary, which should be the same as someone else checking out your node_modules on a different platform. When then run, node-sass is so smart to detect that the required binary doesn't exist. It exits gracefully, recommending to run npm rebuild node-sass (also hinting that this is usually necessary when you change the node version). This will download the binary for the current platform, and then all is fine.
However, that's totally specific to node-sass, other packages with binaries may behave entirely different.
(See Node.js release versions for an explanation of process.versions.modules aka. NODE_MODULES_VERSION that determines the last number in the binary download URL. The binary name is constructed in function getBinaryName() in ~/.npm/node-sass/{version}/package.tgz/lib/extension.js )
I am trying to install socket.io on windows with npm for use on a nodeJS server.
First, when I typed "npm install socket.IO" i had an error in the log saying something about python and node-gyp. I installed python 2.7.3 and set the environment variables.
Now I got a new error, which has something to do with visual studio (what the hell does VS have to do with npm ? Is it about the compiler? ).
The error is the same as here npm install for some packages (sqlite3, socket.io) fail with error MSB8020 on Windows 7
But when I use the option in the answer instead of the error it tells me something about a possible data loss (c4267) but doesn't log any error.
Then when I start my app, it tells me cannot find module socket.io still
What could this come from ?
Oh and also when i do npm config get root it tells me "undefined" could it have anything to do with it ?
Should I install the modules globally or locally ?
At least one of the packages in Socket.IO's dependency tree is a C/C++ addons which needs to be compiled on your system as it's installed. And, since it's a dependency, if it doesn't succeed in installing, neither will Socket.IO.
To enable cross-system compilation, Node.js uses node-gyp as its build system. You'll need to have it installed as a global package:
npm install -g node-gyp
As well as have its dependencies installed. Abridged version:
Python 2
C/C++ Compiler / Build Tools
For Windows, Microsoft Visual Studio 2013 (C++ or Windows Desktop) (Express edition)
For 64-bit, may need Windows 7 64-bit SDK
Then, you should be able to install Socket.IO as a local package so you can require it:
npm install socket.io
I had a similar problem on Mac.
What resolved my problem is installing a slightly older version of Socket.io.
I did:
npm install socket.io#"~0.8.1"
which would install the latest version between 0.8.0 to 0.8.9, but not 0.9.0 or above.
Socket.io then installed perfectly.
Make sure you have all the required software to run node-gyp:
https://github.com/TooTallNate/node-gyp
You can configure version of Visual Studio used by gyp via an environment variable so you can avoid having to set the --msvs_version=2012 property.
Examples:
set GYP_MSVS_VERSION=2012 for Visual Studio 2012
set GYP_MSVS_VERSION=2013e (the 'e' stands for 'express edition')
For the full list see
- https://github.com/joyent/node/blob/v0.10.29/tools/gyp/pylib/gyp/MSVSVersion.py#L209-294
This is still painful for Windows users of NodeJS as it assumes you have a copy of Visual Studio installed and many end users will never have this. So I'm lobbying Joyent to the encourage them to include web sockets as part of CORE node and also to possible ship a GNU gcc compiler as part of NodeJS install so we can permanently fix this problem.
Feel free to add your vote at:
https://github.com/joyent/node/issues/8005#issuecomment-50545326
The problem causing the compile failure is that the ws module installed by the engine.io module required by socket.io pulls in a backlevel version of nan. See https://github.com/BrowserSync/grunt-browser-sync/issues/95 for details. To work around the problem after the build failure:
cd to node_modules/socket.io/node_modules/engine.io/node_modules/ws
edit package.json to change the release of nan from 1.4.x to 1.6.0
issue command node-gyp rebuild
You should now be able to use socket.io
Another approach is to use Docker for Windows and spin up a NodeJS environment. While developing you can mount your Node code as a Docker volume and so continue to update your code from Windows but execute it and install it's dependencies inside a Linux VM. When you deploy you might prefer to use a Dockerfile that COPY's your Node code into your Docker image and so bakes it into the release image you deploy.
This approach might be required if you don't want to risk changing the socket.io version of your code or its dependencies.
It also may be a valuable solution if you planned to deploy to a corporate Intranet or public/private Cloud.
Docker can also be very handy for testing deployment under different versions of Node without disturbing the development environment of your Windows computer (e.g. for testing a NodeJS lib).
Official NodeJS Docker images
An explanation of how to use these images
this problem makes me very troubled..
I tried many solutions.
I installed .NET Framework 2.0 SDK.
I installed Python 2.7.x
I installed VS 2012 Express
I set some paths
I executed npm install xxx with the argument --msvs_version=2010(or 2012/2013..)...
But all failed.
finally, I uninstalled Python & .NET Framework 2.0 SDK & VS 2012, clear those paths,enable Windows Update, install all essential updates, restart my computer
then execute commands below:
npm install node-gyp -g
npm install socket.io -g
npm install browser-sync -g
there is no errors in installation logs.
Note : this solution may not work for you, but for me