What is node-gyp and why does it use my system files to build Node.js packages?
If I build a Node project that uses node-gyp internally then I just tar that project and move it to a different system, untar it and try to use it, will that work?
node-gyp is a tool which compiles Node.js Addons. Node.js Addons are native Node.js Modules, written in C or C++, which therefore need to be compiled on your machine. After they are compiled with tools like node-gyp, their functionality can be accessed via require(), just as any other Node.js Module.
If you do what you suggested the module won't work, you will need to compile it/build it with node-gyp on the system you moved the program to.
node-gyp: https://github.com/nodejs/node-gyp
Node.js Addons: https://nodejs.org/api/addons.html
Related
I'm using Rosetta 2 with Homebrew and have sqlite3 installed.
I added these to my ~/.zshrc so that the node compiler can find the brew installs:
export PATH="/usr/local/opt/sqlite/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"
I'm using installing using npm install sqlite3, building from source with --build-from-source flag, I'm also specifying the homebrew version of sqlite with --sqlite=/usr/local/opt/sqlite/
node-gyp goes to its fallback build node-pre-gyp install --fallback-to-build
After installation, I'm rebuilding it's native dependencies with electron-builder and install-app-deps
It even rebuilds to the platform and arch I'm looking for, darwin and x64. Which is promising.
• electron-builder version=23.0.3
• loaded parent configuration preset=react-cra
• rebuilding native dependencies dependencies=sqlite3#5.0.8 platform=darwin arch=x64
However, when run the app with npm run dev which runs concurrently " cross-env BROWSER=none npm start" "wait-on http://localhost:3000 && electron ."
It still is trying to find the arm64 version of it:
Error: Cannot find module '[..]/node_modules/sqlite3/lib/binding/napi-v6-darwin-unknown-arm64/node_sqlite3.node'
Even though I can confirm, it did build the x64 version because I have one located at:
/napi-v6-darwin-unknown-x64/node_sqlite3.node
Any help to get this working would be greatly appreciated, thanks!
After 4 days of digging, I have finally gotten it to work!!!
For anyone that might stumble on this:
The reason why sqlite3 won't behave is threefold:
When node sqlite3 is installed using npm install sqlite3, it fetches all dependencies and installs it. It then fetches the precompiled binary binding file for the target arch and platform. In my case we would want napi-v6-darwin-unknown-arm64 for ARM64 and darwin for Apple M1. There is no precompiled binary available yet for this Apple ARM64 and even if there is, the next paragraph will detail why it still won't work.
The problem is that it determines the the system's platform and architecture using the binary compiling package node-pre-gyp and this very savior of a Github issue details how node-pre-gyp is not handling ARM architecture detection properly and basically mixing everything up. Because it's not detecting properly, even if we build our own binding with --build-from-source when installing, it still won't work because it is compiling the wrong binding file for the wrong architecture. To make matters worse, if we don't use --build-from-source, it just simply fetches the Intel precompiled binding file. napi-v6-darwin-unknown-x64
Now for some reason, during runtime, it now detects the architecture properly and tries to look for the ARM64 version of the binding file, it won't find it and will throw the feared error 'module not found' for napi-v6-darwin-unknown-arm64. It obviously won't find it and even if it does, it will throw wrong architecture error because we have the Intel version on board napi-v6-darwin-unknown-x64.
So finally after digging at this for 4 days, here's how I got it working:
Uninstall sqlite3: npm uninstall sqlite3
Install a fresh sqlite3, build it from source, specify the target arch and use fallback build with node-pre-gyp just to be safe: npm install sqlite3 --build-from-source --target_arch=arm64 --fallback-to-build
Now the correct binding file is compiled for the correct platform and architecture, on runtime it will find it and will run!
i have nodejs x86 on windows 10 x64 and i installed electron with npm using this npm i -g electron and have a .dll file for driving external device.
i am using electron to develop desktop application.
i searched for finding a way for calling dll functions from js and i found ffi package. in first place i installed node x64 but i faced this error
App threw an error during load
Error: %1 is not a valid Win32 application.
then i searched it and i found this issue on github.
i uninstall my nodejs and replaced it with x86 version and this error still remain on my project. it seems that when i install ffi with npm npm i ffi, npm downloads source files and compile it with host architecture(x64) then ffi compile it self with my visual studio 2015 on x64 mode.
i even try to install ffi with npm i ffi --arch=ia32 but it did not work.
i donot have dll source files so i cannot rebuild it for any specific architecture.
How can i use electron 32bit version with ffi 32bit?
Is there any way to download ffi prebuild version and attaching it to project?
I want to use any version of electron (x64 and x86) and using my same 32bit dll.
The error typically happens when trying to load a 64-bit DLL from a 32-bit application.
In most cases, electron-rebuild should solve this issue for you, by rebuilding modules for the correct environment.
Is there a way to do an npm install that downloads dependencies but then it does not try to compile native addons?
Long story: My company has a build server running on Windows (can't change that) and it's getting complicated to compile some native addons there. This application is eventually deployed to Linux, so an npm rebuild is necessary anyways. Also, I can't do the npm install on the destination servers because they don't have access to the registry.
thanks
You can create a script (for Windows a JS file, for every other platform a shell script) that does nothing and is used instead of the normal node-gyp.
After that use
npm config set node_gyp <script_with_full_path>
to force npm to use your script instead of the normal node-gyp and by that skip the compile step.
Please note that requiring the module that used gyp may fail because the built file is now missing.
Hope this helps.
Is it possible to copy a set of NPM installed files and associated files from a Mac computer to a Windows computer, and for all those files to work?
For example, transfering Node.js files with some other NPM files from Mac to Windows, then running node app.js in that directory (on the Windows Command Prompt).
Thanks! :)
The binary, npm, that you install is platform dependent, as is node.js. That's why there are different releases for each platform available on the download site.
For the most part, your project files are platform independent. For example, most of your JavaScript files will all be used by node.js and work just fine without having to worry about what platform you are on because the system details will be dealt with by node.js itself.
However, some modules are platform dependent. For example, anything that uses node-gyp will try to compile on your platform whenever the module is installed by npm. You do not have to worry about that though because it is handled by npm, that's why you're using a package manager.
Copying node_modules can be done; but it's more often than not better and easier to just run npm i on whatever machine is going to be running your application. You can avoid having to worry about version problems using something like npm shrinkwrap which will lock down the version of a package that your module depends on.
NPM packages that contain native addons/dependencies are tied to the OS + NodeJS version and have to be rebuilt specifically for the system you intend to use them on. This is why you're seeing the error mentioning bson.node, it is a native addon that has to be rebuilt, this can be done with the npm rebuild command.
I frequently hear from experienced node developers that checking in node_modules is good practice. But most developers write on Mac/Darwin x64, but deploy on Linux x64.
If a node module happens to be written in C, and I install it on OS X, wouldn't I have to rebuild it on Linux?
The answer is: it depends on the package
Most packages do require reinstall, as node gyp the compiler does not cross compile by default - thanks #tkone.
Some packages like node-sass dynamically download prebuilt binaries for the relevant platform (previously node-sass used to include binaries for all platforms, however this has changed recently).
The now current node-sass 3.4.2 doesn't include the binaries in the originally downloaded npm package (as found in the npm package cache under ~/.npm).
What it does is that its install.js script will download the platform specific binary and store it under vendor/{platform}-{arch}-{process.versions.module}.node. Once installed, the install.js script normally isn't invoked again by npm.
So when you check in node_modules, it will contain the binary only for your initial platform.
For the fun of it I moved away the downloaded binary, which should be the same as someone else checking out your node_modules on a different platform. When then run, node-sass is so smart to detect that the required binary doesn't exist. It exits gracefully, recommending to run npm rebuild node-sass (also hinting that this is usually necessary when you change the node version). This will download the binary for the current platform, and then all is fine.
However, that's totally specific to node-sass, other packages with binaries may behave entirely different.
(See Node.js release versions for an explanation of process.versions.modules aka. NODE_MODULES_VERSION that determines the last number in the binary download URL. The binary name is constructed in function getBinaryName() in ~/.npm/node-sass/{version}/package.tgz/lib/extension.js )