How do you automatically download external (C++) libraries when using native Node addons? - node.js

Question

I'd like to include libpng in my native Node addon. How can I include it, so that when my library is installed, it will automatically download a specified version of libpng? Is it possible to use npm's package.json for this? If this is not possible, what is the accepted way of including an external library's source code in your repository?

Answer 1

I recommend that you create a gyp file to build the dependency library and add a script to your package.json to download it for you.
I often use my own native addon modules to demonstrate the answers I give to these questions. My own native addon module, node-dvbtee, demonstrates this.
You will notice the following inside package.json:
"scripts": {
"preinstall": "npm install mkdirp && scripts/prepare-build.sh && node scripts/configure-build.js",
"install": "node-gyp rebuild -j 8",
"test": "mocha"
},
What matters here is the preinstall section of the scripts section. It calls scripts/prepare-build.sh, which contains the following:
#!/bin/sh
cd "$(dirname "$0")"/..
if [ -e libdvbtee ]; then
echo libdvbtee sources present
else
git clone git://github.com/mkrufky/libdvbtee.git
fi
cd libdvbtee
if [ -e libdvbpsi/bootstrap ]; then
echo libdvbpsi sources present
else
rm -rf libdvbpsi
git clone git://github.com/mkrufky/libdvbpsi.git
cd libdvbpsi
touch .dont_del
cd ..
fi
As you can see, the above script checks to see if the libdvbtee directory is present. If not, it will clone it from github. After that, it checks to see if the full libdvbpsi sources are present. If not, it will clone them from github.
Now, for the gyp files:
My project has the gyp files stored in the deps directory.
libdvbpsi.gyp looks like this:
{
'target_defaults': {
'default_configuration': 'Debug',
'configurations': {
'Debug': {
'defines': [ 'DEBUG', '_DEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 1, # static debug
},
},
},
'Release': {
'defines': [ 'NDEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 0, # static release
},
},
}
},
'msvs_settings': {
'VCLinkerTool': {
'GenerateDebugInformation': 'true',
},
},
'include_dirs': [
'../libdvbtee/libdvbpsi/src',
'../libdvbtee/libdvbpsi/src/tables',
'../libdvbtee/libdvbpsi/src/descriptors',
'../libdvbtee/libdvbpsi'
],
'defines': [
'PIC',
'HAVE_CONFIG_H'
],
},
'targets': [
# libdvbpsi
{
'target_name': 'dvbpsi',
'product_prefix': 'lib',
'type': 'static_library',
'sources': [
'../libdvbtee/libdvbpsi/src/dvbpsi.c',
'../libdvbtee/libdvbpsi/src/psi.c',
'../libdvbtee/libdvbpsi/src/demux.c',
'../libdvbtee/libdvbpsi/src/descriptor.c',
'../libdvbtee/libdvbpsi/src/tables/pat.c',
'../libdvbtee/libdvbpsi/src/tables/pmt.c',
'../libdvbtee/libdvbpsi/src/tables/sdt.c',
'../libdvbtee/libdvbpsi/src/tables/eit.c',
# '../libdvbtee/libdvbpsi/src/tables/cat.c',
'../libdvbtee/libdvbpsi/src/tables/nit.c',
'../libdvbtee/libdvbpsi/src/tables/tot.c',
# '../libdvbtee/libdvbpsi/src/tables/sis.c',
# '../libdvbtee/libdvbpsi/src/tables/bat.c',
# '../libdvbtee/libdvbpsi/src/tables/rst.c',
'../libdvbtee/libdvbpsi/src/tables/atsc_vct.c',
'../libdvbtee/libdvbpsi/src/tables/atsc_stt.c',
'../libdvbtee/libdvbpsi/src/tables/atsc_eit.c',
'../libdvbtee/libdvbpsi/src/tables/atsc_ett.c',
'../libdvbtee/libdvbpsi/src/tables/atsc_mgt.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_02.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_03.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_04.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_05.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_06.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_07.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_08.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_09.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_0a.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_0b.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_0c.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_0d.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_0e.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_0f.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_10.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_11.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_12.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_13.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_14.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_1b.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_1c.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_24.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_40.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_41.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_42.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_43.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_44.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_45.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_47.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_48.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_49.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_4a.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_4b.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_4c.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_4d.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_4e.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_4f.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_50.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_52.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_53.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_54.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_55.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_56.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_58.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_59.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_5a.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_62.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_66.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_69.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_73.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_76.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_7c.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_81.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_83.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_86.c',
# '../libdvbtee/libdvbpsi/src/descriptors/dr_8a.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_a0.c',
'../libdvbtee/libdvbpsi/src/descriptors/dr_a1.c',
],
'conditions': [
['OS=="mac"',
{
'xcode_settings': {
'WARNING_CFLAGS': [
'-Wno-deprecated-declarations'
]
}
}
]
],
'cflags!': ['-Wdeprecated-declarations','-Wimplicit-function-declaration'],
'cflags+': ['-Wno-deprecated-declarations','-Wno-implicit-function-declaration','-std=c99'],
},
]
}
Of course, there are a lot of specifics in this gyp file that are specific to libdvbpsi and my use case. As such, you will notice that quite a few of the source files in the library are not actually needed for the version of it that we're going to build for my node.js addon module. The source files that we are not going to build are commented out by preceding that line with a hash # character.
We link this library to the node module we're currently building by including it in the dependency section of the node.js addon modules bindings.gyp. Here is the one used in my addon module:
{
"targets": [
{
"target_name": "dvbtee",
"sources": [
"src/node-dvbtee.cc",
"src/dvbtee-parser.cc"
],
"dependencies": [
'deps/libdvbtee.gyp:dvbtee_parser'
],
"include_dirs": [
"libdvbtee/usr/include",
"libdvbtee/libdvbtee",
"libdvbtee/libdvbtee/decode",
"libdvbtee/libdvbtee/decode/table",
"libdvbtee/libdvbtee/decode/descriptor",
"<!(node -e \"require('nan')\")"
],
'cflags': [ '-DDEBUG_CONSOLE=1' ],
'cflags_cc': [ '-DDEBUG_CONSOLE=1', '-Wno-deprecated-declarations' ],
'cflags!': [ '-fno-exceptions' ],
'cflags_cc!': [ '-fno-exceptions', '-Wdeprecated-declarations' ],
'conditions': [
['OS=="mac"', {
'xcode_settings': {
'WARNING_CFLAGS': [
'-Wno-deprecated-declarations'
],
'GCC_ENABLE_CPP_EXCEPTIONS': 'YES'
}
}]
]
}
]
}
As you can see, deps/libdvbtee.gyp:dvbtee_parser is listed in the dependencies section, above. deps/libdvbtee.gyp:dvbtee_parser itself contains its own dependencies section:
"dependencies": [
'libdvbpsi.gyp:dvbpsi'
],
So, when npm install is executed, npm will run the preinstall script to fetch the sources, then it will build the custom libdvbpsi library based on libdvbpsi.gyp, then build the custom libdvbtee based on libdvbtee.gyp which depends on that custom libdvbpsi library, and finally it will build and link the node.js addon module that depends on the libdvbtee library build.
In my specific case, the libraries need to be configured before we attempt to build them. This step is required to write the config.h header file that these libraries depend on. I handle that step within the scripts/configure-build.js script which is run after downloading the sources. In most cases, you will want to simply run ./configure for each library, but that depends on the libraries that you're including.
This is a cross-platform solution, provided that the libraries you're building are themselves cross-platform.

Answer 2

You can add it in scrips section in package.json. But you have to be careful about which all devices your application will be executed. Such as ARM, Intel 32 bit or Intel 64 bit, or so. You have options, I am just adding some hints here and you can put your code accordingly. Here script will get executed during npm install command.
1. In script, you have to check the machine type and do download library accordingly.
//package.json
{
"scripts": {
"preinstall": ""
,"install": ""
,"test" : ""
}
}
In script, you have to check the machine type and do download library accordingly, something like using wget abc.so in install section of script. You have to do some scripting to take right lib for machine and put in right place.
In other way if you want, you can add build script, which will do download the source code and build in the system on the fly.
git clone git://xyz/abc.git
cd abc
./configure
make
make install.
You can also look for babel cli for source compilation. https://babeljs.io/docs/usage/cli/
And all these within scripts section, within preinstall or install or test.
In your case, you would prefer to go for 1st way.