Loading Nodejs Module at runtime in electron app - node.js

Currently I am playing around with electron using vue-cli-plugin-electron-builder along side a simple vue project. This is the project https://github.com/nklayman/vue-cli-plugin-electron-builder .
vue create my-project
cd my-project
vue add electron-builder
npm run electron:serve
My goal is to add a simple plugin-like architecture. The app serves only base functionality but can be extended with "plugins". Those plugins therefore are not included in the built, but will be loaded at runtime by electron. I would prefere when those plugins just behave like node modules ( module.exports = ) with its own dependencies ( probably with a package.json file inside ). I would locate those plugins at app.getPath('userData') + '/Plugins.
I looked at a few approaches on how to tackle this problem :
1. Using Nodejs vm module
First, I tried using Nodejs vm module to read and execute a script from an external file, all at runtime. It works great so far, although I would not be able to use external dependencies inside those loaded scripts. If I want to use external dependencies inside the plugin scripts, those dependencies must have been included in the electron build beforehand. Somehow defeats the whole purpose of having plugins ... only vanilla js + nodejs base modules would be possible .
2. using global.require
I saw this solution in another SO answer.
Using node require with Electron and Webpack
Webpack/electron require dynamic module
They say to use global.require but it throws an error saying global.require is not a function. The solution looked promising first, but somehow I can't get it to work.
3. simply use require
Of course I had to try it. When I try to require an external module from a non-project location it won't find the module, even if the path is correct. Again, the path I am trying to locate the module should be at app.getPath("userData"), not in the projects root directory. When however, I locate the plugins inside the root directory of the project it gets included in the built. This again defeats the purpose of having plugins.
Goal
So far, I haven't found a viable solution to this. I simply want my electron app to be extendible with basic node modules at runtime ( following a pre-defined schema to simplify ) . Of course there is atom, made with electron, using their own apm manager to install and load plugins, but this seems way to overpowered. Its enough for me to only have plugin files located locally, to have a public "marketplace" is no goal. Also, it's ok if the app has to reload / restart to load plugins.
Any ideas ?

After more and more research I stumbled over 2 packages :
https://www.npmjs.com/package/live-plugin-manager
https://github.com/getstation/electron-package-manager
both integrating npm to programmatically handle package installation at runtime. I settled for live-plugin-manager for now since its better documented and even allow package installation from local file system.
Pro
I was able to integrate the system out-of-the-box into a vanilla electron app. Works like a charm.
Cons
.I was not able to use it inside a vue electron boilerplate (like the one I said I was using in OP), since webpack is interferring with the require environment. But there sure is a solution to this.
Update : I was able to get it to work eventually inside a webpack bundled electron vue boilerplate. I accidentally mixed import and require . The following code works for me using live-plugin-manager
// plugin-loader.js
const path = require('path');
const { PluginManager } = require('live-plugin-manager');
const pluginInstallFolder = path.resolve(app.getPath('userData'), '.plugins');
const pluginManager = new PluginManager();
module.exports = async (pkg) => {
// installs pkg from npm
await pluginManager.install(pkg);
const package = pluginManager.require(pkg);
return package
}
// main.js
const pluginLoader = require('./plugin-loader');
pluginLoader("moment").then((moment) => {
console.log(moment().format());
})
This will install "moment" package from npm during runtime into a local directory and load it into the app, without bundling it into the executable files.

Related

Dynamic require not supported in React Native

I have just tried to use wasm bindings in a react-native project. But the wasm bindings require access to node modules like fs so I used rn-nodeify as a work-around to get the React Native representation of fs and to be able to use require. That worked fine but I think that the wasm bindings are incompatible with react native, because I get the error Dynamic require defined at line 10; not supported by Metro. The error is talking about the line below:
//line 10
const { TextDecoder, TextEncoder, inspect } = require(String.raw`util`);
The bindings I'm using are from this package #iota/identity-wasm
Steps to reproduce
If you want to reproduce this issue I have created a sample RN Project that throws the error by startup.
git clone https://github.com/JonasHiltl/DigitalIdentityNodeified.git
cd ./DigitalIdentityNodeified
npm install
npx react-native start
npx react-native run-android
I'm interested to know what exactly a dynamic require is and if it's possible to replace the dynamic require with a normal one.
Sadly you cannot use require dynamicly in react-native :/
You will have to make a require for every single thing you will need.
Check out this for example:
Dynamic require not supported in React Native

If an npm package ships with multiple dist folders, how do I know which one is used when my App is built?

I am using an npm package (aurelia-google-maps) in my application. The package ships with AMD, System, CommonJS, Native Modules, and ES2015 dist folders like this:
/node_modules/
/aurelia-google-maps/
/dist/
/amd
/system
/native-modules
/es2015
/commonjs
In my typescript app I am simply importing all the classes and functions as:
import {Configure} from "aurelia-google-maps"
Is there a way that I can find out which distribution is used when I build my application?
I don't think you can determine it without the help of your build tool. It can be done in 2 steps:
using type of to check the availability of a variable in runtime code:
const distType = typeof DIST_TYPE !== 'undefined' ? DIST_TYPE : void 0;
configure the build tool to replace DIST_TYPE with the distribution target
And then you can use it in normal code, via distType variable.
For typescript, you just need an extra declaration
declare const DIST_TYPE: string | undefined;
I have figured it out. It seems like Aurelia automatically registers it's own plugin called DistPlugin that resolves dependencies using the native-modules folder. See this wiki for an explanation on how to change this behaviour.

Node package dependencies on IBM Cloud Foundry - require/module is not defined (Package not loading)

I am working on an application via the toolchain tool on IBM Cloud and editing the code via the Eclipse Orion IDE. As I am not accessing this through my local cli, my understanding is that in order to so call npm install {package}, I would just need to include the package in the package.json file under dependencies and require it in my app. However, when I load the application, I get the require is not defined indicating that the package has not been installed. Moreover, the require() is being used in the app.js file with the application being launched but not from files in my public directory.
After playing around further, it seems it might have to do with the way the directory tree is being traced as the error is only thrown in subdirectories. For example, require('express') works in app.js which is in the main directory ./ but fails when it is called in test.js in ./subdirectory/test.js. I feel like I'm missing something painfully simple like configuration of endpoint or something.
I've been searching around but I can't seem to find how to get the packages loaded, preferably without using the cli. Appreciate any pointers. Thanks!
Update: After playing around further, I am also getting module is not defined error when trying to require from another file in the same directory. For example module.exports = 'str' returns this error. While trying to require('./file') returns the require is not defined. It might have to do with how node is wrapping the functions?
Update 2: Tried "start": "npm install && node app.js" in package.json but no luck. Adding a build stage which calls npm install before deployment also does not work
Update 3: After adding npm install build stage, I am able to see that the dependencies have been successfully built via the logs. However, the require is not defined error still persists.
Update 4: Trying npm install from my CLI doesn't work as well even though all packages and dependencies are present
Update 5: Running cf restage or configuring cache via cacheDirectories does not work as well
Opened a related question regarding deployment here
Found out my confusion was caused due to me not realizing that require() cannot be used on the client side unless via tools such as Browserify.

Browserify - JsSip

I have a new project where I'm using browserify to convert node modules into an sdk that can run inside the browser.
I'm requiring a number of other npm packages like:
var log4js = require('log4js');
That run fine and give me no problems in the browser, however JsSip just will not cooperate. When I do
var JsSIP = require('jssip');
I get
plivowebsdk.js:2 Uncaught Error: Cannot find module '../../package.json'
Looking through the code, it's obvious when it makes this call
var pkg = require('../../package.json');
is where it bombs out. Clearly it cannot find the package.json file, which it uses to pull out version information. I know JsSip is actually built with browersify itself (or used to be) so that it can run in either node or a browser. Is this causing a conflict?
Still sort of new to browserify, is their a configuration option or transformation I can perform to get around this?
Turned out the be browserify errors, re did the build process using the gulp recipes for browersify and works as expected.

node_modules packaging fail (Webpack or Browserify)

I'm trying to bundle my node.js application with webpack or browserify, but I need some backend modules such as knex, bookshelf and others.
But webpack and browserify fail to package these modules since they do some strange stuff with require()...
I got these kind of errors :Cannot find module 'sqlite3' or Error: Cannot find module './dialects/postgres/index.js'
I can't ignore my node_modules directory since I need the modules in the package because I can't access the environment where my package will be used. (AWS Lambda)
I don't need webpack or browserify to optimize my node_modules but I need them, is there a way to tell webpack or browserify to just bundle the node_module folder and trust me if a require is missing ?
EDIT: I'm using serverless to test and deploy my lambdas and the only plugins allowing me to use ES2015 with babel force me to use webpack / browserify
Thank you :)
You can incorporate node_modules in the your Lambda package (the zip file containing your code that you upload in Amazon Lambda) and don't need to package it (I mean create a file with Webpack or Browserify).
Some node modules are not meant to be used in the browser and do not support packaging because they use dynamic requires. They can have good reasons to do that, specially ORMs like Bookshelf or Sequelize.
Webpack can manage simple dynamic requires, but it works only for rules easy to parse. If you really want to use webpack for whatever reason, you could have a look to the ContextReplacementPlugin but I don't think it worth the effort.
I had this exact problem and finally got it to work with webpack (don't know about browserify) by adding this to my webpack config:
{
plugins: [
new webpack.NormalModuleReplacementPlugin(/\.\.\/migrate/, '../util/noop.js'),
new webpack.NormalModuleReplacementPlugin(/\.\.\/seed/, '../util/noop.js'),
new webpack.IgnorePlugin(/mariasql/, /\/knex\//),
new webpack.IgnorePlugin(/mssql/, /\/knex\//),
new webpack.IgnorePlugin(/mysql/, /\/knex\//),
new webpack.IgnorePlugin(/mysql2/, /\/knex\//),
new webpack.IgnorePlugin(/oracle/, /\/knex\//),
new webpack.IgnorePlugin(/oracledb/, /\/knex\//),
new webpack.IgnorePlugin(/pg-query-stream/, /\/knex\//),
new webpack.IgnorePlugin(/sqlite3/, /\/knex\//),
new webpack.IgnorePlugin(/strong-oracle/, /\/knex\//),
new webpack.IgnorePlugin(/pg-native/, /\/pg\//)
]
}
If you're using serverless-webpack like me, you'll have to explicitly npm install the normal webpack module in your project and require it in your webpack config file.
This config is specifically for my setup where I use postgres without pg-native. Just ignore the modules you're not using.
The two top plugins aren't needed for webpack to build, but they get rid of a ton of annoying warnings. They're probably a little dangerous since they could match requires in other modules than knex. I couldn't find a better way without having to write my own plugin.
I was struggling to get webpack to bundle knex properly and want to share my configuration.
We could instead use ContextReplacementPlugin to avoid listing all drivers not needed for the project as when using webpack.IgnorePlugin. A side benefit is that it would avoid ignoring these packages in other modules (as mention by #Erik Frisk).
For example, I use a mysql database with mysql2 driver and my webpack config looks like this:
plugins: [
new webpack.ContextReplacementPlugin(/knex\/lib\/dialects/, /mysql2\/index.js/),
]
This will only bundle node_modules/knex/lig/dialects/mysql2/index.js excluding other dialects thus effectively ignoring the dependency in other packages like oracledb, mssql, etc.
To find more information about ContextReplacementPlugin have a look at Webpack’s ContextReplacementPlugin examples:

Resources