When I package the electron app on macOS I can never obfuscate the file with Nightmare because of its limitations. Would I need to re-write the whole library, or is there a way I can get around this?
There is a solution. Absolutely unstable and untested, try at your own risk. And ofc, if you can find and fix the bugs that comes as side effect, feel free to share them :)
We are going to use a module called pkg to bundle up the script with node. Also for simplicity, we will use npx.
There were some side-effects but that was first time something worked with electron and nightmare.
Consider the following script,
const Nightmare = require("nightmare");
const nightmare = Nightmare({
show: true
});
nightmare
.goto("https://example.com")
.title()
.end()
.then(console.log)
.catch(error => {
console.error(error);
});
This is a simple script that says, go to example.com and give me the title.
Cool! Let's try using it thru npx and pkg. The code for that is,
npx pkg app.js --target 'host'
However, we got some nasty errors,
> Warning Cannot include file %1 into executable.
The file must be distributed with executable as %2.
node_modules/nightmare/lib/runner.js
path-to-executable/nightmare/runner.js
> Warning Cannot include file %1 into executable.
The file must be distributed with executable as %2.
node_modules/nightmare/lib/frame-manager.js
...
etc etc. and the file wouldn't run.
Error: spawn /home/someone/Desktop/a/electron/dist/electron ENOENT
It cannot find the required files as they were not bundled. We will use process.cwd() to get those data which resides in relevant folder.
const nodeDir = process.cwd() + "/node_modules/"; // <- Get node modules folder
const nightmareDir = nodeDir + "nightmare"; // <-- Get nightmarejs path
const electronDir = nodeDir + "electron"; // <-- Get electron path
const Nightmare = require(nightmareDir);
const electronPath = require(electronDir);
const nightmare = Nightmare({
show: true,
electronPath // <-- use the specific electron path
});
nightmare
.goto("https://example.com")
.title()
.end()
.then(console.log)
.catch(error => {
console.error(error);
});
When I ran it, it showed me some more warning, but that's because I did not optimize the process.cwd() yet. Then I ran it and voila!
➜ a npx pkg app.js --target 'host'
> pkg#4.3.1
> Warning Cannot resolve 'nightmareDir'
/home/someone/Desktop/a/app.js
Dynamic require may fail at run time, because the requested file
is unknown at compilation time and not included into executable.
Use a string literal as an argument for 'require', or leave it
as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve 'electronDir'
/home/someone/Desktop/a/app.js
Dynamic require may fail at run time, because the requested file
is unknown at compilation time and not included into executable.
Use a string literal as an argument for 'require', or leave it
as is and specify the resolved file name in 'scripts' option.
➜ a ./app
Example Domain // <-- Our sweet result :D
➜ a
This can be improved and tweaked, but I'll leave that to you.
Related
I'm trying to setup SSR for react app with #loadable/components. I setup all based on docs with babel and webpack plugins. When I try to run node server.js it runs ok but when I open a browser and throws the following error (into node console):
TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received an instance of Object
at validateString (internal/validators.js:118:11)
at Module.require (internal/modules/cjs/loader.js:1033:3)
at require (internal/modules/cjs/helpers.js:72:18)
at smartRequire (/Users/max/Documents/repos/app/node_modules/#loadable/server/lib/util.js:44:25)
at new ChunkExtractor (/Users/max/Documents/repos/app/node_modules/#loadable/server/lib/ChunkExtractor.js:181:50)
at renderer (webpack://app/./node_modules/#MYSCOPE/elm/dist/elm.esm.js?:3619:19)
at eval (webpack://app/./src/server.tsx?:64:90)
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
code: 'ERR_INVALID_ARG_TYPE'
}
As you can see there is #MYSCOPE in the traceback which holds some of my internal packages (if it matters).
#loadable/server/lib/util.js is the following function:
And when I try to console.log(modulePath) on line 42 I see a whole stats JSON output which seems wrong and I should get a single module path (as I understand).
Any help?
I can share some specific parts of my configuration files if needed. Because I see my own package in console output seems like something is wrong with it's build (it works perfectly on the client-side with cjs build), but having full stats object as module path is very confusing.
UPD: Demo https://www.dropbox.com/s/9r947cgg4qvqbu4/loadable-test.zip?dl=0
Run
yarn
yarn dev:server
# go to localhost:3000 and see the error in console
to rebuild:
yarn
yarn dev:build-client
yarn dev:build-server
yarn dev:server # go to localhost:3000
The statsFile option passed to ChunkExtractor expects a path to the loadable-stats.json file, not the actual JSON content of it. By doing require('../loadable-stats.json'), webpack actually resolve the JSON during build time and assign it to the loadableJson variable.
You can change your loadableJson as follow:
import path from 'path';
const loadableJson = path.resolve(__dirname, '../bundle_client/loadable-stats.json');
This will solve the problem you had on your question. But, if you only do this, you will notice that you have another problem. Loadable by default assumes that your entry chunk name is main. This is not the case in your demo, as you have set the entry chunk name to be app instead.
entry: {
app: ['#babel/polyfill', './src/index.tsx']
},
To solve this, simply tell loadable about your entrypoints names by passing an array to the ChunkExtractor contructor as such:
const extractor = new ChunkExtractor({
statsFile: loadableJson,
entrypoints: ["app"], // name of your entry chunk
});
That's it, everything should now be working properly!
If it helps, I set up the demo on GitHub so you can easily see the changes I made here.
I try to import a node module inside an Angular 8 web worker, but get an compile error 'Cannot find module'. Anyone know how to solve this?
I created a new worker inside my electron project with ng generate web-worker app, like described in the above mentioned ng documentation.
All works fine until i add some import like path or fs-extra e.g.:
/// <reference lib="webworker" />
import * as path from 'path';
addEventListener('message', ({ data }) => {
console.log(path.resolve('/'))
const response = `worker response to ${data}`;
postMessage(response);
});
This import works fine in any other ts component but inside the web worker i get a compile error with this message e.g.
Error: app/app.worker.ts:3:23 - error TS2307: Cannot find module 'path'.
How can i fix this? Maybe i need some additional parameter in the generated tsconfig.worker.json?
To reproduce the error, run:
$ git clone https://github.com/hoefling/stackoverflow-57774039
$ cd stackoverflow-57774039
$ yarn build
Or check out the project's build log on Travis.
Note:
1) I only found this as a similar problem, but the answer handles only custom modules.
2) I tested the same import with a minimal electron seed which uses web workers and it worked, but this example uses plain java script without angular.
1. TypeScript error
As you've noticed the first error is a TypeScript error. Looking at the tsconfig.worker.json I've found that it sets types to an empty array:
{
"compilerOptions": {
"types": [],
// ...
}
// ...
}
Specifying types turns off the automatic inclusion of #types packages. Which is a problem in this case because path has its type definitions in #types/node.
So let's fix that by explicitly adding node to the types array:
{
"compilerOptions": {
"types": [
"node"
],
// ...
}
// ...
}
This fixes the TypeScript error, however trying to build again we're greeted with a very similar error. This time from Webpack directly.
2. Webpack error
ERROR in ./src/app/app.worker.ts (./node_modules/worker-plugin/dist/loader.js!./src/app/app.worker.ts)
Module build failed (from ./node_modules/worker-plugin/dist/loader.js):
ModuleNotFoundError: Module not found: Error: Can't resolve 'path' in './src/app'
To figure this one out we need to dig quite a lot deeper...
Why it works everywhere else
First it's important to understand why importing path works in all the other modules. Webpack has the concept of targets (web, node, etc). Webpack uses this target to decide which default options and plugins to use.
Ordinarily the target of a Angular application using #angular-devkit/build-angular:browser would be web. However in your case, the postinstall:electron script actually patches node_modules to change that:
postinstall.js (parts omitted for brevity)
const f_angular = 'node_modules/#angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/browser.js';
fs.readFile(f_angular, 'utf8', function (err, data) {
var result = data.replace(/target: "electron-renderer",/g, '');
var result = result.replace(/target: "web",/g, '');
var result = result.replace(/return \{/g, 'return {target: "electron-renderer",');
fs.writeFile(f_angular, result, 'utf8');
});
The target electron-renderer is treated by Webpack similarily to node. Especially interesting for us: It adds the NodeTargetPlugin by default.
What does that plugin do, you wonder? It adds all known built in Node.js modules as externals. When building the application, Webpack will not attempt to bundle externals. Instead they are resolved using require at runtime. This is what makes importing path work, even though it's not installed as a module known to Webpack.
Why it doesn't work for the worker
The worker is compiled separately using the WorkerPlugin. In their documentation they state:
By default, WorkerPlugin doesn't run any of your configured Webpack plugins when bundling worker code - this avoids running things like html-webpack-plugin twice. For cases where it's necessary to apply a plugin to Worker code, use the plugins option.
Looking at the usage of WorkerPlugin deep within #angular-devkit we see the following:
#angular-devkit/src/angular-cli-files/models/webpack-configs/worker.js (simplified)
new WorkerPlugin({
globalObject: false,
plugins: [
getTypescriptWorkerPlugin(wco, workerTsConfigPath)
],
})
As we can see it uses the plugins option, but only for a single plugin which is responsible for the TypeScript compilation. This way the default plugins, configured by Webpack, including NodeTargetPlugin get lost and are not used for the worker.
Solution
To fix this we have to modify the Webpack config. And to do that we'll use #angular-builders/custom-webpack. Go ahead and install that package.
Next, open angular.json and update projects > angular-electron > architect > build:
"build": {
"builder": "#angular-builders/custom-webpack:browser",
"options": {
"customWebpackConfig": {
"path": "./extra-webpack.config.js"
}
// existing options
}
}
Repeat the same for serve.
Now, create extra-webpack.config.js in the same directory as angular.json:
const WorkerPlugin = require('worker-plugin');
const NodeTargetPlugin = require('webpack/lib/node/NodeTargetPlugin');
module.exports = (config, options) => {
let workerPlugin = config.plugins.find(p => p instanceof WorkerPlugin);
if (workerPlugin) {
workerPlugin.options.plugins.push(new NodeTargetPlugin());
}
return config;
};
The file exports a function which will be called by #angular-builders/custom-webpack with the existing Webpack config object. We can then search all plugins for an instance of the WorkerPlugin and patch its options adding the NodeTargetPlugin.
I use node pkg to create a .exe of my nodejs service: https://www.npmjs.com/package/pkg
My question is: how do I make the .exe use a config.js for some setup values? Basic stuff like ip, port, database name etc. Because I have 3 environments, and I would like to use the same exe for all, but different config.js files for each.
So far, if I do pkg app.js then it creates an .exe that doesn't look at any other files. Totally stand alone. How do I make it look at config.js when it is started up?
On the website they do have a section on config https://github.com/zeit/pkg#config but I do not understand how to make use of it. At the moment I have my app.js, and I have secrets.js which holds the config information.
I am not sure this is right way, but I hope this can be helpful to somebody.
Refer to pkg document, on the run time, __dirname becomes "/snapshot/project".
So, by checking __dirname, you can identify in which environment you are.
(node app.js or app.exe).
Then we can separate require sentence like below.
const PKG_TOP_DIR = 'snapshot';
const runInPKG = (function(){
const pathParsed = path.parse(__dirname);
const root = pathParsed.root;
const dir = pathParsed.dir;
const firstDepth = path.relative(root, dir).split(path.sep)[0];
return (firstDepth === PKG_TOP_DIR)
})();
let config = require('./appconfig.json');
if(runInPKG) {
const deployPath = path.dirname(process.execPath);
config = require(path.join(deployPath, 'appconfig.json'));
}
Adding above code to your app.js makes some warning when pkg build.
pkg . --targets node8-win-x64 --out-path ./dist
pkg#4.4.0
Warning Cannot resolve 'path.join(deployPath, 'appconfig.json')'
app.js
Dynamic require may fail at run time, because the requested file
is unknown at compilation time and not included into executable.
Use a string literal as an argument for 'require', or leave it
as is and specify the resolved file name in 'scripts' option.
https://github.com/vercel/pkg/issues/195
use fs to read config file insead of require or import
eg:
const configPath = path.join(process.cwd(), './config/config.json');
lset data = fs.readFileSync(configPath);
same question link:excluding config file while converting node js files to exe using pkg
I am trying to include a precompiled binary with an electron app. I began with electron quick start app and modified my renderer.js file to include this code that is triggered when a file is dropped on the body:
spawn = require('child_process').spawn,
ffmpeg = spawn('node_modules/.bin/ffmpeg', ['-i', clips[0], '-an', '-q:v', '1', '-vcodec', 'libx264', '-y', '-pix_fmt', 'yuv420p', '-vf', 'setsar=1,scale=trunc(iw/2)*2:trunc(ih/2)*2,crop=in_w:in_h-50:0:50', '/tmp/out21321.mp4']);
ffmpeg.stdout.on('data', data => {
console.log(`stdout: ${data}`);
});
ffmpeg.stderr.on('data', data => {
console.log(`stderr: ${data}`);
});
I have placed my precompiled ffmpeg binary in node_modules/.bin/. Everything works great in the dev panel, but when I use electron-packager to set up the app, it throws a spawn error ENOENT to the console when triggered. I did find a very similar question on SO, but the question doesn't seem to be definitively answered. The npm page on electron-packager does show that they can be bundled, but I cannot find any documentation on how to do so.
The problem is that electron-builder or electron-packager will bundle your dependency into the asar file. It seems that if the dependency has a binary into node_modules/.bin it is smart enough to not package it.
This is the documentation for asar packaging for electron-builder on that topic. It says
Node modules, that must be unpacked, will be detected automatically
I understand that it is related to existing binaries in node_modules/.bin.
If the module you are using is not automatically unpacked you can disable asar archiving completely or explicitly tell electron-builder to not pack certain files. You do so in your package.json file like this:
"build": {
"asarUnpack": [
"**/app/node_modules/some-module/*"
],
For your particular case
I ran into the same issue with ffmpeg and this is what I've done:
Use ffmpeg-static. This package bundles statically compiled ffmpeg binaries for Windows, Mac and Linux. It also provides a way to get the full path of the binary for the OS you are running: require('ffmpeg-static').path
This will work fine in development, but we still need to troubleshoot the distribution problem.
Tell electron-builder to not pack the ffmpeg-static module:
"build": {
"asarUnpack": [
"**/app/node_modules/ffmpeg-static/*"
],
Now we need to slightly change the code to get the right path to ffmpeg with this code: require('ffmpeg-static').path.replace('app.asar', 'app.asar.unpacked') (if we are in development the replace() won't replace anything which is fine).
If you are using webpack (or other javascript bundler)
I ran into the issue that require('ffmpeg-static').path was returning a relative path in the renderer process. But the issue seemed to be that webpack changes the way the module is required and that prevents ffmpeg-static to provide a full path. In the Dev Tools the require('ffmpeg-static').path was working fine when run manually, but when doing the same in the bundled code I was always getting a relative path. So this is what I did.
In the main process add this before opening the BrowserWindow: global.ffmpegpath = require('ffmpeg-static').path.replace('app.asar', 'app.asar.unpacked'). The code that runs in the main process is not bundled by webpack so I always get a full path with this code.
In the renderer process pick the value this way: require('electron').remote.getGlobal('ffmpegpath')
I know I'm a bit late but just wanted to mention ffbinaries npm package I created a while ago exactly for this purpose.
It'll allow you to download ffmpeg/ffplay/ffserver/ffprobe binaries to specified location either during application boot (so you don't need to bundle it with your application) or in a CI setup. It can autodetect platform, you can also specify it manually.
If anyone happens to need an answer to this question: I do have a solution to this, but I have no idea if this is considered best practice. I couldn't find any good documentation for including 3rd party precompiled binaries, so I just fiddled with it until it finally worked. Here's what I did (starting with the electron quick start, node.js v6):
From the app directory I ran the following commands to include the ffmpeg binary as a module:
mkdir node_modules/ffmpeg
cp /usr/local/bin/ffmpeg node_modules/ffmpeg/
ln -s ../ffmpeg/ffmpeg node_modules/.bin/ffmpeg
(replace /usr/local/bin/ffmpeg with your current binary path, download it from here) Placing the link allowed electron-packager to include the binary I saved to node_modules/ffmpeg/.
Then to get the bundled app path I installed the npm package app-root-dir by running the following command:
npm i -S app-root-dir
Since I could then get the app path, I just appended the subfolder for my binary and spawned from there. This is the code that I placed in renderer.js:.
var appRootDir = require('app-root-dir').get();
var ffmpegpath=appRootDir+'/node_modules/ffmpeg/ffmpeg';
console.log(ffmpegpath);
const
spawn = require( 'child_process' ).spawn,
ffmpeg = spawn( ffmpegpath, ['-i',clips_input[0]]); //add whatever switches you need here
ffmpeg.stdout.on( 'data', data => {
console.log( `stdout: ${data}` );
});
ffmpeg.stderr.on( 'data', data => {
console.log( `stderr: ${data}` );
});
This is how I would do it:
Taking cues from tsuriga's answer, here is my code:
Note: replace or add OS path accordingly.
Create a directory ./resources/mac/bin
Place you binaries inside this folder
Create file ./app/binaries.js and paste the following code:
'use strict';
import path from 'path';
import { remote } from 'electron';
import getPlatform from './get-platform';
const IS_PROD = process.env.NODE_ENV === 'production';
const root = process.cwd();
const { isPackaged, getAppPath } = remote.app;
const binariesPath =
IS_PROD && isPackaged
? path.join(path.dirname(getAppPath()), '..', './Resources', './bin')
: path.join(root, './resources', getPlatform(), './bin');
export const execPath = path.resolve(path.join(binariesPath, './exec-file-name'));
Create file ./app/get-platform.js and paste the following code:
'use strict';
import { platform } from 'os';
export default () => {
switch (platform()) {
case 'aix':
case 'freebsd':
case 'linux':
case 'openbsd':
case 'android':
return 'linux';
case 'darwin':
case 'sunos':
return 'mac';
case 'win32':
return 'win';
}
};
Add the following code inside the ./package.json file:
"build": {
....
"extraFiles": [
{
"from": "resources/mac/bin",
"to": "Resources/bin",
"filter": [
"**/*"
]
}
],
....
},
import binary file path as:
import { execPath } from './binaries';
#your program code:
var command = spawn(execPath, arg, {});
Why this is better?
Most of the answers require an additional package called app-root-dir
The original answer doesn't handle the (env=production) build or the pre-packed versions properly. He/she has only taken care of development and post-packaged versions.
I feel like this must be so obvious but it's escaping me.
I'd like to run requirejs's r.js compilation from a node module instead of from the command line, and every bit of documentation I've seen just shows the command line option. Something like this is what I'm looking for:
var r = require('requirejs');
r('./build/common.js');
r('./build/app-main.js');
Let me explain the underlying motivation in case there's a better way to do it:
I've got a few different build.js files that I want to run r.js on (separate bundles for common dependencies and the main app). I'd like to wrap this up inside a gulpfile or gruntfile that runs both, and without putting all the r.js config in the actual grunt/gulp file like the grunt and gulp require.js plugins all seem to do. Leaving the r.js config in the separate build/*.js files would let us use grunt/gulp OR command line when we want to.
Any way to accomplish this?
Using the optimizer as a Node module is documented but it is not in the most evident place. This is the example that the documentation gives:
var requirejs = require('requirejs');
var config = {
baseUrl: '../appDir/scripts',
name: 'main',
out: '../build/main-built.js'
};
requirejs.optimize(config, function (buildResponse) {
//buildResponse is just a text output of the modules
//included. Load the built file for the contents.
//Use config.out to get the optimized file contents.
var contents = fs.readFileSync(config.out, 'utf8');
}, function(err) {
//optimization err callback
});