I made a nodeJS app using puppeteer-extra and some puppeteer-extra-plugins, debugging the source code works just fine. It also runs OK when I pack it into an .exe file using https://www.npmjs.com/package/pkg.
However, when the .exe is moved to a different directory than the 'output' directory, I'm getting the error
Cannot find module 'debug'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings
and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable
and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call
using process.cwd() or process.execPath.
I never use the debug module, it must be required internally by some of my dependencies... but how can I handle this error?
I've got these requires
const puppeteer = require(path.join(process.cwd(), 'dependencies', 'puppeteer-extra'));
const StealthPlugin = require(path.join(process.cwd(), 'dependencies', 'puppeteer-extra-plugin-stealth'));
const RecaptchaPlugin = require(path.join(process.cwd(), 'dependencies', 'puppeteer-extra-plugin-recaptcha'));
const UserPrefsPlugin = require(path.join(process.cwd(), 'dependencies', 'puppeteer-extra-plugin-user-preferences'));
and my package.json (config for pkg)
"pkg": {
"scripts": [
"node_modules/puppeteer/lib/*.js"
],
"assets": [
"./node_modules/#types",
"./node_modules/typescript/lib/*.d.ts",
"src/**/*.ts",
"./tsconfig.json"
],
"targets": [
"node8-win32"
]
}
I solved it by changing my requires to
const puppeteer = require('puppeteer-extra');
const RecaptchaPlugin = require('puppeteer-extra-plugin-recaptcha');
const UserPrefsPlugin = require('puppeteer-extra-plugin-user-preferences');
and later, launching the browser setting explicitly the executablePath in puppeteer's options.
const options = {
headless: false,
devtools: false,
args: ['--start-maximized'],
timeout: 300000,
downloadPath: path.join(process.env.USERPROFILE, 'Downloads'),
askDownload: false,
executablePath: path.join(process.cwd(), "puppeteer-extra",
"node_modules\\puppeteer\\.local-chromium\\win64-722234\\chrome-win\\chrome.exe")
}
await puppeteer.launch(this.options)
Note however, that I had to leave out the 'puppeteer-extra-plugin-stealth', because it complained about
"A plugin listed
'puppeteer-extra-plugin-stealth/evasions/chrome.runtime' as
dependency, which is currently missing. Please install it:"
and adding a pkg/asset as suggested here, did not help
Related
I am working on an Obsidian plugin that requires bundling using rollup.js. This plugin needs to import inrupt solid libraries that, when imported, are causing the following error:
Error: crypto.getRandomValues() not supported. See https://github.com/uuidjs/uuid#getrandomvalues-not-supported
at rng (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:16740:10)
at v4 (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:17186:53)
at new Session (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:23526:88)
at Repro.<anonymous> (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:23669:17)
at step (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:160:15)
at Object.next (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:111:11)
at /Users/candide/work/sekund/solid-build-issues/main_rollup.js:83:65
at new Promise (<anonymous>)
at __awaiter (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:65:9)
at Repro.login (/Users/candide/work/sekund/solid-build-issues/main_rollup.js:23664:10)
When debugging the code, it turns out that the crypto constant is not defined. However, at the start of the generated bundle, I can see:
var crypto_1 = require("crypto");
So it looks like my problem basically boils down to rollup redefining global variables when it should not.
Indeed, using the typescript compiler (tsc) on the same source file outputs a perfectly working program.
Here's my rollup config:
import commonjs from "#rollup/plugin-commonjs";
import json from "#rollup/plugin-json";
import { nodeResolve } from "#rollup/plugin-node-resolve";
import typescript from "#rollup/plugin-typescript";
const banner = `/*
THIS IS A GENERATED/BUNDLED FILE BY ROLLUP
if you want to view the source visit the plugins github repository
*/
`;
export default {
input: "src/main.ts",
inlineDynamicImports: true,
output: [
{
file: "main.js",
sourcemap: "inline",
format: "cjs",
exports: "default",
banner,
},
],
external: ["obsidian", "fs", "os", "path"],
plugins: [json(), nodeResolve({ preferBuiltins: true }), commonjs(), typescript({ sourceMap: true })],
};
I created a repro repo at https://github.com/ckemmler/solid-build-issues
man this wrecked my brain for a bit, i managed to find a solution but i don't understand it 100%
please see the solution suggested in this comment https://github.com/uuidjs/uuid/issues/544#issuecomment-740394448
resolved the problem for me
When running jest locally, it instantiates my app and runs tests without any issues.
When running jest inside github actions, I'm getting this error:
Error: Jest: Failed to parse the TypeScript config file /home/runner/work/myproject/myproject/jest.config.ts
TypeError: registerer.enabled is not a function
at readConfigFileAndSetRootDir (/home/runner/work/myproject/myproject/node_modules/#jest/core/node_modules/jest-config/build/readConfigFileAndSetRootDir.js:118:13)
the package.json script entry is just:"test": "jest"
and the jest.config.ts file is:
import tsJestUtils from 'ts-jest/utils'
import tsConf from './tsconfig.json'
const rootDir = __dirname
const { pathsToModuleNameMapper } = tsJestUtils
const {
compilerOptions: { paths },
} = tsConf
const config = {
preset: 'ts-jest',
testEnvironment: 'node',
roots: [`${rootDir}/src`],
transform: {
'^.+\\.tsx?$': 'ts-jest',
},
moduleNameMapper: pathsToModuleNameMapper(paths, {
prefix: `${rootDir}/src`,
}),
}
export default config
So I just bypassed use of typescript for my jest config entirely, and went with an equivalent jest.config.js file based on the docs. Works in Github Actions now, runner does not fail! \o/
I am still not sure what the issue was, but I think ts-node just wasn't processing the config file properly. I feel like the actual failure was with the attempt to load a .ts config file, specifically at this point in the source code when it tries to call registerer.enabled().
It can be fixed by upgrading to "ts-node": "^8.5.0"
I am having a bit of trouble reconciling the path of a dynamic import for i18n locales. Here's the relevant code -
function getLoader(
lang: SupportedLanguage,
ns: SupportedNamespace
): NamespaceLoader | undefined {
const matrixToCheck = UNSUPPORTED_MATRIX[ns];
const isSupported = matrixToCheck && matrixToCheck.indexOf(lang) === -1;
if (isSupported) {
const path = `./locales/${lang}/${ns}.json`;
const name = `${lang}_${ns}`;
const named = {
[name]: () => import(`${path}`),
};
return named[name];
}
}
...
// eventual output
const SUPPORTED_LANGUAGES = {en: {namespace1: () => import('./locales/en/namespace1.json')}
My goal is manage all of the relevant translations in a single npm package, handle all of the dynamic import set-up at build time, and then consumers can invoke the getter (getTranslation in this case) in their respectives apps for the language and namespace of their choice to get the payload at runtime.
Based on this GH thread, I wanted to reconcile the locale dist path via the package.json
...
"exports": {
".": "./dist/src/main.js",
"./": "./dist/"
},
...
e.g. when I publish the package, based on that exports config, the consumer would know know how to reconcile the path, either relative or package-name-prefix when the getter is invoked
const fn = () => import('./locales/fr/myNamespace.json') /// doesn't work
const anotherFn = () => import('#examplePackageName/locales/fr/myNamespace.json') /// doesn't work
Since everything is dynamic, I am using the CopyWebpackPlugin to include the locales in the dist folder.
This works as expected locally, but when I create the dist, I get the error Error: Module not found ./relative/path/to/the/json/I/want.json.
My question:
What am I missing? Is there a simple way to expose these translations so that other apps can include them in their bundles via an npm-installed package?
Here's my Webpack config, happy to provide other info as needed
const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");
const { CleanWebpackPlugin } = require("clean-webpack-plugin");
const getPlugins = () => {
return [
new CleanWebpackPlugin(),
new CopyPlugin({
patterns: [{ from: "locales", to: "locales" }],
}),
];
};
module.exports = {
mode: "production",
entry: {
main: "./src/main.ts",
},
output: {
path: path.join(__dirname, "dist"),
filename: "src/[name].js",
chunkFilename: "chunk.[name].js",
libraryTarget: "commonjs2",
},
resolve: {
extensions: [".json", ".ts", ".js"],
alias: {
"#locales": path.resolve(__dirname, "locales/*"),
},
},
plugins: getPlugins(),
module: {
rules: [
{
test: /\.ts$/,
exclude: [/\.test\.ts$/],
include: path.join(__dirname, "src"),
loader: "ts-loader",
},
],
},
};
Exports directive prescribes to define all files allowed for import explicitly (documentation). It allows developer to hide internal package file structure. What's not exported by this directive is only available to import inside the package and not outside of it. It's made to simplify maintenance. It allows developers to rename files or change file structure without fear of breaking dependent packages and applications.
So if you want to make internal files visible for import, you should export them with exports directive explicitly, like this:
{
"exports": {
".": "./dist/esm/src/main.js",
"./dist/shared/locale/fr_fr.json": "./dist/shared/locale/fr_fr.json"
}
}
I'm not sure wether Webpack handling this case, because it's an experimental feature yet. But this is how Node.js works now.
Why it is so
Changing your app file structure is a major change in semver terms, so you need to bump a version everytime you rename or delete files. To avoid it you can specify which files are part of public interface of the package.
const invokeApi = require("/opt/nodejs/kiwiCall");
const decrypt = require("/opt/nodejs/encryption");
const cors = require("/opt/nodejs/cors");
When I am testing my index.js file by manual mocking these dependencies in mocks directory as follows:
__mocks__
|_invokeApi
|_decrypt
|_cors
it says
FAIL ./index.test.js
● Test suite failed to run
Cannot find module '/opt/nodejs/kiwiCall' from 'index.js'
However, Jest was able to find:
'../../../../lambdas/Flights/Locations/index.js'
You might want to include a file extension in your import, or update your 'moduleFileExtensions', which is currently ['js', 'json', 'jsx', 'ts', 'tsx', 'node'].
See https://jestjs.io/docs/en/configuration#modulefileextensions-array-string
1 | "use strict";
2 |
> 3 | const invokeApi = require("/opt/nodejs/kiwiCall");
Wanted to know how can I mock the dependencies of AWS lambda in inedx.test.js file
In your package.json or jest.config you could add a moduleNameMapper for that directory.
"jest": {
"moduleNameMapper": {
"/opt/nodejs/(.*)": "<rootDir>/../nodejs/$1"
},
},
So I managed to figure out something based on my repository.
I'm using the moduleNameMapper to map the absolute path to another location in my repository to where I have the layer stored.
Eg.
moduleNameMapper: {'^/opt/config/config': '<rootDir>/src/layers/layers-core/config/config'}
In your case you could use a regex expression to match /opt/nodejs/ and map it elsewhere. Hope that helped.
EDIT:
I completely changed my approach and used babel-plugin-module-resolver with babel-rewire. I did this because the above method was incompatible with rewire. It's quite easy setup and you just need to setup a babel alias within .babelrc.
eg.
{
"plugins": [
["rewire"],
["babel-plugin-module-resolver", {
"alias": {
"/opt/config/config": "./src/layers/layers-core/config/config",
"/opt/utils/util-logger": "./src/layers/layers-core/utils/util-logger",
"/opt/slack": "./src/layers/layers-slack/slack"
}
}]
]
}
Combine this with IDE jsconfig.json path alias and you get full IDE support.
{
"compilerOptions": {
"module": "commonjs",
"target": "es2018",
"baseUrl": "./",
"paths": {
"/opt/config/config": ["src/layers/layers-core/config/config"],
"/opt/utils/util-logger": ["src/layers/layers-core/utils/util-logger"],
"/opt/slack/*": ["src/layers/layers-slack/slack/*"],
}
},
"exclude": ["node_modules", "dist"]
}
You can then reference your layers with jest.doMock('/opt/config/config', mockConfig);
EDIT 2:
Found a way to get Jest to mock it. Just slip {virtual: true} into the mock!
jest.doMock('/opt/config/config', mockConfig, {virtual: true});
I have pretty much the same issue. I have defined a layer which contains common code that's shared between other functions in my project. My project structure looks something like this:
project/
functions/
function1/
app.js
function2/
app.js
shared/
shared.js
I import my shared library like this:
const { doSomething } = require('/opt/shared');
exports.handler = async (event) => {
const result = await doSomething();
// etc...
return {statusCode: 200};
}
This works when I deploy to AWS Lambda because the /opt/shared exists and it can be referenced correctly. It also works if I run this on my machine using sam local invoke Function1 because it's running in a container, which makes /opt/shared available to the code.
However, I'm struggling to work out how I can mock this dependency in a unit test. If I simply do this: jest.mock('/opt/shared'), I'm getting: Cannot find module '/opt/shared' from app.test.js
You can use the modulePaths option, from this post.
Documentation
jest.config.js
"jest": {
"modulePaths": [
"<rootDir>/src/layers/base/nodejs/node_modules/"
]
}
You can dynamically create this array by scanning a directory
const testFolder = './functions/';
const fs = require('fs');
const modulePaths = fs.readdirSync(testFolder)
.reduce((modulePaths, dirName) => {
modulePaths.push(`functions/${dirName}/dependencies/nodejs/node_modules/`);
return modulePaths;
}, []);
I am trying to implement webpack in my project which contains node-red. However, I keep getting the following warning. Please suggest how to solve this error -
WARNING in ./node_modules/node-red/red/runtime/storage/localfilesystem/projects/git/node-red-ask-pass.sh 1:26
Module parse failed: Unexpected token (1:26)
You may need an appropriate loader to handle this file type.
> "$NODE_RED_GIT_NODE_PATH" "$NODE_RED_GIT_ASKPASS_PATH" "$NODE_RED_GIT_SOCK_PATH" $#
|
# ./node_modules/node-red/red/runtime/storage sync ^\.\/.*$ ./localfilesystem/projects/git/node-red-ask-pass.sh
# ./node_modules/node-red/red/runtime/storage/index.js
# ./node_modules/node-red/red/runtime/index.js
# ./app.js
My webpack.config.js is -
const path = require('path');
var nodeExternals = require('webpack-node-externals');
module.exports = {
target: 'node',
externals: [nodeExternals()],
entry: './app.js',
output: {
path: path.resolve(__dirname, './output'),
filename: 'bundle.js'
},
resolve: {
extensions: ['.js','.json', '.sh'],
modules: [
'node_modules'
],
},
module: {
rules: [
{
test:/\.css$/,
use:['style-loader','css-loader']
},
{
test: /\.coffee$/,
use: [ 'coffee-loader' ]
}
]
}
};
For Webpack, every file is a .js. In order to handle other extensions, like .css or .sh, you're supposed to use a loader, like you did with css-loader, that will tranform CSS rules into JS.
The issue you're facing is that you've got an import chain (./app.js -> .../index.js -> .../index.js -> .../node-red-ask-pass.sh), so Webpack will, at some point, will import a .sh file, but will throw an error because shell code is obviousouly invalid JavaScript. that is why you're seeing the error that you have.
By the way, I couldn't reproduce the issue you're facing:
npm init -y
npm i node-red
# ./node_modules/node-red/red is not a directory
So it was probably a node-red bug. Update the package to the latest version.