Environment: webpack 5.44 + vue.js 3.0 + node 12.21
I'm trying to generate a module at compile-time, in order to avoid a costly computation at run-time (as well as 10Mb of dependencies that will never be used except during said computation). Basically run this at compile-time:
import * as BigModule from "big-module";
function extract_info(module) { ... }
export default extract_info(BigModule);
which will be imported at run-time as:
export default [ /* static info */ ];
I tried using val-loader (latest 4.0) which seems designed exactly for this use case.
Problem: big-module is an ESM, but val-loader apparently only supports CJS. So I can neither import ("Cannot use import statement outside a module" error) nor require ("Unexpected token 'export'" error).
Is there any way to make val-loader somehow load the ESM module? Note that I'm not bent on using val-loader, any other technique that achieves the same goal is just as welcome.
After learning way more than I wanted about this issue and node/webpack internals, there seems to be two possible approaches to import ESM from CJS:
Use dynamic import(). But it is asynchronous which makes it unfit here, as val-loader requires a synchronous result.
Transpile the ESM into CJS, which is the approach I took.
In my case, full transpiling is overkill and rewriting imports/exports is sufficient, so I'm using ascjs to rewrite the ESM files, along with eval to safely evaluate the resulting string.
All in all:
// require-esm.js
const fs = require('fs');
const ascjs = require('ascjs');
const _eval = require('eval');
function requireESM(file) {
file = require.resolve(file);
return _eval(ascjs(fs.readFileSync(file)), file, { require: requireESM }, true);
}
module.exports = requireESM;
// val-loader-target.js
const requireESM = require('./require-esm');
const BigModule = requireESM('big-module');
function extract_info(module) { ... }
module.exports = extract_info(BigModule);
Note that:
ascjs is safe to use on CJS modules, since it only rewrites ESM imports/exports. So it's OK for big-module or its dependencies to require CJS files.
the third argument to _eval enables recursive rewriting, otherwise only the top-level file (the one passed to requireESM) is translated.
Related
The question is simple, how do we make es6 modules act like the ImportScript function used on the web browser.
Explanation
The main reason is to soften the blow for developers as they change their code from es5 syntax to es6 so that the transition does not blow up your code the moment you make the change and find out there are a thousand errors due to missing inclusions. It also give's people the option to stay as is indefinitely if you don't want to make the full change at all.
Desired output
ImportScript(A file path/'s); can be applied globally(implicitly) across subsequently required code and vise-verse inside a main file to avoid explicit inclusion in all files.
ES6 Inclusion
This still does not ignore the fact that all your libraries will depend on modules format as well. So it is inevitable that we will still have to include the export statement in every file we need to require. However, this should not limit us to the ability to have a main file that interconnects them all without having to explicitly add includes to every file whenever you need a certain functionality.
DISCLAIMER'S
(Numbered):
(Security) I understand there are many reasons that modules exist and going around them is not advisable for security reasons/load times. However I am not sure about the risk (if any) of even using a method like "eval()" to include such scripts if you are only doing it once at the start of an applications life and to a constant value that does not accept external input. The theory is that if an external entity is able to change the initial state of your program as is launched then your system has already been compromised. So as it is I think the whole argument around Globalization vs modules boils down to the project being done(security/speed needed) and preference/risk.
(Not for everyone) This is a utility I am not implying that everyone uses this
(Already published works) I have searched a lot for this functionality but I am not infallible to err. So If a simple usage of this has already been done that follows this specification(or simpler), I'd love to know how/where I can attain such code. Then I will promptly mark that as the answer or just remove this thread entirely
Example Code
ES5 Way
const fs = require('fs');
let path = require('path');
/* only accepts the scripts with global variables and functions and
does not work with classes unless declared as a var.
*/
function include(f) {
eval.apply(global, [fs.readFileSync(f).toString()])
}
Main file Concept example:
ImportScript("filePath1");loaded first
ImportScript("filePath2");loaded second
ImportScript("filePath3");loaded third
ImportScript("filePath4");loaded fourth
ImportScript("filePath5");loaded fifth
ImportScript("someExternalDependency");sixth
/* where "functionNameFromFile4" is a function defined in
file4 , and "variableFromFile2" is a global dynamic
variable that may change over the lifetime of the
application.
*/
functionNameFromFile4(variableFromFile2);
/* current context has access to previous scripts contexts
and those scripts recognize the current global context as
well in short: All scripts should be able to access
variables and functions from other scripts implicitly
through this , even if they are added after the fact
*/
Typical exported file example (Covers all methods of export via modules):
/*where "varFromFile1" is a dynamic variable created in file1
that may change over the lifetime of the application and "var" is a
variable of type(varFromFile4) being concatenated/added together
with "varFromFile4".
*/
functionNameFromFile4(var){
return var+varFromFile1;
}
//Typical export statement
exportAllHere;
/*
This is just an example and does not cover all usage cases , just
an example of the possible functionality
*/
CONCLUSION
So you still need to export the files as required by the es6 standard , however you only need to import them once in a main file to globalize their functionality across all scripts.
I'm not personally a fan of globalizing all the exports from a module, but here's a little snippet that shows you how one ESM module's exports can be all assigned to the global object:
Suppose you had a simple module called operators.js:
export function add(a, b) {
return a + b;
}
export function subtract(a, b) {
return a - b;
}
You can import that module and then assign all of its exported properties to the global object with this:
import * as m from "./operators.js"
for (const [prop, value] of Object.entries(m)) {
global[prop] = value;
}
// can now access the exports globally
add(1, 2);
FYI, I think the syntax:
include("filePath1")
is going to be tough in ESM modules because dynamic imports in an ESM module using import (which is presumably what you would have to use to implement the include() function you show) are asynchronous (they return a promise), not synchronous like require().
I wonder if a bundler or a transpiler would be an option?
There is experimental work in nodejs related to custom loaders here: https://nodejs.org/api/esm.html#hooks.
If you can handle your include() function returning a promise, here's how you put the above code into that function:
async function include(moduleName) {
const m = await import(moduleName);
for (const [prop, value] of Object.entries(m)) {
global[prop] = value;
}
return m;
}
I've written a library published to a private npm repo which is used by my applications.
This library contains utilities and has dependencies to other libraries, as an example let's choose #aws-sdk/client-lambda.
Some of my applications use only some of the utilities and don't need the dependencies to the external libraries, while some applications use all of the utilities.
To avoid having all applications getting a lot of indirect dependencies they don't need, I tried declaring the dependencies as peerDependencies and having the applications resolve the ones they need. It works well to publish the package, and to use it from applications who declare all of the peerDependencies as their own local dependencies, but applications failing to declare one of the dependencies get build errors when the included .d.ts files of the library are imported in application code:
error TS2307: Cannot find module '#aws-sdk/client-kms' or its corresponding type declarations.
Is it possible to resolve this situation so that my library can contain many different utils but the applications may "cherry-pick" the dependencies they need to fulfill the requirements of those utilities in runtime?
Do I have to use dynamic imports to do this or is there another way?
I tried using #ts-ignore in the library code, and it was propagated to the d.ts file imported by the applications, but it did not help.
Setup:
my-library
package.json:
peerDependencies: {
"#aws-sdk/client-lambda": "^3.27.0"
}
foo.ts:
import {Lambda} from '#aws-sdk/client-lambda';
export function foo(lambda: Lambda): void {
...
}
bar.ts:
export function bar(): void {
...
}
index.ts:
export * from './foo';
export * from './bar';
my-application1 - works fine
package.json:
dependencies: {
"my-library": "1.0.0",
"#aws-sdk/client-lambda": "^3.27.0"
}
test.ts:
import {foo} from 'my-library';
foo();
my-application2 - does not compile
package.json:
dependencies: {
"my-library": ...
}
test:ts:
import {bar} from 'my-library';
bar();
I found two ways of dealing with this:
1. Only use dynamic imports for the optional dependencies
If you make sure that types exported by the root file of the package only include types and interfaces and not classes etc, the transpiled JS will not contain any require statement to the optional library. Then use dynamic imports to import the optional library from a function so that they are required only when the client explicitly uses those parts of the library.
In the case of #aws-sdk/client-lambda, which was one of my optional dependencies, I wanted to expose function that could take an instance of a Lambda object or create one itself:
import {Lambda} from '#aws-sdk/client-lambda';
export function foo(options: {lambda?: Lambda}) {
if (!lambda) {
lambda = new Lambda({ ... });
}
...
}
Since Lambda is a class, it will be part of the transpiled JS as a require statement, so this does not work as an optional dependency. So I had to 1) make that import dynamic and 2) define an interface to be used in place of Lambda in my function's arguments to get rid of the require statement on the package's root path. Unfortunately in this particular case, the AWS SDK does not offer any type or interface which the class implements, so I had to come up with a minimal type such as
export interface AwsClient {
config: {
apiVersion: string;
}
}
... but of course, lacking a type ot represent the Lambda class, you might even resort to any.
Then comes the dynamic import part:
export async function foo(options: {lambda?: AwsClient}) {
if (!lambda) {
const {Lambda} = await import('#aws-sdk/client-lambda');
lambda = new Lambda({ ... });
}
...
}
With this code, there is no longer any require('#aws-sdk/client-lambda') on the root path of the package, only within the foo function. Only clients calling the foo function will have to have the dependency in their
node_modules.
As you can see, a side-effect of this is that every function using the optional library must be async since dynamic imports return promises. In my case this worked out, but it may complicate things. In one case I had a non-async function (such as a class constructor) needing an optional library, so I had no choice but to cache the promised import and resolve it later when used from an async member function, or do a lazy import when needed. This has the potential of cluttering code badly ...
So, to summarize:
Make sure any code that imports code from the optional library is put inside functions that the client wanting to use that functionality calls
It's OK to have imports of types from the optional library in the root of your package as it's stripped out when transpiled
If needed, defined substitute types to act as place-holders for any class arguments (as classes are both types and code!)
Transpile and investigate the resulting JS to see if you have any require statement for the optional library in the root, if so, you've missed something.
Note that if using webpack etc, using dynamic imports can be tricky as well. If the import paths are constants, it usually works, but building the path dynamically (await import('#aws-sdk/' + clientName)) will usually not unless you give webpack hints. This had me puzzled for a while since I wrote a wrapper in front of my optional AWS dependencies, which ended up not working at all for this reason.
2. Put the files using the optional dependencies in .ts files not exported by the root file of the package (i.e., index.ts).
This means that clients wanting to use the optional functionality must
import those files by sub-path, such as:
import {OptionalStuff} from 'my-library/dist/optional;
... which is obviously less than ideal.
in my case, the typescript IDE in vscode fails to import the optional type, so im using the relative import path
// fix: Cannot find module 'windows-process-tree' or its corresponding type declarations
//import type * as WindowsProcessTree from 'windows-process-tree';
import type * as WindowsProcessTree from '../../../../../node_modules/#types/windows-process-tree';
// global variable
let windowsProcessTree: typeof WindowsProcessTree;
if (true) { // some condition
windowsProcessTree = await import('windows-process-tree');
windowsProcessTree.getProcessTree(rootProcessId, tree => {
// ...
});
}
package.json
{
"devDependencies": {
"#types/windows-process-tree": "^0.2.0",
},
"optionalDependencies": {
"windows-process-tree": "^0.3.4"
}
}
based on vscode/src/vs/platform/terminal/node/windowsShellHelper.ts
I'm trying to use my rust module from the rust webassembly book in my gatsby project. When I try to import the module like so:
import { <rust-struct> } from 'rust_wasm_npm_package';
I get the following error:
The module seem to be a WebAssembly module, but module is not flagged as WebAssembly module for
webpack.
BREAKING CHANGE: Since webpack 5 WebAssembly is not enabled by default and flagged as experimental
feature.
You need to enable one of the WebAssembly experiments via 'experiments.asyncWebAssembly: true' (based
on async modules) or 'experiments.syncWebAssembly: true' (like webpack 4, deprecated).
For files that transpile to WebAssembly, make sure to set the module type in the 'module.rules'
section of the config (e. g. 'type: "webassembly/async"').
(Source code omitted for this binary file)
I'm unable to add the experiments option to the gatsby config file, so I'm not sure what is the best way to import a wasm-pack rust module into gatsby.
I was able to get this working by adding a gatsby-node.js file with the following code:
exports.onCreateWebpackConfig = ({ actions }) => {
actions.setWebpackConfig({
experiments: {
syncWebAssembly: true,
},
});
};
I was then able to import the web assembly asynchronously. Not sure why I did not need to use asyncWebassembly: true instead, but it works!
// The reason for this useless concatenation
// is to get rid of a really specific issue
// with Webpack and WASM modules being imported
// all in one line.
/*eslint no-useless-concat: "off"*/
const module = await import("path/" + "toJSFile.js");
const memModule = await import("path/" + "toWasmModule.wasm");
const memory = memModule.memory;
setMem(memory);
When I try to import node.js module in TypeScript like this:
import co = require('co');
import co from 'co';
without providing type definitions, both lines reports same error:
error TS2307: Cannot find module 'co'.
How to import it correctly?
The trick is to use purely JavaScript notation:
const co = require('co');
Your options are to either import it outside TypeScript's module system (by calling a module API like RequireJS or Node directly by hand) so that it doesn't try to validate it, or to add a type definition so that you can use the module system and have it validate correctly. You can stub the type definition though, so this can be very low effort.
Using Node (CommonJS) imports directly:
// Note there's no 'import' statement here.
var loadedModule: any = require('module-name');
// Now use your module however you'd like.
Using RequireJS directly:
define(["module-name"], function (loadedModule: any) {
// Use loadedModule however you'd like
});
Be aware that in either of these cases this may mix weirdly with using real normal TypeScript module imports in the same file (you can end up with two layers of module definition, especially on the RequireJS side, as TypeScript tries to manage modules you're also managing by hand). I'd recommend either using just this approach, or using real type definitions.
Stubbing type definitions:
Getting proper type definitions would be best, and if those are available or you have time to write them yourself you should definitely should.
If not though, you can just give your whole module the any type, and put your module into the module system without having to actually type it:
declare module 'module-name' {
export = <any> {};
}
This should allow you to import module-name and have TypeScript know what you're talking about. You'll still need to ensure that importing module-name does actually load it successfully at runtime with whatever module system you're using, or it will compile but then fail to actually run.
I got an error when I used the "Stubbing type definitions" approach in Tim Perry's answer: error TS2497: Module ''module-name'' resolves to a non-module entity and cannot be imported using this construct.
The solution was to rework the stub .d.ts file slightly:
declare module 'module-name' {
const x: any;
export = x;
}
And then you can import via:
import * as moduleName from 'module-name';
Creating your own stub file lowers the barrier to writing out real declarations as you need them.
Just import the module the following way:
import 'co';
I need to check if the module that is going to be loaded is a built-in or an external module. For example, suppose that you have a module named fs inside the node_modules directory. If you do require("fs") the built-in module will be loaded instead of the module inside node_modules, so I'm sure that this question has a solution.
Example:
var loadModule = function (moduleName){
if (isCoreModule (moduleName)){
...
}else{
...
}
};
loadModule ("fs");
process.binding('natives'); returns an object that provides access to all built-in modules, so getting the keys of this object will get you the module names. So you could simply do something like:
var nativeModules = Object.keys(process.binding('natives'));
function loadModule(name) {
if (~nativeModules.indexOf(name)) {
// `name` is a native module name
} else {
// ...
}
};
loadModule('fs');
you can use require.resolve.paths(str):
1- if str is core module the call will return null.
2- if str is not core module you will get an array of strings.
My first attempt would be: require.resolve(moduleName).indexOf('/') <= 0. If that is true, it's a core module. Might not be portable to windows as implemented, but you should be able to use this idea to get going in the right direction.
Aside: beware require.resolve does synchronous filesystem IO. Be careful about using it within a network server.
Aside: Careful with the term "native" which generally means a native-code compiled add-on, or a C/C++ implementation. Both "core" and community modules can be either pure JS or native.
I think "core" is the most accurate term for built-in modules.
Aside: best not to shadow global variable names, so moduleName instead of just module which can be confusing with the global of the same name.
Module.isBuiltin Added in: v18.6.0, v16.17.0
import { isBuiltin } from 'node:module';
isBuiltin('node:fs'); // true
isBuiltin('fs'); // true
isBuiltin('fs/promises'); // true
isBuiltin('wss'); // false