I wrote this little example:
import { isBrowser, isNode } from "browser-or-node";
// runs at load
async function run() {
if (isBrowser) {
document.body.style.backgroundColor = "black";
} else if (isNode) {
const fs = await import("./myLittleScript");
console.log(fs);
}
}
run()
And ./myLittleScript is actually importing the node filesystem module to use if the user runs the script in node.
Is this idea wrong ? What is the right approach ?
I get an error that the script is not found, even though it is in the same directory than the index.ts which is the code pasted here.
This is when bundling it with webpack, but I am interested in knowing whether this is an acceptable approach or not. (I know I could just write 2 functions.)
Related
I created a test WASM program using Go. In the program's main, it adds an API to the "global" and waits on a channel to avoid from exiting. It is similar to the typical hello-world Go WASM that you can find anywhere in the internet.
My test WASM program works well in Browsers, however, I hope to run it and call the API using Node.js. If it is possible, I will create some automation tests based on it.
I tried many ways but I just couldn't get it work with Node.js. The problem is that, in Node.js, the API cannot be found in the "global". How can I run a GO WASM program (with an exported API) in Node.js?
(Let me know if you need more details)
Thanks!
More details:
--- On Go's side (pseudo code) ---
func main() {
fmt.Println("My Web Assembly")
js.Global().Set("myEcho", myEcho())
<-make(chan bool)
}
func myEcho() js.Func {
return js.FuncOf(func(this js.Value, apiArgs []js.Value) any {
for arg := range(apiArgs) {
fmt.Println(arg.String())
}
}
}
// build: GOOS=js GOARCH=wasm go build -o myecho.wasm path/to/the/package
--- On browser's side ---
<html>
<head>
<meta charset="utf-8"/>
</head>
<body>
<p><pre style="font-family:courier;" id="my-canvas"/></p>
<script src="wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(fetch("myecho.wasm"), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
// it also works without "window."
document.getElementById("my-canvas").innerHTML = window.myEcho("hello", "ahoj", "ciao");
})
})
</script>
</body>
</html>
--- On Node.js' side ---
globalThis.require = require;
globalThis.fs = require("fs");
globalThis.TextEncoder = require("util").TextEncoder;
globalThis.TextDecoder = require("util").TextDecoder;
globalThis.performance = {
now() {
const [sec, nsec] = process.hrtime();
return sec * 1000 + nsec / 1000000;
},
};
const crypto = require("crypto");
globalThis.crypto = {
getRandomValues(b) {
crypto.randomFillSync(b);
},
};
require("./wasm_exec");
const go = new Go();
go.argv = process.argv.slice(2);
go.env = Object.assign({ TMPDIR: require("os").tmpdir() }, process.env);
go.exit = process.exit;
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
}).catch((err) => {
console.error(err);
process.exit(1);
});
This pseudo code represents 99% content of my real code (only removed business related details). The problem is that I not only need to run the wasm program (myecho.wasm) by Node.js, but I also need to call the "api" (myEcho), and I need to pass it parameters and receive the returned values, because I want to create automation tests for those "api"s. With Node.js, I can launch the test js scripts and validate the outputs all in the command line environment. The browser isn't a handy tool for this case.
Running the program by node wasm_exec.js myecho.wasm isn't enough for my case.
It would be nice to know more details about your environment and what are you actually trying to do. You can post the code itself, compilation commands, and versions for all the tools involved.
Trying to answer the question without these details:
Go WASM is very browser oriented, because the go compiler needs the glue js in wasm_exec.js to run. Nodejs shouldn't have a problem with that, and the following command should work:
node wasm_exec.js main.wasm
where wasm_exec.js is the glue code shipped with your go distribution, usually found at $(go env GOROOT)/misc/wasm/wasm_exec.js, and main.wasm is your compiled code. If this fails, you can post the output as well.
There is another way to compile go code to wasm that bypasses wasm_exec.js, and that way is by using the TinyGo compiler to output wasi-enabled code. You can try following their instructions to compile your code.
For example:
tinygo build -target=wasi -o main.wasm main.go
You can build for example a javascript file wasi.js:
"use strict";
const fs = require("fs");
const { WASI } = require("wasi");
const wasi = new WASI();
const importObject = { wasi_snapshot_preview1: wasi.wasiImport };
(async () => {
const wasm = await WebAssembly.compile(
fs.readFileSync("./main.wasm")
);
const instance = await WebAssembly.instantiate(wasm, importObject);
wasi.start(instance);
})();
Recent versions of node have experimental wasi support:
node --experimental-wasi-unstable-preview1 wasi.js
These are usually the things you would try with Go and WASM, but without further details, it is hard to tell what exactly is not working.
After some struggling, I noticed that the reason is simpler than I expected.
I couldn't get the exported API function in Node.js simply because the API has not been exported yet when I tried to call them!
When the wasm program is loaded and started, it runs in parallel with the caller program (the js running in Node).
WebAssembly.instantiate(...).then(...go.run(result.instance)...).then(/*HERE!*/)
The code at "HERE" is executed too early and the main() of the wasm program hasn't finished exporting the APIs yet.
When I changed the Node script to following, it worked:
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
let retry = setInterval(function () {
if (typeof(go.exports.myEcho) != "function") {
return;
}
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
clearInterval(retry);
}, 500);
}).catch((err) => {
console.error(err);
process.exit(1);
});
(only includes the changed part)
I know it doesn't seem to be a perfect solution, but at least it proved my guess about the root cause to be true.
But... why it didn't happen in browser? sigh...
We're building a NodeJS Framework MidwayJS, and we recommend our users to use Jest to make test cases. We modified Jest Env like this:
'use strict';
const NodeEnvironment = require('jest-environment-node');
/* eslint-disable no-useless-constructor */
class JestEnvironment extends NodeEnvironment {
constructor(config) {
super(config);
}
async setup() {
require('ts-node/register');
this.global.process.env.MIDWAY_TS_MODE = 'true';
this.global.process.env.MIDWAY_JEST_MODE = 'true';
this.global.setTimeout(3000)
await super.setup();
}
async teardown() {
await super.teardown();
}
runScript(script) {
return super.runScript(script);
}
}
module.exports = JestEnvironment;
With configuration above it goes well, but we got a problem:
Some users create huge project directory and our framework will scan the whole directory before application start, when running test cases it acts also in this way, which makes it in some suitations take more than 3000ms--Jest set default async callback called time to be 3000ms, so if the scanning didn't finished in 3000ms the test would crash.
we solved this easily by use jest.setup.js in one line : jest.setTimeout(30000), but now we want to solve it in the file which showed in the first, and I didn't find a way to modify setup config in it.
I'd appreciate it greatly if you can help me to make it out.
I apologise for the phrasing of the question - it's a bit difficult to sum up as a question - please feel free to edit it if you can clarify. Also, as this quite a complex and long query - thank you to all those who are putting in the time to read through it!
I have 4 files (listed with directory tree from project root) as part of a project I'm building which aims to scrape blockchains and take advantage of multiple cores do get the job done:
./main.js
./scraper.js
./api/api.js
./api/litecoin_api.js
main.js
const { scraper } = require('./scraper.js')
const blockchainCli = process.env.BLOCKSCRAPECLI || 'litecoin-cli'
const client = (args) => {
// create child process which returns a promise which resolves after
// data has finished buffering from locally hosted node using cli
let child = spawn(`${blockchainCli} ${args.join(' ')}`, {
shell: true
})
// ... wrap command in a promise here, etc
}
const main = () => {
// count cores, spawn a worker per core using node cluster, add
// message handlers, then begin scraping blockchain with each core...
scraper(blockHeight)
}
main()
module.exports = {
client,
blockchainCli
}
scraper.js
const api = require('./api/api.js')
const scraper = async (blockHeight) => {
try {
let blockHash = await api.getBlockHashByHeight(blockHeight)
let block = await api.getBlock(blockHash)
// ... etc, scraper tested and working, writes to shared writeStream
}
module.exports = {
scraper
}
api.js
const { client, blockchainCli } = require('../main.js')
const litecoin = require('./litecoin_api')
let blockchain = undefined
if (blockchainCli === 'litecoin-cli' || blockchainCli === 'bitcoin-cli') {
blockchain = litecoin
}
// PROBLEM HERE: blockchainCli (and client) are both undefined if and
// only if running scraper from main.js (but not if running scraper
// from scraper.js)
const decodeRawTransaction = (txHash) => {
return client([blockchain.decodeRawTransaction, txHash])
}
const getBlock = (blockhash) => {
return client([blockchain.getBlock, blockhash])
}
const getBlockHashByHeight = (height) => {
return client([blockchain.getBlockHash, height])
}
const getInfo = () => {
return client([blockchain.getInfo])
}
const getRawTransaction = (txHash, verbose = true) => {
return client([blockchain.getRawTransaction, txHash, verbose])
}
module.exports = {
decodeRawTransaction,
getBlock,
getBlockHashByHeight,
getInfo,
getRawTransaction
}
So, I've taken out most the noise in the files which I don't think is necessary but it's open source so if you need more take a look here.
The problem is that, if I start the scraper from inside scraper.js by doing, say, something like this: scraper(1234567) it works like a charm and outputs the expected data to a csv file.
However if I start the scraper from inside the main.js file, I get this error:
Cannot read property 'getBlockHash' of undefined
at Object.getBlockHashByHeight (/home/grayedfox/github/blockscrape/api/api.js:19:29)
at scraper (/home/grayedfox/github/blockscrape/scraper.js:53:31)
at Worker.messageHandler (/home/grayedfox/github/blockscrape/main.js:81:5)
I don't know why, when launching the scraper from main.js, the blockchain is undefined. I thought it might be from the destructuring, but removing the curly braces from around the first line in the example main.js file doesn't change anything (same error).
Things are a bit messy at the moment (in the middle of developing this branch) - but the essential problem now is that it's not clear to me why the require would fail (cannot see variables inside main.js) if it's used in the following way:
main.js (execute scraper()) > scraper.js > api.js
But not fail (can see variables inside main.js) if it's run like this:
scraper.js (execute scraper()) > api.js
Thank you very much for your time!
You have a circular dependency between main and api, each requiring in the other. main requires api through scraper and api directly requires main. That causes things not to work.
You have to remove the circular dependency by putting common shared code into its own module that can be included by both, but doesn't include others that include it. It just needs better modularity.
I have node.js application written in TypeScript and I need to switch between two interface implementations based on the config file. Currently I have this code which seems to be working.
"use strict";
import { config } from "../config";
let Something;
if (config.useFakeSomething) {
Something = require("./FakeSomething").FakeSomething;
} else {
Something = require("./RealSomething").RealSomething;
}
...
let s = new Something();
s.DoStuff();
...
But I have bad gut feeling about that (mainly because of the mixing require and import for loading modules). Is there some other way how to achieve implementation switching based on config file without importing both modules?
If you want to keep the client-code for your Something class clean, you can move the conditional importing to a single file. You can have the following directory structure for your Something module:
/Something
RealSomething.ts
FakeSomething.ts
index.ts
And in your index.ts, you can have the following:
import { config } from '../config';
const Something = config.useFakeSomething ?
require('./FakeSomething').FakeSomething :
require('./RealSomething').RealSomething;
export default Something;
And in your client code, you can just import Something:
import Something from './Something/index';
I can see nothing wrong with your approach. In fact lines like
import { config } from "../config";
When targeting commonjs will be compiled to the following javascript (ES6):
const config = require('../config');
So they are effectively identical and you are not mixing different module loading techniques.
You can do it like that:
let moduleLoader:any;
if( pladform == 1 ) {
moduleLoader = require('./module1');
} else {
moduleLoader = require('./module2');
}
and then
if( pladform === 1 ) {
platformBrowserDynamic().bootstrapModule(moduleLoader.module1, [ languageService ]);
}
else if ( pladform === 2 ) {
platformBrowserDynamic().bootstrapModule(moduleLoader.module2, [ languageService ]);
}
In addition to the correct answers above, in case you need this switching for many files within a single folder, you can use a symbolic-link (not available on Windows) that referenced to the right folder, so your code remains clean.
This approach is good for switching between real code and stubs, for instance
A modern answer using the import() function:
import { config } from "../config";
let Something;
if (config.useFakeSomething) {
Something = (await import("./FakeSomething")).FakeSomething;
} else {
Something = (await import("./RealSomething")).RealSomething;
}
...
let s = new Something();
s.DoStuff();
...
Remember that import() is non blocking, so you need to add await if the code that follows needs the result from the import().
maybe this question is a little silly, but is it possible to load multiple .js files with one require statement? like this:
var mylib = require('./lib/mylibfiles');
and use:
mylib.foo(); //return "hello from one"
mylib.bar(): //return "hello from two"
And in the folder mylibfiles will have two files:
One.js
exports.foo= function(){return "hello from one";}
Two.js
exports.bar= function(){return "hello from two";}
I was thinking to put a package.json in the folder that say to load all the files, but I don't know how. Other aproach that I was thinking is to have a index.js that exports everything again but I will be duplicating work.
Thanks!!
P.D: I'm working with nodejs v0.611 on a windows 7 machine
First of all using require does not duplicate anything. It loads the module and it caches it, so calling require again will get it from memory (thus you can modify module at fly without interacting with its source code - this is sometimes desirable, for example when you want to store db connection inside module).
Also package.json does not load anything and does not interact with your app at all. It is only used for npm.
Now you cannot require multiple modules at once. For example what will happen if both One.js and Two.js have defined function with the same name?? There are more problems.
But what you can do, is to write additional file, say modules.js with the following content
module.exports = {
one : require('./one.js'),
two : require('./two.js'),
/* some other modules you want */
}
and then you can simply use
var modules = require('./modules.js');
modules.one.foo();
modules.two.bar();
I have a snippet of code that requires more than one module, but it doesn't clump them together as your post suggests. However, that can be overcome with a trick that I found.
function requireMany () {
return Array.prototype.slice.call(arguments).map(function (value) {
try {
return require(value)
}
catch (event) {
return console.log(event)
}
})
}
And you use it as such
requireMany("fs", "socket.io", "path")
Which will return
[ fs {}, socketio {}, path {} ]
If a module is not found, an error will be sent to the console. It won't break the programme. The error will be shown in the array as undefined. The array will not be shorter because one of the modules failed to load.
Then you can bind those each of those array elements to a variable name, like so:
var [fs, socketio, path] = requireMany("fs", "socket.io", "path")
It essentially works like an object, but assigns the keys and their values to the global namespace. So, in your case, you could do:
var [foo, bar] = requireMany("./foo.js", "./bar.js")
foo() //return "hello from one"
bar() //return "hello from two"
And if you do want it to break the programme on error, just use this modified version, which is smaller
function requireMany () {
return Array.prototype.slice.call(arguments).map(require)
}
Yes, you may require a folder as a module, according to the node docs. Let's say you want to require() a folder called ./mypack/.
Inside ./mypack/, create a package.json file with the name of the folder and a main javascript file with the same name, inside a ./lib/ directory.
{
"name" : "mypack",
"main" : "./lib/mypack.js"
}
Now you can use require('./mypack') and node will load ./mypack/lib/mypack.js.
However if you do not include this package.json file, it may still work. Without the file, node will attempt to load ./mypack/index.js, or if that's not there, ./mypack/index.node.
My understanding is that this could be beneficial if you have split your program into many javascript files but do not want to concatenate them for deployment.
You can use destructuring assignment to map an array of exported modules from require statements in one line:
const requires = (...modules) => modules.map(module => require(module));
const [fs, path] = requires('fs', 'path');
I was doing something similar to what #freakish suggests in his answer with a project where I've a list of test scripts that are pulled into a Puppeteer + Jest testing setup. My test files follow the naming convention testname1.js - testnameN.js and I was able use a generator function to require N number of files from the particular directory with the approach below:
const fs = require('fs');
const path = require('path');
module.exports = class FilesInDirectory {
constructor(directory) {
this.fid = fs.readdirSync(path.resolve(directory));
this.requiredFiles = (this.fid.map((fileId) => {
let resolvedPath = path.resolve(directory, fileId);
return require(resolvedPath);
})).filter(file => !!file);
}
printRetrievedFiles() {
console.log(this.requiredFiles);
}
nextFileGenerator() {
const parent = this;
const fidLength = parent.requiredFiles.length;
function* iterate(index) {
while (index < fidLength) {
yield parent.requiredFiles[index++];
}
}
return iterate(0);
}
}
Then use like so:
//Use in test
const FilesInDirectory = require('./utilities/getfilesindirectory');
const StepsCollection = new FilesInDirectory('./test-steps');
const StepsGenerator = StepsCollection.nextFileGenerator();
//Assuming we're in an async function
await StepsGenerator.next().value.FUNCTION_REQUIRED_FROM_FILE(someArg);