I created a test WASM program using Go. In the program's main, it adds an API to the "global" and waits on a channel to avoid from exiting. It is similar to the typical hello-world Go WASM that you can find anywhere in the internet.
My test WASM program works well in Browsers, however, I hope to run it and call the API using Node.js. If it is possible, I will create some automation tests based on it.
I tried many ways but I just couldn't get it work with Node.js. The problem is that, in Node.js, the API cannot be found in the "global". How can I run a GO WASM program (with an exported API) in Node.js?
(Let me know if you need more details)
Thanks!
More details:
--- On Go's side (pseudo code) ---
func main() {
fmt.Println("My Web Assembly")
js.Global().Set("myEcho", myEcho())
<-make(chan bool)
}
func myEcho() js.Func {
return js.FuncOf(func(this js.Value, apiArgs []js.Value) any {
for arg := range(apiArgs) {
fmt.Println(arg.String())
}
}
}
// build: GOOS=js GOARCH=wasm go build -o myecho.wasm path/to/the/package
--- On browser's side ---
<html>
<head>
<meta charset="utf-8"/>
</head>
<body>
<p><pre style="font-family:courier;" id="my-canvas"/></p>
<script src="wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(fetch("myecho.wasm"), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
// it also works without "window."
document.getElementById("my-canvas").innerHTML = window.myEcho("hello", "ahoj", "ciao");
})
})
</script>
</body>
</html>
--- On Node.js' side ---
globalThis.require = require;
globalThis.fs = require("fs");
globalThis.TextEncoder = require("util").TextEncoder;
globalThis.TextDecoder = require("util").TextDecoder;
globalThis.performance = {
now() {
const [sec, nsec] = process.hrtime();
return sec * 1000 + nsec / 1000000;
},
};
const crypto = require("crypto");
globalThis.crypto = {
getRandomValues(b) {
crypto.randomFillSync(b);
},
};
require("./wasm_exec");
const go = new Go();
go.argv = process.argv.slice(2);
go.env = Object.assign({ TMPDIR: require("os").tmpdir() }, process.env);
go.exit = process.exit;
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
}).catch((err) => {
console.error(err);
process.exit(1);
});
This pseudo code represents 99% content of my real code (only removed business related details). The problem is that I not only need to run the wasm program (myecho.wasm) by Node.js, but I also need to call the "api" (myEcho), and I need to pass it parameters and receive the returned values, because I want to create automation tests for those "api"s. With Node.js, I can launch the test js scripts and validate the outputs all in the command line environment. The browser isn't a handy tool for this case.
Running the program by node wasm_exec.js myecho.wasm isn't enough for my case.
It would be nice to know more details about your environment and what are you actually trying to do. You can post the code itself, compilation commands, and versions for all the tools involved.
Trying to answer the question without these details:
Go WASM is very browser oriented, because the go compiler needs the glue js in wasm_exec.js to run. Nodejs shouldn't have a problem with that, and the following command should work:
node wasm_exec.js main.wasm
where wasm_exec.js is the glue code shipped with your go distribution, usually found at $(go env GOROOT)/misc/wasm/wasm_exec.js, and main.wasm is your compiled code. If this fails, you can post the output as well.
There is another way to compile go code to wasm that bypasses wasm_exec.js, and that way is by using the TinyGo compiler to output wasi-enabled code. You can try following their instructions to compile your code.
For example:
tinygo build -target=wasi -o main.wasm main.go
You can build for example a javascript file wasi.js:
"use strict";
const fs = require("fs");
const { WASI } = require("wasi");
const wasi = new WASI();
const importObject = { wasi_snapshot_preview1: wasi.wasiImport };
(async () => {
const wasm = await WebAssembly.compile(
fs.readFileSync("./main.wasm")
);
const instance = await WebAssembly.instantiate(wasm, importObject);
wasi.start(instance);
})();
Recent versions of node have experimental wasi support:
node --experimental-wasi-unstable-preview1 wasi.js
These are usually the things you would try with Go and WASM, but without further details, it is hard to tell what exactly is not working.
After some struggling, I noticed that the reason is simpler than I expected.
I couldn't get the exported API function in Node.js simply because the API has not been exported yet when I tried to call them!
When the wasm program is loaded and started, it runs in parallel with the caller program (the js running in Node).
WebAssembly.instantiate(...).then(...go.run(result.instance)...).then(/*HERE!*/)
The code at "HERE" is executed too early and the main() of the wasm program hasn't finished exporting the APIs yet.
When I changed the Node script to following, it worked:
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
let retry = setInterval(function () {
if (typeof(go.exports.myEcho) != "function") {
return;
}
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
clearInterval(retry);
}, 500);
}).catch((err) => {
console.error(err);
process.exit(1);
});
(only includes the changed part)
I know it doesn't seem to be a perfect solution, but at least it proved my guess about the root cause to be true.
But... why it didn't happen in browser? sigh...
Related
I wrote this little example:
import { isBrowser, isNode } from "browser-or-node";
// runs at load
async function run() {
if (isBrowser) {
document.body.style.backgroundColor = "black";
} else if (isNode) {
const fs = await import("./myLittleScript");
console.log(fs);
}
}
run()
And ./myLittleScript is actually importing the node filesystem module to use if the user runs the script in node.
Is this idea wrong ? What is the right approach ?
I get an error that the script is not found, even though it is in the same directory than the index.ts which is the code pasted here.
This is when bundling it with webpack, but I am interested in knowing whether this is an acceptable approach or not. (I know I could just write 2 functions.)
I'm building an Electron app that needs to use Web-Assembly (WASM), however I'm hitting an issue with Fetch throwing a TypeError: Only absolute URLs are supported when importing my WASM file.
Also, perhaps this raises the broader question as to whether the Electron main process or the renderer process should be used to run the WASM ? It does seem to work in the render process.
Here's the complete error:
TypeError: Only absolute URLs are supported
at parseURL (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:897:8)
at new Request (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:922:17)
at /Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:1175:19
at new Promise (<anonymous>)
at fetch (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:1173:9)
at IpcMainImpl.<anonymous> (/Users/cbourne/development/electron-api-demos/main-process/communication/async-msg.js:20:36)
at IpcMainImpl.emit (events.js:223:5)
at WebContents.<anonymous> (electron/js2c/browser_init.js:4093:15)
at WebContents.emit (events.js:223:5)
And here's the main-process code I'm testing with:
const {ipcMain} = require('electron')
require('/Users/devuser/development/electron-api-demos/script/wasm_exec.js')
const fetch = require("node-fetch");
ipcMain.on('asynchronous-message', (event, arg) => {
if (!WebAssembly.instantiateStreaming) { // polyfill
WebAssembly.instantiateStreaming = async (resp, importObject) => {
const source = await (await resp).arrayBuffer();
return await WebAssembly.instantiate(source, importObject);
};
}
const go = new Go();
let mod, inst;
WebAssembly.instantiateStreaming(fetch("test.wasm"), go.importObject).then((result) => {
mod = result.module;
inst = result.instance;
document.getElementById("runButton").disabled = false;
}).catch((err) => {
console.error(err);
});
async function run() {
console.clear();
await go.run(inst);
inst = await WebAssembly.instantiate(mod, go.importObject); // reset instance
}
event.sender.send('asynchronous-reply', 'pong')
})
The problem is not WASM at all, but the request that is supposed to get the binary. Your fetch comes from node-fetch; the main process runs in Node.js and as such, does not have a base address like a normal page. Either provide a full file:/// absolute URL to fetch, or, more easily, use fs.readFileSync:
const fs = require('fs');
WebAssembly.instantiate(fs.readFileSync("text.wasm"));
I can only answer part of your question.
WASM should probably not be run in the Main process. Even though WASM will run in an independent thread, you should reduce the load on the Main process to the maximum extent possible. When the Main process is blocked, even something like minimizing your app will not occur until it becomes unblocked.
For more info, this is a good article: https://medium.com/actualbudget/the-horror-of-blocking-electrons-main-process-351bf11a763c
Have you tried changing fetch("test.wasm") to fetch("./test.wasm") or hardcoding a direct path to local file at least for dev purposes?
I apologise for the phrasing of the question - it's a bit difficult to sum up as a question - please feel free to edit it if you can clarify. Also, as this quite a complex and long query - thank you to all those who are putting in the time to read through it!
I have 4 files (listed with directory tree from project root) as part of a project I'm building which aims to scrape blockchains and take advantage of multiple cores do get the job done:
./main.js
./scraper.js
./api/api.js
./api/litecoin_api.js
main.js
const { scraper } = require('./scraper.js')
const blockchainCli = process.env.BLOCKSCRAPECLI || 'litecoin-cli'
const client = (args) => {
// create child process which returns a promise which resolves after
// data has finished buffering from locally hosted node using cli
let child = spawn(`${blockchainCli} ${args.join(' ')}`, {
shell: true
})
// ... wrap command in a promise here, etc
}
const main = () => {
// count cores, spawn a worker per core using node cluster, add
// message handlers, then begin scraping blockchain with each core...
scraper(blockHeight)
}
main()
module.exports = {
client,
blockchainCli
}
scraper.js
const api = require('./api/api.js')
const scraper = async (blockHeight) => {
try {
let blockHash = await api.getBlockHashByHeight(blockHeight)
let block = await api.getBlock(blockHash)
// ... etc, scraper tested and working, writes to shared writeStream
}
module.exports = {
scraper
}
api.js
const { client, blockchainCli } = require('../main.js')
const litecoin = require('./litecoin_api')
let blockchain = undefined
if (blockchainCli === 'litecoin-cli' || blockchainCli === 'bitcoin-cli') {
blockchain = litecoin
}
// PROBLEM HERE: blockchainCli (and client) are both undefined if and
// only if running scraper from main.js (but not if running scraper
// from scraper.js)
const decodeRawTransaction = (txHash) => {
return client([blockchain.decodeRawTransaction, txHash])
}
const getBlock = (blockhash) => {
return client([blockchain.getBlock, blockhash])
}
const getBlockHashByHeight = (height) => {
return client([blockchain.getBlockHash, height])
}
const getInfo = () => {
return client([blockchain.getInfo])
}
const getRawTransaction = (txHash, verbose = true) => {
return client([blockchain.getRawTransaction, txHash, verbose])
}
module.exports = {
decodeRawTransaction,
getBlock,
getBlockHashByHeight,
getInfo,
getRawTransaction
}
So, I've taken out most the noise in the files which I don't think is necessary but it's open source so if you need more take a look here.
The problem is that, if I start the scraper from inside scraper.js by doing, say, something like this: scraper(1234567) it works like a charm and outputs the expected data to a csv file.
However if I start the scraper from inside the main.js file, I get this error:
Cannot read property 'getBlockHash' of undefined
at Object.getBlockHashByHeight (/home/grayedfox/github/blockscrape/api/api.js:19:29)
at scraper (/home/grayedfox/github/blockscrape/scraper.js:53:31)
at Worker.messageHandler (/home/grayedfox/github/blockscrape/main.js:81:5)
I don't know why, when launching the scraper from main.js, the blockchain is undefined. I thought it might be from the destructuring, but removing the curly braces from around the first line in the example main.js file doesn't change anything (same error).
Things are a bit messy at the moment (in the middle of developing this branch) - but the essential problem now is that it's not clear to me why the require would fail (cannot see variables inside main.js) if it's used in the following way:
main.js (execute scraper()) > scraper.js > api.js
But not fail (can see variables inside main.js) if it's run like this:
scraper.js (execute scraper()) > api.js
Thank you very much for your time!
You have a circular dependency between main and api, each requiring in the other. main requires api through scraper and api directly requires main. That causes things not to work.
You have to remove the circular dependency by putting common shared code into its own module that can be included by both, but doesn't include others that include it. It just needs better modularity.
I've been following the pattern for setting up TypeScript, RequireJS, and Jasmine that Steve Fenton describes here:
https://www.stevefenton.co.uk/Content/Blog/Date/201407/Blog/Combining-TypeScript-Jasmine-And-AMD-With-RequireJS/
That pattern as really worked well and truly unblocked me (yay!), but I'm now at the point where I need to customize some settings for RequireJS but I can't seem to figure out where to put my require.config call. Everywhere I've tried has caused breaks and regressions. Here are the two approaches that seem most logical/promising
In SpecRunner.cshtml
<script data-main="/Scripts/TypeScript/RequireJsConfig" src="/Scripts/require.js"></script>
In RequireJsConfig.ts
require.config({
baseUrl: "../Scripts",
paths: {
jquery: "../jquery-2.1.3"
}
});
// =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
// Attempt 1: When I try it this way I immediately get this error
//
// JavaScript runtime error: Object doesn't support property or method 'config'
//
import TestLoader = require("Tests/TestLoader");
TestLoader.Run();
// =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
// Attempt 2: When I try it this way, everything builds and runs without errors, but
// Jasmine doesn't find any of the tests. All I get is "No specs found" even
// though I see the breakpoints on my "it" statements getting hit.
//
require(["Tests/TestLoader"], (testLoader) => {
testLoader.Run();
});
// =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
jasmine.getEnv().execute();
In TestLoader.ts
import GuidHelperTests = require("Tests/T3/Helpers/GuidHelperTests");
import ObjectHelperTests = require("Tests/T3/Helpers/ObjectHelperTests");
class TestLoader {
public static Run: () => void = () => {
GuidHelperTests.Run();
ObjectHelperTests.Run();
}
}
export var Run = () => TestLoader.Run();
In GuidHelperTests.ts
import T3 = require("T3/T3Lib");
export var Run = () => {
describe("GuidHelper tests", () => {
it("GUID validator validates good GUID", () => {
// etc. ...
My guess is that Attempt 2 doesn't work because of some kind of sequencing issue where the test discovery process is happening before modules are loaded, or something like that. I'm just not versed enough in RequireJS to know what my options are here.
I prefer to keep my configuration away from my application - you can pre-register the configuration like this, and it will be picked up by RequireJS when it loads. No need to add it to your first file.
<script>
var require = {
baseUrl: "../Scripts",
paths: {
jquery: "../jquery-2.1.3"
}
};
</script>
<script data-main="/Scripts/TypeScript/RequireJsConfig" src="/Scripts/require.js"></script>
I have RequireJS implemented fine, and a Grunt based build process which is optimizing the all the JS files app into one file via r.js which is also working fine. All my app files are concatenated into one big JS file for efficient production deployment.
Now I'm having the following requirements:
I need to write a plugin for requirejs, that will not load(not include the file) into the optimized file in the build process, but will required on demand:
Meaning in my code I'll have:
var myObj = require("myplugIn!jsFile");
So in the end when this line runs, it will runs in 2 options:
on build process, the file is not included in the optimized file
The application is running, it will be request the file on demand.
I wrote the following plugin, but is not working:
define(function () {
"use strict";
return {
load : function (name, req, onload, config) {
// we go inside here we are running the application not in build process
if (!config.isBuild) {
req([name], function () {
onload(arguments[0]);
});
}
}
};
});
What I'm missing here.
In your build configuration you can exclude files that you don't want to bundle. They will still be loaded on demand when needed. You may also do something like this:
define(function (){
// module code...
if (condition){
require(['mymodule'], function () {
// execute when mymodule has loaded.
});
}
}):
This way mymodule will be loaded only if condition is met. And only once, if you use same module dependency elsewhere it will return loaded module.
It was more simpler that I though, if helps someone, I'm posting the solution, I create a plugin , that in build process return nothing and in run time, returns the required file, hope helps someone.
define(function () {
"use strict";
return {
load : function (name, req, onload, config) {
if (config.isBuild) {
onload(null);
} else {
req([name], function () {
onload(arguments[0]);
});
}
}
};
});