I'm building an Electron app that needs to use Web-Assembly (WASM), however I'm hitting an issue with Fetch throwing a TypeError: Only absolute URLs are supported when importing my WASM file.
Also, perhaps this raises the broader question as to whether the Electron main process or the renderer process should be used to run the WASM ? It does seem to work in the render process.
Here's the complete error:
TypeError: Only absolute URLs are supported
at parseURL (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:897:8)
at new Request (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:922:17)
at /Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:1175:19
at new Promise (<anonymous>)
at fetch (/Users/devuser/development/electron-api-demos/node_modules/node-fetch/dist/index.cjs:1173:9)
at IpcMainImpl.<anonymous> (/Users/cbourne/development/electron-api-demos/main-process/communication/async-msg.js:20:36)
at IpcMainImpl.emit (events.js:223:5)
at WebContents.<anonymous> (electron/js2c/browser_init.js:4093:15)
at WebContents.emit (events.js:223:5)
And here's the main-process code I'm testing with:
const {ipcMain} = require('electron')
require('/Users/devuser/development/electron-api-demos/script/wasm_exec.js')
const fetch = require("node-fetch");
ipcMain.on('asynchronous-message', (event, arg) => {
if (!WebAssembly.instantiateStreaming) { // polyfill
WebAssembly.instantiateStreaming = async (resp, importObject) => {
const source = await (await resp).arrayBuffer();
return await WebAssembly.instantiate(source, importObject);
};
}
const go = new Go();
let mod, inst;
WebAssembly.instantiateStreaming(fetch("test.wasm"), go.importObject).then((result) => {
mod = result.module;
inst = result.instance;
document.getElementById("runButton").disabled = false;
}).catch((err) => {
console.error(err);
});
async function run() {
console.clear();
await go.run(inst);
inst = await WebAssembly.instantiate(mod, go.importObject); // reset instance
}
event.sender.send('asynchronous-reply', 'pong')
})
The problem is not WASM at all, but the request that is supposed to get the binary. Your fetch comes from node-fetch; the main process runs in Node.js and as such, does not have a base address like a normal page. Either provide a full file:/// absolute URL to fetch, or, more easily, use fs.readFileSync:
const fs = require('fs');
WebAssembly.instantiate(fs.readFileSync("text.wasm"));
I can only answer part of your question.
WASM should probably not be run in the Main process. Even though WASM will run in an independent thread, you should reduce the load on the Main process to the maximum extent possible. When the Main process is blocked, even something like minimizing your app will not occur until it becomes unblocked.
For more info, this is a good article: https://medium.com/actualbudget/the-horror-of-blocking-electrons-main-process-351bf11a763c
Have you tried changing fetch("test.wasm") to fetch("./test.wasm") or hardcoding a direct path to local file at least for dev purposes?
Related
I created a test WASM program using Go. In the program's main, it adds an API to the "global" and waits on a channel to avoid from exiting. It is similar to the typical hello-world Go WASM that you can find anywhere in the internet.
My test WASM program works well in Browsers, however, I hope to run it and call the API using Node.js. If it is possible, I will create some automation tests based on it.
I tried many ways but I just couldn't get it work with Node.js. The problem is that, in Node.js, the API cannot be found in the "global". How can I run a GO WASM program (with an exported API) in Node.js?
(Let me know if you need more details)
Thanks!
More details:
--- On Go's side (pseudo code) ---
func main() {
fmt.Println("My Web Assembly")
js.Global().Set("myEcho", myEcho())
<-make(chan bool)
}
func myEcho() js.Func {
return js.FuncOf(func(this js.Value, apiArgs []js.Value) any {
for arg := range(apiArgs) {
fmt.Println(arg.String())
}
}
}
// build: GOOS=js GOARCH=wasm go build -o myecho.wasm path/to/the/package
--- On browser's side ---
<html>
<head>
<meta charset="utf-8"/>
</head>
<body>
<p><pre style="font-family:courier;" id="my-canvas"/></p>
<script src="wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(fetch("myecho.wasm"), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
// it also works without "window."
document.getElementById("my-canvas").innerHTML = window.myEcho("hello", "ahoj", "ciao");
})
})
</script>
</body>
</html>
--- On Node.js' side ---
globalThis.require = require;
globalThis.fs = require("fs");
globalThis.TextEncoder = require("util").TextEncoder;
globalThis.TextDecoder = require("util").TextDecoder;
globalThis.performance = {
now() {
const [sec, nsec] = process.hrtime();
return sec * 1000 + nsec / 1000000;
},
};
const crypto = require("crypto");
globalThis.crypto = {
getRandomValues(b) {
crypto.randomFillSync(b);
},
};
require("./wasm_exec");
const go = new Go();
go.argv = process.argv.slice(2);
go.env = Object.assign({ TMPDIR: require("os").tmpdir() }, process.env);
go.exit = process.exit;
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
}).catch((err) => {
console.error(err);
process.exit(1);
});
This pseudo code represents 99% content of my real code (only removed business related details). The problem is that I not only need to run the wasm program (myecho.wasm) by Node.js, but I also need to call the "api" (myEcho), and I need to pass it parameters and receive the returned values, because I want to create automation tests for those "api"s. With Node.js, I can launch the test js scripts and validate the outputs all in the command line environment. The browser isn't a handy tool for this case.
Running the program by node wasm_exec.js myecho.wasm isn't enough for my case.
It would be nice to know more details about your environment and what are you actually trying to do. You can post the code itself, compilation commands, and versions for all the tools involved.
Trying to answer the question without these details:
Go WASM is very browser oriented, because the go compiler needs the glue js in wasm_exec.js to run. Nodejs shouldn't have a problem with that, and the following command should work:
node wasm_exec.js main.wasm
where wasm_exec.js is the glue code shipped with your go distribution, usually found at $(go env GOROOT)/misc/wasm/wasm_exec.js, and main.wasm is your compiled code. If this fails, you can post the output as well.
There is another way to compile go code to wasm that bypasses wasm_exec.js, and that way is by using the TinyGo compiler to output wasi-enabled code. You can try following their instructions to compile your code.
For example:
tinygo build -target=wasi -o main.wasm main.go
You can build for example a javascript file wasi.js:
"use strict";
const fs = require("fs");
const { WASI } = require("wasi");
const wasi = new WASI();
const importObject = { wasi_snapshot_preview1: wasi.wasiImport };
(async () => {
const wasm = await WebAssembly.compile(
fs.readFileSync("./main.wasm")
);
const instance = await WebAssembly.instantiate(wasm, importObject);
wasi.start(instance);
})();
Recent versions of node have experimental wasi support:
node --experimental-wasi-unstable-preview1 wasi.js
These are usually the things you would try with Go and WASM, but without further details, it is hard to tell what exactly is not working.
After some struggling, I noticed that the reason is simpler than I expected.
I couldn't get the exported API function in Node.js simply because the API has not been exported yet when I tried to call them!
When the wasm program is loaded and started, it runs in parallel with the caller program (the js running in Node).
WebAssembly.instantiate(...).then(...go.run(result.instance)...).then(/*HERE!*/)
The code at "HERE" is executed too early and the main() of the wasm program hasn't finished exporting the APIs yet.
When I changed the Node script to following, it worked:
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
let retry = setInterval(function () {
if (typeof(go.exports.myEcho) != "function") {
return;
}
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
clearInterval(retry);
}, 500);
}).catch((err) => {
console.error(err);
process.exit(1);
});
(only includes the changed part)
I know it doesn't seem to be a perfect solution, but at least it proved my guess about the root cause to be true.
But... why it didn't happen in browser? sigh...
I am trying to build a Next.js web application in which the browser is an IPFS node. When I am to make the IPFS node, I get this error:
This is the code where I make the IPFS instance:
import * as IPFSCORE from 'ipfs-core';
const IPFS = (() => {
let IPFSInstance = undefined;
const createInstance = async () => {
return await IPFSCORE.create();
}
return {
getInstance: async () => {
if (!IPFSInstance) IPFSInstance = await createInstance();
return IPFSInstance;
}
}
})();
export default IPFS;
Here is a little more background: I experimented with IPFS on Node.js first. There I got this repo.lock error when I saved my app and then it refreshed (with nodemon). So I thought that making a singleton kind of code in order to create the IPFS instance only once will do the job, and it did. But at the same time, when working locally with Node.js, the repo.lock file was saved on my machine, and when I got the error (prior to making the singleton), I just deleted the file. When I switched to Next.js, I can't find the repo.lock file, like it's not on my machine. Where is the file? Or how can I get rid of this error? Or should I take another approach in order to create the IPFS instance only once?
I need to pass a function that is written in typescript which should run in the browser. The issue that I am having is that either I need to have the module I am testing transpiled and them encoded so I can pass it to the browser in puppeteer and it will run normally. This was the approach I was using, and it works. in short I was using es-build to bundle the module. and using readFile then encoding so I can, in the browser import it and run it there.
I am thinking if there is a better way to do this with jest-puppeteer? I can't use page.exposeFunction() because that is running on node environment. and passing the encoded function will give the browser ts code which is not what I want. To understand better look at the code bellow.
//file: module_under-test.e2e.test.ts
//importing does not help us because we might need the whole module encoded
import { testFn } from './module_under-test';
import fs, { readFileSync } from 'fs';
import util from 'util';
const readFile = util.promisify(fs.readFile);
//this will encode the module in a string, so it can be imported in the browser.
async function importer(path) {
return `data:text/javascript;utf-8,${encodeURIComponent((await readFile(path, { encoding: 'utf-8' })))}`;
}
describe('Basic authentication e2e tests', () => {
beforeAll(async () => {
await page.setViewport( {
width: 1920,
height: 1080,
deviceScaleFactor: 1
} );
//we do stuff like opening the page and logging in, etc
});
it('testToRunOnBrowser', async () => {
//module should be already transpiled but this was the old approach. I would use importer from the dist folder.
//with this the test pass but we don't want to have to transpile code everytime to run it.
//since we could already do it with only esbuild and puppeteer
expect(await page.evaluate(testToRunOnBrowser,await importer('../dist/module_under-test.mjs'))).toBe(true);
})
});
export async function testToRunOnBrowser(deps) {
const {testFn} = await import(deps)
const ctx = new browserGlobalFunctionCtx();
const data = ctx.DoGLobalBrowserThings();
ctx.load(data);
const dataLoaded = await testFn()
return dataLoaded === 'what I want to assert'
}
One way I did think but I was not able to do, is servng the whole src folder since the code from this project should all be tested on the browser. With that I can use babel standalone with "#babel/plugin-transform-modules-umd" and just import ts on the browser. any ideas or pointers how to do that with jest-puppeteer?
Hello stackoverflow community,
The task
I'm running Jasmine tests programmatically using jasmine.execute(). My task is to run onPrepare function, where I do some setup work like setting up reporters etc, and I need that function to be synchronous (has to be finished before running specs)
Approach #1
The first approach I tried was to just declare an async onPrepare function, which also includes the code for specifying reporters, and then do
await onPrepare();
await jasmine.execute();
Problem
In the result I get jasmine.getEnv() is not a function. I assume because getEnv() becomes available as .execute() is ran. So I understand this won't work
Approach #2
The next thing I tried was to create a helper file with my sync code, specify it in the config and run jasmine.execute();.
So, if simplified, I have
// conf.js
(async () => {
let Jasmine = require('jasmine');
let jasmine = new Jasmine();
let variables = require("./variables.json");
let {spawn} = require("child_process");
let childProcess = spawn(variables.webdriver);
console.log(`Webdriver started, pid: ${childProcess.pid}`);
jasmine.exitOnCompletion = false;
jasmine.loadConfig({
'spec_files': ['specs/*.spec.js'],
'helpers': ['on-jasmine-prepare.js'],
'stopSpecOnExpectationFailure': false,
'random': false,
})
const result = await jasmine.execute();
console.log('Test status:', result.overallStatus);
console.log('Closing library and webdriver process');
await library.close();
await childProcess.kill();
console.log('webdriver killed:', childProcess.killed);
})()
// on-jasmine-prepare.js
(async () => {
const {SpecReporter} = require("jasmine-spec-reporter");
const library = require("./library/library");
const variables = require("./variables.json");
const errorHandler = require("./modules/on-error-handler");
jasmine.getEnv().clearReporters();
jasmine.DEFAULT_TIMEOUT_INTERVAL = 3 * 60 * 1000;
jasmine.getEnv().addReporter(new SpecReporter({}))
global.library = new library.Library(variables.IP);
console.log('library instantiated')
await library.deleteSessions();
console.log('sessions deleted')
await library.launch(library.home);
console.log('home page launched')
jasmine.getEnv().addReporter(
errorHandler(library)
)
console.log('debugger reporter added' )
})();
Problem
The problem that I noticed that the helper file is executed asynchronously with the specs and I get a spec error before the helper function finishes (basically because of race condition). Below the output example, where you can see some console.log from onPrepare was ran after spec started
Webdriver started, pid: 40745
library instantiated
Jasmine started
sessions deleted
Example
✗ App is loaded (7 secs)
- Error: Session already exist
home page launched
debugger reporter added
The question
How do I run onPrepare function synchronously, before specs? Preferably natively (using only jasmine capabilities). Otherwise maybe using third party packages. I know it's possible because #protractor had this feature, however I couldn't back-engineer it
MacOS
node v16.13.2
jasmine v4.1.0
Thank you
I apologise for the phrasing of the question - it's a bit difficult to sum up as a question - please feel free to edit it if you can clarify. Also, as this quite a complex and long query - thank you to all those who are putting in the time to read through it!
I have 4 files (listed with directory tree from project root) as part of a project I'm building which aims to scrape blockchains and take advantage of multiple cores do get the job done:
./main.js
./scraper.js
./api/api.js
./api/litecoin_api.js
main.js
const { scraper } = require('./scraper.js')
const blockchainCli = process.env.BLOCKSCRAPECLI || 'litecoin-cli'
const client = (args) => {
// create child process which returns a promise which resolves after
// data has finished buffering from locally hosted node using cli
let child = spawn(`${blockchainCli} ${args.join(' ')}`, {
shell: true
})
// ... wrap command in a promise here, etc
}
const main = () => {
// count cores, spawn a worker per core using node cluster, add
// message handlers, then begin scraping blockchain with each core...
scraper(blockHeight)
}
main()
module.exports = {
client,
blockchainCli
}
scraper.js
const api = require('./api/api.js')
const scraper = async (blockHeight) => {
try {
let blockHash = await api.getBlockHashByHeight(blockHeight)
let block = await api.getBlock(blockHash)
// ... etc, scraper tested and working, writes to shared writeStream
}
module.exports = {
scraper
}
api.js
const { client, blockchainCli } = require('../main.js')
const litecoin = require('./litecoin_api')
let blockchain = undefined
if (blockchainCli === 'litecoin-cli' || blockchainCli === 'bitcoin-cli') {
blockchain = litecoin
}
// PROBLEM HERE: blockchainCli (and client) are both undefined if and
// only if running scraper from main.js (but not if running scraper
// from scraper.js)
const decodeRawTransaction = (txHash) => {
return client([blockchain.decodeRawTransaction, txHash])
}
const getBlock = (blockhash) => {
return client([blockchain.getBlock, blockhash])
}
const getBlockHashByHeight = (height) => {
return client([blockchain.getBlockHash, height])
}
const getInfo = () => {
return client([blockchain.getInfo])
}
const getRawTransaction = (txHash, verbose = true) => {
return client([blockchain.getRawTransaction, txHash, verbose])
}
module.exports = {
decodeRawTransaction,
getBlock,
getBlockHashByHeight,
getInfo,
getRawTransaction
}
So, I've taken out most the noise in the files which I don't think is necessary but it's open source so if you need more take a look here.
The problem is that, if I start the scraper from inside scraper.js by doing, say, something like this: scraper(1234567) it works like a charm and outputs the expected data to a csv file.
However if I start the scraper from inside the main.js file, I get this error:
Cannot read property 'getBlockHash' of undefined
at Object.getBlockHashByHeight (/home/grayedfox/github/blockscrape/api/api.js:19:29)
at scraper (/home/grayedfox/github/blockscrape/scraper.js:53:31)
at Worker.messageHandler (/home/grayedfox/github/blockscrape/main.js:81:5)
I don't know why, when launching the scraper from main.js, the blockchain is undefined. I thought it might be from the destructuring, but removing the curly braces from around the first line in the example main.js file doesn't change anything (same error).
Things are a bit messy at the moment (in the middle of developing this branch) - but the essential problem now is that it's not clear to me why the require would fail (cannot see variables inside main.js) if it's used in the following way:
main.js (execute scraper()) > scraper.js > api.js
But not fail (can see variables inside main.js) if it's run like this:
scraper.js (execute scraper()) > api.js
Thank you very much for your time!
You have a circular dependency between main and api, each requiring in the other. main requires api through scraper and api directly requires main. That causes things not to work.
You have to remove the circular dependency by putting common shared code into its own module that can be included by both, but doesn't include others that include it. It just needs better modularity.