Worker thread postMessage() vs command line command - node.js

I recently learned about Worker threads in Node JS. I was trying to create a worker thread to run Stockfish chess engine in node js.
The npm package I am using for this is called stockfish. I tried using node-stockfish before this but it was not installing with npm as it was using an older version of the type definition for the "AbortSignal" variable apparently causing compatibility issues.
For the current npm package that I am using even though I was able to install it successfully, I could find very little documentation on how to use it. So I tried out a few ideas.
import { Worker } from "worker_threads";
const engine = new Worker("./node_modules/stockfish/src/stockfish.js")
engine.on('message', (data) => console.log(data))
engine.postMessage('position startpos move e2e4 e7e5')
engine.postMessage('go movetime 3000')
Here I tried to run the stockfish.js as a worker thread and send commands to it with the postMessage() function. This however did not work and it gave the following output:
worker.js received unknown command undefined
position startpos move e2e4 e7e5
worker.js received unknown command undefined
go movetime 3000
But I know these commands are valid commands if I run the same js from the command line like so:
It might be because I am using the flags --experimental-wasm-threads and --experimental-wasm-simd when I am running it from the command line. I found this command to run it from the little documentation that was present. But I don't know how to mention these flags when I run it through a worker thread.
Otherwise it could also be that I don't understand how worker threads work yet and postMessage() is not the same as sending it a command from the command line.
Any help is greatly appreciated.

I switched to using stockfish.wasm library instead. With this library I was able to achieve what I wanted and I don't need to use any worker threads for now. Maybe I can add this to a worker thread if required later. Here is a simple example:
const Stockfish = require("stockfish.wasm")
Stockfish().then((engine) => {
engine.addMessageListener((output) => {
console.log(output);
// Do something with the output data here
})
engine.postMessage("uci");
engine.postMessage("ucinewgame");
engine.postMessage("position startpos");
engine.postMessage("go depth 20");
});

Related

nodejs child_process spawn silently fails on windows

I'm wondering if anyone has had a similar problem with nodejs child_process spawn on windows.
I can no longer execute a nodejs script that calls child_proces.spawn. (This was working fine until yesterday, when suddenly node stopped working properly. No new software installed or anything that I am aware of that could have triggered this.) The call to spawn simply exits the process and fails with no error code, no error message, just exits silently.
My testcase script looks like this:
const { spawn } = require('child_process');
console.log('test 1');
try {
spawn('c:\\windows\\system32\\cmd.exe', ['/d', '/c', 'dir'], { stdio: 'inherit' });
console.log('test 2');
} catch (err) {
console.log('err', err);
}
The output looks like this:
PS C:\test> node .\index.js
test 1
PS C:\test>
Running nodejs v14.17.1 on Windows 10. (I've also tried with nodejs v10, v12, v14.15.1 and v16, both x86 and x64 varieties - there is no difference in behavior.) I've tried uninstalling nodejs and re-installing multiple times. I've tried with Powershell and with the windows Command Prompt.
I had this problem intermittently about 6 months ago and previously just rebooting windows seemed to clear this up. I'm guessing that something has been corrupted in the OS somehow.
Tracing through nodejs through the nodejs debug inspect shows it hits this line and then just quits silently when stepping into this line of code:
https://github.com/nodejs/node/blob/e46c680bf2b211bbd52cf959ca17ee98c7f657f5/lib/internal/child_process.js#L390
const err = this._handle.spawn(options);
I don't see anything odd/wrong in the options object and I've tried with various different environment variable settings for PATH, etc.
UPDATE:
Looks like things work fine if windows is started in safe mode or started with early launch anti-malware protection disabled. Now, I'm suspecting something with anti-virus software is interfering here.
Turns out some new anti-virus rules were blocking all processes attempting to spawn a new cmd.exe child process.

child_process.fork not starting an express server inside of packaged electron app

I have an electron app where I need not only to run the interface to the user but also start an express server that will serve files for people connected through the network.
I have everything working if I start both electron and the express server normally, but I'm pretty confident that I will need the server running in a different thread to avoid slugish interface and even problems with the server.
For that matter I tried to run my express server using the child_process.fork and it worked when I use npm start, but when I use electron-builder to create an .exe, the installed program doesn't start the express server.
I tried to run my server right away using:
require('child_process').fork('app/server/mainServer.js')
I tried several changes, prefixing the file with __dirname, process.resourcesPath and even hard coding the generated file path; changing the fork options to pass cwd: __dirname, detached: true and stdio: 'ignore'; and even tried using spawn with process.execPath, which will also work with npm start but won't when packaged (it keeps opening new instances of my app, seems obvious after you do hehe)
Note: If I don't fork and require the server script right away, using require('server/mainServer.js') it works on the packaged app, so the problem most like isn't the express itself.
Note 2: I have asar: false to solve other problems, so this is not the problem solver here.
I put up a small git project to show my problem:
https://github.com/victorivens05/electron-fork-error
Any help will be highly appreciated.
With the great help from Samuel Attard (https://github.com/MarshallOfSound) I was able to solve the problem (he solved for me actually)
As he said:
the default electron app will launch the first file path provided to it
so `electron path/to/thing` will work
in a packaged state, that launch logic is not present
it will always run the app you have packaged regardless of the CLI args passed to it
you need to handle the argument manually yourself
and launch that JS file if it's passed in as the 1st argument
The first argument to fork simply calls `process.execPath` with the first
argument being the path provided afaik
The issue is that when packaged Electron apps don't automatically run the
path provided to them
they run the app that is packaged within them
In other words. fork is actually spawn being executed with process.execPath and passing the fork's first argument as the second for spawn.
What happens in a packaged app is that the process.execPath isn't electron but the packaged app itself. So if you try to spawn, the app will be open over and over again.
So, what Samuel suggest was implemented like this:
if (process.argv[1] === '--start-server') {
require('./server/mainServer.js')
return
}
require('./local/mainLocal.js')
require('child_process').spawn(process.execPath, ['--start-server'])
That way, the first time the packaged app will be executed, the process.argv[1] will be empty, so the server won't start. It will then execute the electron part (mainLocal in my case) and start the app over, but this time passing the argv. Next time the app starts, it will start the server and stop the execution, so the app won't open again because spawn is never reached.
Huge thanks to Samuel.

Can't spawn `gcloud app deploy` from a Node.js script on Windows

I'm building an Electron application (Node.js) which needs to spawn gcloud app deploy from the application with realtime feedback (stdin/stdout/stderr).
I rapidly switched from child_process to execa because I had some issues on Mac OS X with the child_process buffer which is limited to 200kb (and gcloud app deploy sends some big chunk of string > 200kb which crash the command).
Now, with execa everything seems to work normally on OSX but not on Windows.
The code looks something like this:
let bin = `gcloud${/^win/.test(process.platform) ? '.cmd' : ''}`
//which: https://github.com/npm/node-which
which(bin, (err, fullpath) => {
let proc = execa(fullpath, ['app', 'deploy'], {
cwd: appPath
})
proc.stdout.on('data', data => {
parseDeploy(data.toString())
})
proc.stderr.on('data', data => {
parseDeploy(data.toString())
})
proc.then(() => {
...
}).catch(e => {
...
})
})
This code works perfectly on Mac OS X while I haven't the same result on Windows
I have tried lots of thing:
execa()
execa.shell()
options shell:true
I tried maxBuffer to 1GB (just in case)
It works with detached:true BUT I can't read stdout / stderr in realtime in the application as it prompts a new cmd.exe without interaction with the Node.js application
Lots of child_process variant.
I have made a GIST to show the responses I get for some tests I have done on Windows with basic Child Process scripts:
https://gist.github.com/thyb/9b53b65c25cd964bbe962d8a9754e31f
I also opened an issue on execa repository: https://github.com/sindresorhus/execa/issues/97
Does someone already got this issue ? I've searched around and found nothing promising except this reddit thread which doesn't solve this issue.
Behind the scene, gcloud.cmd is running a python script. After reading tons of Node.js issue with ChildProcess / Python and Windows, I fell on this thread: https://github.com/nodejs/node-v0.x-archive/issues/8298
There is some known issue about running Python scripts from a Node.js Child Process.
They talk in this comment about an unbuffered option for python. After updating the shell script in gcloud.cmd by adding the -u option, I noticed everything was working as expected
This comment explains how to set this option as an environment variable (to not modify the windows shell script directly): https://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED
So adding PYTHONUNBUFFERED to the environment variable fix this issue !
execa(fullpath, ['app', 'deploy'], {
cwd: appPath,
env: Object.assign({}, process.env, {
PYTHONUNBUFFERED: true
})
})

Package.on_test runs even when not testing

This is what our app looks like when starting:
=> Started proxy.
=> Started MongoDB.
>>>>> IN ON_TEST
meteor-collection-management: updating npm dependencies -- mongodb...
Argh. In meteor-collection-management/package.js (our own package), there is this:
Package.on_test(function(api) {
console.log(">>>>> IN ON_TEST");
Npm.depends({
mongodb: "1.4.1"
});
api.use(['meteor-collection-management', 'tinytest', 'test-helpers']);
api.add_files('tests/dbobject-test.js', ['client', 'server']);
api.add_files('tests/enums-test.js', ['client', 'server']);
});
Why is Package.on_test running? I am not running in test mode not even in node debug mode.
The on test function just runs to build a dependency map, even though its not actually used. I see you've opened an issue on it too. Theres more info on what it does here: https://github.com/meteor/meteor/blob/a40a6273953c0e18eddcd67919754814461c5dd4/tools/packages.js#L1434
So it builds out .test, and needs to run the method to get the required files. Meteor needs to know what it needs before the project can run, which is probably why both run. (Package's need to be built into single files, as slightly different to the rest of Meteor)

Node.js – events js 72 throw er unhandled 'error' event

I'm new to Node.js and wish to run a program using streams. With other programs, I had to start a server simultaneously (mongodb, redis, etc) but I have no idea if I'm supposed to run one with this. Please let me know where I am going wrong and how I can rectify this.
This is the program:
var http = require('http'),
feed = 'http://isaacs.iriscouch.com/registry/_changes?feed=continuous';
function decide(cb) {
setTimeout(function () {
if (Date.now()%2) { return console.log('rejected'); }
cb();
}, 2000);
}
http.get(feed, function (res) {
decide(res.pipe.bind(res, process.stdout));
//using anonymous function instead of bind:
// decide(function () {
// res.pipe(process.stdout)
// });
});
This is the cmd output:
<b>C:\05-Employing Streams\05-Employing Streams\23-Playing with pipes>node npm_stre
am_piper.js
events.js:72
throw er; // Unhandled 'error' event
^
Error: Parse Error
at Socket.socketOnData (http.js:1583:20)
at TCP.onread (net.js:527:27)
</b>
Close nodejs app running in another shell.
Restart the terminal and run the program again.
Another server might be also using the same port that you have used for nodejs. Kill the process that is using nodejs port and run the app.
To find the PID of the application that is using port:8000
$ fuser 8000/tcp
8000/tcp: 16708
Here PID is 16708 Now kill the process using the kill [PID] command
$ kill 16708
I had the same problem. I closed terminal and restarted node. This worked for me.
Well, your script throws an error and you just need to catch it (and/or prevent it from happening). I had the same error, for me it was an already used port (EADDRINUSE).
I always do the following whenever I get such error:
// remove node_modules/
rm -rf node_modules/
// install node_modules/ again
npm install // or, yarn
and then start the project
npm start //or, yarn start
It works fine after re-installing node_modules. But I don't know if it's good practice.
Check your terminal it happen only when you have your application running on another terminal..
The port is already listening..
For what is worth, I got this error doing a clean install of nodejs and npm packages of my current linux-distribution
I've installed meteor using
npm install metor
And got the above referenced error. After wasting some time, I found out I should have used meteor's way to update itself:
meteor update
This command output, among others, the message that meteor was severely outdated (over 2 years) and that it was going to install itself using:
curl https://install.meteor.com/ | sh
Which was probably the command I should have run in the first place.
So the solution might be to upgrade/update whatever nodejs package(js) you're using.

Resources