I do not know, if this is related to koa, or is problem of some other npm module or something else. I am going to start from here.
So to the problem. I am having REST api written in koa v1. We are running node server in the Docker image. One of the endpoints we have, starts the import and returns the status 200 with message: "import started", and when the import finishes, we send Slack message to notify us.
So first I tested the server on my local machine, everything works (endpoint does not throw any errors). Then I built docker image. I run container localy, everything works (endpoint does not throw any errors). I deploy my image to Mesos environment, everything works so far. Container runs, every endpoint works, beside import endpoint. When I call it, after few seconds (5 to 10), I get ECONNRESET error, the running container gets killed and new running instance is started. So import is terminated.
At the beginning we assigned 128 MB ram to the docker container and that seems to be enough. After import error occurred, we thought maybe OOM killed process. So we decided to check dmesg and we could not find any log entries related to the OOM and the process of the running container. Then we checked ram usage of the container locally (with htop) and found out it uses aprox. 250+ MB, so we decided to add more ram in marathon config (512 MB). That however did not help, same error occurred.
Because the error was not explicit enough we installed longjohn module, so we could get more detailed error message. That got us just a little bit more information, but not as much as we thought it would.
Error: read ECONNRESET
at exports._errnoException (util.js:1026:11)
at TCP.onread (net.js:569:26)
---------------------------------------------
at Application.app.callback (/src/node_modules/koa/lib/application.js:130:45)
at Application.app.listen (/src/node_modules/koa/lib/application.js:73:39)
at Promise.then.result (/src/server.js:97:13)
Error: read ECONNRESET
at exports._errnoException (util.js:1026:11)
at TCP.onread (net.js:569:26)
Line 97 of the server.js is:
96:if(!module.parent) {
97: app.listen(port, (err) => {
98: if (err) {
99: console.error('Server error', err);
100: }
101: console.log('Listening on the port', port);
102: });
103:}
So what exactly happens in the endpoint logic. We are using postgres npm module pg. We are passing pg.Pool to the context, so later we can use it in our models. We are executing insert query encapsulated in promise and push promises in the array. There are roughly 2700+ records. Later we do Promise.all on the array of promises and with then we send the message to Slack.
As you can see I do not know if the error is related to koa or pg or some other thing. What is more intriguing is that locally everything works (node server as well as in docker container), but on Mesos it does not. How can I find out what is wrong?
version of koa npm module: 1.2.0
version of pg npm module: 6.1.0
version of Postgres 9.5
version of Mesos: 1.0.1
According to this github issue this is an error caused by tiny-lr.
It seems that downgrading to version 0.2.1 stops it, but this is usually a dependency of other packages you're using that you've got no control over. You might be able to filter out the error by displaying all errors except this, as such:
if (error.code !== 'ECONNRESET') { console.log(error) }
The issue is still open, and dates from Oct 27, 2016. Don't know if it will get fixed or not. But as far as feedback goes, it doesn't seem like a dangerous error, or to have any impact whatsoever. But heh, I'd rather fix mine too, if there was a way.
Thanks to another developer, we found out what was the cause of the ERROR. We used all connections in the pool when there was an import running.
When the marathon was requesting the service status at the time of the import, service tried to connect to the database to test the connection and at that time the connection to the database was terminated. Service became unhealthy and marathon restarted the service. We re-factored the import code. We are limiting the number of pool connections.
Related
I recently learned about Worker threads in Node JS. I was trying to create a worker thread to run Stockfish chess engine in node js.
The npm package I am using for this is called stockfish. I tried using node-stockfish before this but it was not installing with npm as it was using an older version of the type definition for the "AbortSignal" variable apparently causing compatibility issues.
For the current npm package that I am using even though I was able to install it successfully, I could find very little documentation on how to use it. So I tried out a few ideas.
import { Worker } from "worker_threads";
const engine = new Worker("./node_modules/stockfish/src/stockfish.js")
engine.on('message', (data) => console.log(data))
engine.postMessage('position startpos move e2e4 e7e5')
engine.postMessage('go movetime 3000')
Here I tried to run the stockfish.js as a worker thread and send commands to it with the postMessage() function. This however did not work and it gave the following output:
worker.js received unknown command undefined
position startpos move e2e4 e7e5
worker.js received unknown command undefined
go movetime 3000
But I know these commands are valid commands if I run the same js from the command line like so:
It might be because I am using the flags --experimental-wasm-threads and --experimental-wasm-simd when I am running it from the command line. I found this command to run it from the little documentation that was present. But I don't know how to mention these flags when I run it through a worker thread.
Otherwise it could also be that I don't understand how worker threads work yet and postMessage() is not the same as sending it a command from the command line.
Any help is greatly appreciated.
I switched to using stockfish.wasm library instead. With this library I was able to achieve what I wanted and I don't need to use any worker threads for now. Maybe I can add this to a worker thread if required later. Here is a simple example:
const Stockfish = require("stockfish.wasm")
Stockfish().then((engine) => {
engine.addMessageListener((output) => {
console.log(output);
// Do something with the output data here
})
engine.postMessage("uci");
engine.postMessage("ucinewgame");
engine.postMessage("position startpos");
engine.postMessage("go depth 20");
});
I am using the pg package (node.js), and for some reason the connect function gives me nothing. My code gets hung up on that line and I'm unable to see any errors, what's wrong, or what's happening.
i.e.
console.log("HERE");
await pgPool.connect()
console.log("NOW HERE") //this line never prints
I've tried a bunch of variations too:
console.log("HERE");
const client = await pgPool.connect()
console.log(client) //this line never prints
Does anyone know how to get a verbose stream from pg? My pg version is 7.15.0 and my npm version is 6.14.4
I've tried waiting it out for over an hour. For friends running the same code from the same branch on their local machines it connects in under a second. I've confirmed they have the same version of pg as me.
I am able to connect directly to the database using psql in a separate terminal without issues (it immediately connects in < 1 second)
Updated my pg to 8.2.1 and it solved the problem. Must be an incompatibility issue with an earlier version
I am using A2Hosting vendor to deploy my Next js app as a server. I have written all the scripts and tested on my local machine, but when I deploy it to the webserver and run
npm run build this is what happens:
throw errnoException(err, 'spawn'); ^
Error: spawn ENOMEM
at ChildProcess.spawn (internal/child_process.js:394:11)
at spawn (child_process.js:540:9)
at Object.fork (child_process.js:108:10)
at ChildProcessWorker.initialize (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/workers/ChildProcessWorker.js:137:44)
at new ChildProcessWorker (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/workers/ChildProcessWorker.js:127:10)
at WorkerPool.createWorker (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/WorkerPool.js:44:12)
at new BaseWorkerPool (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/base/BaseWorkerPool.js:82:27)
at new WorkerPool (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/WorkerPool.js:30:1)
at new JestWorker (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/jest-worker/build/index.js:131:26)
at TaskRunner.run (/home/briddgyc/nodevenv/Br-Front/12/lib/node_modules/next/dist/build/webpack/plugins/terser-webpack-plugin/src/TaskRunner.js:3:202) {
errno: 'ENOMEM',
code: 'ENOMEM',
syscall: 'spawn'
}
I have checked that this is some kind of memory issue but 2 days and still cannot figure out anything.
I have still 40 Process and 700mb of RAM in my machine it should be sufficient I suppose.
Any recommendations ?
I have received a reply from support saying that the only way is to compile locally.
The process took more than 110% of CPU and reached 50/50 maxing the number of processes, and I'm still halfway through the project. Adding more commands that might even require more resources for compilation later on, I would rather stay safe and locally compile.
Node project worked fine couple of months ago, now after implementation of sockets, when trying to run the app, i see the following error:
Error in console(git bash)
I'm using win 10, tried changing Node versions(4.7, 6.10, 7.2, 7.9), No result there.
The port I'm using is not busy.
We are using socket.io, tried to remove the part of code, that uses the module, but unexpectedly, it did not help.
Any ideas ?
Update: Works fine on OS X, and doesn't work on 3 different win 10 oter computers.
You should listen to the error event.
socket.on('error', function () {
// Error logging here
});
I downloaded node.js (.3), redis (2.0.4), and redis-node-client (git clone). When I start the redis server in one window, then go to the node-client folder and run
node test/test.js
I get
........................................
node.js:66
throw e; // process.nextTick error, or 'error' event on first tick
^
Maximum call stack size exceeded
I'm using the default configs at the moment. Haven't changed anything. Any ideas?
Hm, turns out to be version incompatibilities. Going to v0.2.5 fixes it.
Edit: nevermind, no it doesn't. Comes up with new error:
AssertionError: "testZINTER" "ERR unknown command 'zinter'"
at /Users/vhwanger/Dropbox/Programming/nodejs/redis-node-client/test/test.js:121:25
at Client.onReply_ (/Users/vhwanger/Dropbox/Programming/nodejs/redis-node-client/lib/redis-client.js:400:34)
at /Users/vhwanger/Dropbox/Programming/nodejs/redis-node-client/lib/redis-client.js:143:30
at ReplyParser.feed (/Users/vhwanger/Dropbox/Programming/nodejs/redis-node-client/lib/redis-client.js:160:55)
at Stream.<anonymous> (/Users/vhwanger/Dropbox/Programming/nodejs/redis-node-client/lib/redis-client.js:337:28)
at Stream.emit (events:27:15)
at IOWatcher.callback (net:489:16)
at node.js:773:9
Sadly, redis-node-client is no longer maintained. That's why I've written node_redis, which you can get with:
npm install redis
There are a lot of people using it now, and this has helped us work out a lot of bugs. Let me know if you have any issues with it.