I have a mobile app using PouchDB that listens for changes from a remote CouchDB server. Occasionally the app pushes it's own changes up. Is there a way to check to see if I still have an active "connection" to the remote CouchDB instance?
The best solution I can come up with so far is to periodically call db.info and see if I get an error back, but that seems slightly risky.
As far as I could find out, PouchDB has no event to inform us that the syncing was stopped because of a disruption in the internet connection. I did a small hack to solve this.
When the sync is happening properly, and you make a change to your local PouchDB, the order in which events are fired is:
active
change
paused
However, when the internet is broken, the order is:
active
paused
There is no change event fired when syncing with the remote DB isn't taking place.
So, what I've done is made two flags for the active and change events , pouchDbSyncActiveEvent and pouchDbSyncChangeEvent respectively, and initialized both as false. Turn pouchDbSyncActiveEvent to true in the callback for the 'active' event. This will run regardless of whether there is internet or not. Turn pouchDbSyncChangeEvent to true in the callback for the 'change' event. If there is no internet, this won't run and pouchDbSyncChangeEvent will still remain false, and the 'paused' event will be fired soon after. This informs us that the syncing was stopped.
Code:
let pouchDbSyncActiveEvent = false
let pouchDbSyncChangeEvent = false
localPouchDb.sync(remoteCouchDb, {
live: true,
retry: true
})
.on('active', (change) => {
pouchDbSyncActiveEvent = true
})
.on('change', (change) => {
pouchDbSyncChangeEvent = true
})
.on('paused', (info) => {
if(pouchDbSyncActiveEvent == true && pouchDbSyncChangeEvent == false){
// Gotcha! Syncing with remote DB not happening!
}
else {
// Everything's ok. Syncing with remote DB happening normally.
}
pouchDbSyncActiveEvent = false
pouchDbSyncChangeEvent = false
})
Note:
You can write the .on event listener methods in the order you wish, but it is good to write them in the order they are executed :)
I'm assuming that you included the pouchdb tag because you're using PouchDB locally and are replicating between your local PouchDB and remote CouchDB. Assuming so, replication in PouchDB is an event emitter, so there are a bunch of different events you can define in your sync or replicate call, see the docs including the error event.
Did you try to look for an error in the paused event emitted when connection is lost?
localDB.replicate.to(remoteDB, {
live: true,
retry: true,
}).on('paused', (err) =>{
console.log('paused');
if (err) {
alert(`No connection! ${err}`);
}
// replication was paused, usually because of a lost connection
}).on('change', (change)=>{
// yo, something changed!
}).on('active', (info)=>{
// replication was resumed
}).on('error', (err)=>{
// totally unhandled error (shouldn't happen)
});
If you are using a live replication a paused event is emitted when the replication waits for a change and err is undefined. If there is a problem like a connection interruption a paused event is also emitted but with an err argument.
Note: I'm sure that there is safer ways to check if err is defined.
From the PouchDB documentation:
paused (err) - This event fires when the replication is paused, either
because a live replication is waiting for changes, or replication has
temporarily failed, with err, and is attempting to resume.
You need to set "retry" to false, or the error event won't get caught.
From the API reference:
error (err) - This event is fired when the replication is stopped due to an unrecoverable failure. If retry is false, this will also fire when the user goes offline or another network error occurs (so you can handle retries yourself, if you want).
Try this code:
// Sync pouch - couch
db.sync(couch, {
live: true,
retry: false
}).on('error', (err) => {
console.log(`A sync error occurred: ${err}`);
});
Related
I have an application on NodeJS that uses Cluster, WS, and memcached-client to manage two memcached-servers
During normal times, it works like a charm
But during high load, my application stops working and fetches data from memcached-servers
That is, the logs inside client.get callback do not work, and are not written to the console, when the load is high, therefore the client does not receive its cached value (although it is present on the memcached server and sometimes even with high load it works fine). For a while it will look like it's dead and not doing anything
getValue = function(key, callback){
console.log(`Calculated server for choose: ${strategy(key, client.servers.length)}`) // works with highload
console.log(`Try to get from cache by key: ${key}.`); // works with highload
client.get( key, function(err, data) {
const isError = err || !data // doesn't work with highload
console.log('Data from cache is: ', data) // callback will be never executed
if (!isError) {
console.log(`Found data in cache key-value: ${key} - ${data}`);
}else{
console.log(`Not found value from cache by key: ${key}`);
}
const parsedData = isError ? null : JSON.parse(data.toString())
callback(isError, parsedData); // and this won't work also
});
}
And after some time, socket connection is simply closed (with 1000 code, no errors, looks like user just leaves out)
INFO [ProcessID-100930] Connection close [772003], type [ws], code [1000], message []
Then, after 5-10 seconds, all processes start working again as if nothing had happened and the memcached-client callback starts to execute correctly
I've been trying for so long to catch this moment and understand why this is happening, but I still don't understand. I have changed already several memcached clients(memjs now, memcached, mc) but still get the same behavior under high load
When receiving data from memcached-server, the callback simply does not work, and data from the memcached is not returned (although judging by the memcached logs, they were there at that moment)
Can someone suggest please?
In my current FOSS Discord bot project I have this log.ts file which handles the logging for the bot.
It creates multiple fs.WriteStream objects, which write to each log file. There is a section in the code when await log('CLOSE_STREAMS') is called to run the WriteStream#close() functions on each WriteStream, returning a promise. This is used in the process.on('exit') handler to save the log files before we close.
The problem here is that the 'exit' event handler can not schedule any additional work into the event queue.
How could I handle calling the CLOSE_STREAMS in a way where I can run this exit handler as I am expecting?
Function Implementation, simplified
log.ts log('CLOSE_STREAMS')
// Main
export default function log(mode: 'CLOSE_STREAMS'): Promise<void>;
export default function log(mode: 'v' | 'i' | 'w' | 'e', message: any, _bypassStackPrint?: boolean): void;
// eslint-disable-next-line #typescript-eslint/explicit-module-boundary-types
export default function log(mode: 'v' | 'i' | 'w' | 'e' | 'CLOSE_STREAMS', message?: any, _bypassStackPrint = false): void | Promise<void> {
if (mode === 'CLOSE_STREAMS')
// Close all of the file streams
return new Promise((resolve) => {
errStr.end(() => {
warnStr.end(() => {
allStr.end(() => {
resolve();
});
});
});
});
else {
index.ts
This is the way the log is killed in the uncaught exceptions; this would be how I want to do it for the exit event.
// If we get an uncaught exception, close ASAP.
process.on('uncaughtException', async (error) => {
log('e', 'Killing client...', true);
client.destroy();
log('e', 'Client killed.', true);
log('e', 'Closing databases...', true);
client.closeDatabases();
log('e', 'Closed databases.', true);
log('e', 'An uncaught exception occured!', true);
log('e', `Error thrown was:`, true);
error.stack?.split('\n').forEach((item) => {
log('e', `${item}`, true);
});
log('e', 'Stack trace dump:', true);
let stack = new Error().stack?.split('\n');
stack?.shift();
if (!stack) stack = [];
stack.forEach((item) => {
log('e', `${item}`, true);
});
log('e', 'Process exiting.', true);
log('e', 'Exit code 5.', true);
log('e', 'Goodbye!', true);
await log('CLOSE_STREAMS'); // <<<<<<<<<<<< HERE
process.exit(5);
});
As you know, you can't reliably use asynchronous operations in the processing of an exit event because the process will exit before they complete. As such, I don't think there's any way to do this reliably with a nodejs stream. Streams have a completely asynchronous API and implementation, including flushing and closing.
In a few Google searches I found a few other people asking for the same thing for the same reasons and there was no solution offered. As best I know, these are the options:
Do some hacking on the stream object to add a synchronous flushAndClose() method to your stream. You'd have to work into the internals of the stream to get the buffer and the file handle and do your own synchronous write of any remaining buffer and then do a synchronous close on the file handle. Note, even this has a problem case if there's currently an asynchronous write operation in process.
Abandon the built-in stream and just implement your own lightweight logfile interface that makes it easy for you to have both asynchronous writing (for normal use) and a synchronous flushAndClose() operation for emergency shut-down. Note, even this has a problem case if there's currently an asynchronous write operation in process when you want to do the synchronous close.
Rather than using process.on('exit', ...) to trigger closing of the log files, go up one level higher in the chain. Whatever it is that triggers closing of your app, put it in an async function that will wait for the log files to be properly closed before calling process.exit() so you get the log files closed when you still have the ability to handle asynchronous operations.
Do logging from a different (more stable) process. This process can then message that logging process what it wants logged and that process can manage getting the logging info safely to disk independent of whether the source process shuts down abruptly or not.
Note: Exiting a process will automatically close any open file selectors (the OS takes care of that for you). So, as long as this is an edge case shut-down in some fatal error condition, not a common normal shut-down, then perhaps you don't have a big problem here to really solve.
The worst that could happen is that you might lose some very recently logged lines of data if they hadn't yet been flushed from the stream. Note that streams write data immediately to their descriptor when possible so they don't generally accumulate lots of buffered data. The time when they do buffer data is when a new write to the stream happens, but the previous write is still in operation. Then the data to be written gets buffered until the prior write operation completes. So, data is never left sitting in the buffer with an idle stream. This tends to minimize (though not eliminate) data loss on an immediate shut-down.
If this is a normal, regular shut-down, then you should be able to use option #3 above and reshape how the shut-down occurs so you can use asynchronous code where you want so you can properly shutdown the streams.
I would like to loop throw all documents on a specific collection of my MongoDB. However every attempt I made failed due to the timeout of the cursor. Here is my code
let MongoClient = require('mongodb').MongoClient;
const url = "my connection URI"
let options = { socketTimeoutMS: 120000, connectTimeoutMS: 120000, keepAlive: 100, poolSize: 5 }
MongoClient.connect(url, options,
function(err, db) {
if (err) throw err
let dbo = db.db("notes")
let collection = dbo.collection("stats-network-consumption")
let stream = collection.find({}, { timeout: false }).stream()
stream.on("data", function(item) {
printTask(item)
})
stream.on('error', function (err) {
console.error(err)
})
stream.on("end", function() {
console.log("DONE!")
db.close()
})
})
The code above runs for about 15 seconds and retrieves between 6000 to 8000 documents and then throws the following error:
{ MongoError: cursor does not exist, was killed or timed out
at queryCallback (/Volumes/safezone/development/workspace-router/migration/node_modules/mongodb-core/lib/wireprotocol/2_6_support.js:136:23)
at /Volumes/safezone/development/workspace-router/migration/node_modules/mongodb-core/lib/connection/pool.js:541:18
at process._tickCallback (internal/process/next_tick.js:150:11)
name: 'MongoError',
message: 'cursor does not exist, was killed or timed out' }
I need to retrieve around 50000 documents so I will need to find a way to avoid the cursor timeout.
As seen on the code above, I've tried to increase the socketTimeoutMS and the connectTimeoutMS, which had no effect on the cursor timeout.
I also have tried to replace stream with a forEach and add .addCursorFlag('noCursorTimeout', true) which also did not help.
I've tried everything I found about mongodb, I did not tried mongoose or alternatives because they use schemas and I'll later have to update the current type of an attribute (which can be tricky with the mongoose schemas).
Having a cursor with no timeout is generally not recommended.
The reason is, the cursor won't ever be closed by the server, so if your app crashed and you restart it, it will open another no timeout cursor on the server. Recycle your app often enough, and those will add up.
No timeout cursor on a sharded cluster would also prevent chunk migration.
If you need to retrieve big results, the cursor should not timeout since the results will be sent in batches, and the cursor would be reused to get the next batch.
The standard cursor timeout is 10 minutes, so it is possible to lose the cursor if you need more than 10 minutes to process a batch.
In your code example, your use of stream() might be interfering with your intent. Try using each() (example here) on the cursor instead.
If you need to monitor a collection for changes, you might want to take a look at Change Streams which is a new feature in MongoDB 3.6.
For example, your code may be able to be modified like:
let collection = dbo.collection("stats-network-consumption")
let stream = collection.watch()
document = next(stream)
Note that to enable change stream support, the driver you're using must support MongoDB 3.6 features and the watch() method. See Driver Compatibility Page for details.
With a single server setup, I receive events from the driver.
mongoose.connect('mongodb://localhost/mydb');
mongoose.connection.on('disconnected', function() {...});
mongoose.connection.on('error', function(err) {...});
When using a replica set (mongoose.connect('mongodb://localhost:27017/mydb,mongodb://localhost:27018/mydb');), shutting down all connected set members doesn't trigger those same events.
I'm not very familiar with the internals of the native driver and I'm wondering if this is a bug or if I need to manually detect this condition.
I'm using Mongoose 3.6.17 (mongodb driver 1.3.18)
Sans mongoose, I tried this with the same results (no events from a replica set).
require('mongodb').MongoClient.connect("mongodb://localhost:27017,localhost:27018/mydb", function(err, db) {
if (db) {
db.on('disconnected', function() {
console.log('disconnected');
}).on('error', function(err) {
console.log('error');
});
}
});
I've been having similar problems with Mongoose, asked on SO also. More recently, I've found this issue on Mongoose GitHub repository which led to this issue on the driver repository.
The Mongo driver wasn't emitting any of these events more than once, and today this has been fixed for single connections on v1.3.19.
It seems that it's a "won't fix" for now.
I ended up doing the following:
I set auto_reconnect=true
Until the application has connected to the database for the first time, i disconnect and reconnect. if i don't disconnect and reconnect, any queued queries won't run. after a connection has been established at least once, those queued queries do complete and then...
for single connections:
1. forked mongoose (to use mongodb to 1.3.19) so errors get triggered more than once.
2. catch the connection error and make the app aware of the disconnection, retrying until i give up and panic or the app is reconnected. how that's done is by pinging the server every x milliseconds with a command that will not queue:
var autoReconnect = mongoose.connection.db.serverConfig.isAutoReconnect;
mongoose.connection.db.serverConfig.isAutoReconnect = function(){return false;};
mongoose.connection.db.executeDbCommand({ping:1}, {failFast: true}, function(err) {
if (!err) {
// we've reconnected.
}
});
mongoose.connection.db.serverConfig.isAutoReconnect = autoReconnect;
for a replica set, i ended up polling the mongoose connection with the above ping every x milliseconds until i detect an error, in which case i set my local app state to disconnected and enter the reconnect poll loop above (2.).
here's a gist with the relevant bits. https://gist.github.com/jsas/6299412
This is a nasty inconsistency/oversight in mongoose.
Especially when developing a microservice where you're using a single server setup for development and replica set in production.
This is the way I ended up accurately tracking the status of my mongoose connection.
let alive = false;
function updateAlive() {
return mongoose.connection
&& mongoose.connection.readyState === mongoose.STATES.connected
// This is necessary because mongoose treats a dead replica set as still "connected".
&& mongoose.connection.db.topology.connections().length > 0;
}
mongoose.connection.on('connected', () => {
updateAlive();
// I think '.topology' is available on even single server connections.
// The events just won't be emitted.
mongoose.connection.db.topology.on('joined', () => updateAlive());
mongoose.connection.db.topology.on('left', () => updateAlive());
});
mongoose.connection.on('disconnected', () => {
updateAlive();
});
I am working on a websocket oriented node.js server using Socket.IO. I noticed a bug where certain browsers aren't following the correct connect procedure to the server, and the code isn't written to gracefully handle it, and in short, it calls a method to an object that was never set up, thus killing the server due to an error.
My concern isn't with the bug in particular, but the fact that when such errors occur, the entire server goes down. Is there anything I can do on a global level in node to make it so if an error occurs it will simply log a message, perhaps kill the event, but the server process will keep on running?
I don't want other users' connections to go down due to one clever user exploiting an uncaught error in a large included codebase.
You can attach a listener to the uncaughtException event of the process object.
Code taken from the actual Node.js API reference (it's the second item under "process"):
process.on('uncaughtException', function (err) {
console.log('Caught exception: ', err);
});
setTimeout(function () {
console.log('This will still run.');
}, 500);
// Intentionally cause an exception, but don't catch it.
nonexistentFunc();
console.log('This will not run.');
All you've got to do now is to log it or do something with it, in case you know under what circumstances the bug occurs, you should file a bug over at Socket.IO's GitHub page:
https://github.com/LearnBoost/Socket.IO-node/issues
Using uncaughtException is a very bad idea.
The best alternative is to use domains in Node.js 0.8. If you're on an earlier version of Node.js rather use forever to restart your processes or even better use node cluster to spawn multiple worker processes and restart a worker on the event of an uncaughtException.
From: http://nodejs.org/api/process.html#process_event_uncaughtexception
Warning: Using 'uncaughtException' correctly
Note that 'uncaughtException' is a crude mechanism for exception handling intended to be used only as a last resort. The event should not be used as an equivalent to On Error Resume Next. Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.
Exceptions thrown from within the event handler will not be caught. Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
Attempting to resume normally after an uncaught exception can be similar to pulling out of the power cord when upgrading a computer -- nine out of ten times nothing happens - but the 10th time, the system becomes corrupted.
The correct use of 'uncaughtException' is to perform synchronous cleanup of allocated resources (e.g. file descriptors, handles, etc) before shutting down the process. It is not safe to resume normal operation after 'uncaughtException'.
To restart a crashed application in a more reliable way, whether uncaughtException is emitted or not, an external monitor should be employed in a separate process to detect application failures and recover or restart as needed.
I just did a bunch of research on this (see here, here, here, and here) and the answer to your question is that Node will not allow you to write one error handler that will catch every error scenario that could possibly occur in your system.
Some frameworks like express will allow you to catch certain types of errors (when an async method returns an error object), but there are other conditions that you cannot catch with a global error handler. This is a limitation (in my opinion) of Node and possibly inherent to async programming in general.
For example, say you have the following express handler:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err)
next(err);
else
res.send("yay");
});
});
Let's say that the file "some/file" does not actually exist. In this case fs.readFile will return an error as the first argument to the callback method. If you check for that and do next(err) when it happens, the default express error handler will take over and do whatever you make it do (e.g. return a 500 to the user). That's a graceful way to handle an error. Of course, if you forget to call next(err), it doesn't work.
So that's the error condition that a global handler can deal with, however consider another case:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err)
next(err);
else {
nullObject.someMethod(); //throws a null reference exception
res.send("yay");
}
});
});
In this case, there is a bug if your code that results in you calling a method on a null object. Here an exception will be thrown, it will not be caught by the global error handler, and your node app will terminate. All clients currently executing requests on that service will get suddenly disconnected with no explanation as to why. Ungraceful.
There is currently no global error handler functionality in Node to handle this case. You cannot put a giant try/catch around all your express handlers because by the time your asyn callback executes, those try/catch blocks are no longer in scope. That's just the nature of async code, it breaks the try/catch error handling paradigm.
AFAIK, your only recourse here is to put try/catch blocks around the synchronous parts of your code inside each one of your async callbacks, something like this:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err) {
next(err);
}
else {
try {
nullObject.someMethod(); //throws a null reference exception
res.send("yay");
}
catch(e) {
res.send(500);
}
}
});
});
That's going to make for some nasty code, especially once you start getting into nested async calls.
Some people think that what Node does in these cases (that is, die) is the proper thing to do because your system is in an inconsistent state and you have no other option. I disagree with that reasoning but I won't get into a philosophical debate about it. The point is that with Node, your options are lots of little try/catch blocks or hope that your test coverage is good enough so that this doesn't happen. You can put something like upstart or supervisor in place to restart your app when it goes down but that's simply mitigation of the problem, not a solution.
Node.js has a currently unstable feature called domains that appears to address this issue, though I don't know much about it.
I've just put together a class which listens for unhandled exceptions, and when it see's one it:
prints the stack trace to the console
logs it in it's own logfile
emails you the stack trace
restarts the server (or kills it, up to you)
It will require a little tweaking for your application as I haven't made it generic as yet, but it's only a few lines and it might be what you're looking for!
Check it out!
Note: this is over 4 years old at this point, unfinished, and there may now be a better way - I don't know!)
process.on
(
'uncaughtException',
function (err)
{
var stack = err.stack;
var timeout = 1;
// print note to logger
logger.log("SERVER CRASHED!");
// logger.printLastLogs();
logger.log(err, stack);
// save log to timestamped logfile
// var filename = "crash_" + _2.formatDate(new Date()) + ".log";
// logger.log("LOGGING ERROR TO "+filename);
// var fs = require('fs');
// fs.writeFile('logs/'+filename, log);
// email log to developer
if(helper.Config.get('email_on_error') == 'true')
{
logger.log("EMAILING ERROR");
require('./Mailer'); // this is a simple wrapper around nodemailer http://documentup.com/andris9/nodemailer/
helper.Mailer.sendMail("GAMEHUB NODE SERVER CRASHED", stack);
timeout = 10;
}
// Send signal to clients
// logger.log("EMITTING SERVER DOWN CODE");
// helper.IO.emit(SIGNALS.SERVER.DOWN, "The server has crashed unexpectedly. Restarting in 10s..");
// If we exit straight away, the write log and send email operations wont have time to run
setTimeout
(
function()
{
logger.log("KILLING PROCESS");
process.exit();
},
// timeout * 1000
timeout * 100000 // extra time. pm2 auto-restarts on crash...
);
}
);
Had a similar problem. Ivo's answer is good. But how can you catch an error in a loop and continue?
var folder='/anyFolder';
fs.readdir(folder, function(err,files){
for(var i=0; i<files.length; i++){
var stats = fs.statSync(folder+'/'+files[i]);
}
});
Here, fs.statSynch throws an error (against a hidden file in Windows that barfs I don't know why). The error can be caught by the process.on(...) trick, but the loop stops.
I tried adding a handler directly:
var stats = fs.statSync(folder+'/'+files[i]).on('error',function(err){console.log(err);});
This did not work either.
Adding a try/catch around the questionable fs.statSynch() was the best solution for me:
var stats;
try{
stats = fs.statSync(path);
}catch(err){console.log(err);}
This then led to the code fix (making a clean path var from folder and file).
I found PM2 as the best solution for handling node servers, single and multiple instances
One way of doing this would be spinning the child process and communicate with the parent process via 'message' event.
In the child process where the error occurs, catch that with 'uncaughtException' to avoid crashing the application. Mind that Exceptions thrown from within the event handler will not be caught. Once the error is caught safely, send a message like: {finish: false}.
Parent Process would listen to the message event and send the message again to the child process to re-run the function.
Child Process:
// In child.js
// function causing an exception
const errorComputation = function() {
for (let i = 0; i < 50; i ++) {
console.log('i is.......', i);
if (i === 25) {
throw new Error('i = 25');
}
}
process.send({finish: true});
}
// Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
process.on('uncaughtException', err => {
console.log('uncaught exception..',err.message);
process.send({finish: false});
});
// listen to the parent process and run the errorComputation again
process.on('message', () => {
console.log('starting process ...');
errorComputation();
})
Parent Process:
// In parent.js
const { fork } = require('child_process');
const compute = fork('child.js');
// listen onto the child process
compute.on('message', (data) => {
if (!data.finish) {
compute.send('start');
} else {
console.log('Child process finish successfully!')
}
});
// send initial message to start the child process.
compute.send('start');