I've developed a system with CouchDB 2.2.0 as the master database, PouchDB 7.0.0 in VueJS clients and a database monitor server using PouchDB under NodeJS 8.11.1.
I can change data in CouchDB using Fauxton and the browser and mobile (PWA) clients update quickly even if left running for days. This is NOT true of the server running PouchDB in NodeJS. It will faithfully respond to the same changes unless there are no changes for 20 minutes or more, after that it simply silently ignores any and all events in CouchDB
I am setting about preparing a skeletal implementation with NodeJS and Pouch and as few other dependencies as possible and will update this question if I discover something; in the meantime I would like to ask...
Is there some well known reason why this might be happening?
How can I track down the cause without starting from scratch and gradually rebuilding the complete app brick by brick until it fails?
Update 18-10-03
I seem to have solved the problem by using an fs writeStream instead of console.log, without really understanding why that should make a difference.
My complete test app looks like this :
const fs = require('fs');
const PouchDB = require('pouchdb');
const adptrMemory = require('pouchdb-adapter-memory');
var stream = fs.createWriteStream("/tmp/pouchLog", {flags:'a'});
const LG = (msg) => (stream.write(`${msg}
`));
const movesDB = process.env.LOCAL_DB;
LG(`Local :: ${movesDB}`);
LG(`Remote :: ${process.env.REMOTE_DB}`);
PouchDB.plugin(adptrMemory);
const movesDatabaseLocal = new PouchDB(movesDB);
const movesDatabaseRemote = new PouchDB(process.env.REMOTE_DB);
const repFromFilter = 'post_processing/by_new_inventory';
movesDatabaseLocal.replicate.from(movesDatabaseRemote, {
live: true,
retry: true,
filter: repFromFilter,
})
.on('change', (response) => {
LG(`${movesDB} *** NEW EXCHANGE REQUEST DELTA *** `);
LG(`Database replication from: ${response.docs.length} records.`);
})
.on('active', () => {
LG(`${movesDB} *** NEW EXCHANGE REQUEST REPLICATION RESUMED ***`);
})
.on('paused', () => {
LG(`${movesDB} *** NEW EXCHANGE REQUEST REPLICATION ON HOLD ***`);
})
.on('denied', (info) => {
LG(`${movesDB} *** NEW EXCHANGE REQUEST REPLICATION DENIED *** ${info}`);
})
.on('error', err => LG(`Database error ${err}`));
Note that I still have not built back all the original functionality. I can say that the failure after an idle period does occur if the above code uses console.log, but goes away after switching to streamed logging.
Related
I googled a lot but still have no clear solution to my issue.
Connecting to MongoDB, usually you establish a connection and after the job is done you close it.
Since next.js (and probably node.js) is single threaded. Sometimes it happens that there are two requests processed async while one request established the connection to the database, the otherone is closing the exact same connection. So the first request runs into an Topology closed exception. I have the feeling that the mongodb driver client is shared.
Is there something I did not understood correct in this?
try {
await client.connect()
const database = client.db("test")
const collection = database.collection("test")
const newDataset = await collection.insertOne({})
return newDataset.insertedId.toString()
} finally {
await client.close();
}
As in the comments stated, ive seen a lot of examples & questions here on stackoverflow where in each received request (example below) a database connection is established. This has no benefits and is "bad" because it just takes time and makes no sense. E.g:
app.get("/", (req, res) => {
MongoClient.connect("...", (err, client) => {
// do what ever you want here
client.close();
});
});
If you application needs a database connection, establish the connection "in the startup phase" and keep the connection open. There is no reason to open and close the database connection for each request.
const mongodb = require("monogdb");
const express = require("express");
const app = express();
// some custom init stuff
// e.g. require your route handler etc.
mongodb.MongoClient("...", (err, client) => {
// do what ever you want with the db connection now
// e.g. monkey patch it, so you can use it in other files
// (There are better ways to handle that)
mongodb.client = client;
// or the better way
// pass it as function parameter
require("./routes")(app, client);
app.listen(8080, () => {
console.log("http server listening");
});
});
As you can see in the code above, we first create a database connection and then do other stuff. This has some advantages:
If your credentials are invalid, your application is not externeal reachable because the http server is not started
You have a single connection for all requests
Database queries are potential faster because you dont have to wait to establish first a db connection
NOTE: the code above was "inline coded" here and is not tested.
But i think its illustrated the concept behind my statement.
I have just a question I want to ask if anybody have an idea about it.
I'm building a full stack application backed by nodejs and using typescript for it, in my nodejs app I'm making a fetch for an API that later on I will serve it to the user but I have one small issue, I'm using node-fetch for now but the data which are fetched are changing all the time eg. now I have 10 entries, after 5 seconds I have 30 entries, so is there a way or mechanism to make my fetching to the data with nodejs up to date by fetching them in the background?
Thanks in advance!
Easiest solution to implement and good in actual sense for making your web app realtime https://pusher.com/
This is how you can handle pusher within your NodeJS App
import Pusher from 'pusher'
//Below are the keys that you will get from pusher when you go to getting started
// within your Dashboard
const pusher = new Pusher({
appId: "<Your app id provided by pusher>",
key: "<Key id provided by pusher>",
secret: "<Secret key given by pusher>",
cluster: "<cluster given by pusher",
useTLS: true
});
Now you want to setup a changeStream for your Collection in MongoDB
const db = mongoose.collection;
db.once('open', ()=>{
const postCollection = db.collection('posts')//This will be dependent on the collection you want to watch
const changeStream = postCollection.watch()//Make sure the collection name above are acurate
changeStream.on('change', (change)=>{
const post = change.fullDocument;//Change bring back content that change in DB Collection
if (change.operationType === 'insert'){
pusher.triger('<write channel for your pusher>', '<event in this case inser>', {
newPost:post
})
}
})
})
By that setup your pusher and backend is working now is time to setup frontend
If your usin VanillaJS the Pusher getting started has code for you
If your using ReactJS here's is the code below
import Pusher from 'pusher-js'
useEffect(()=>{
Pusher.logToConsole = true;
var pusher = new Pusher('<Key received from pusher>', {
cluster: '<cluster received from pusher>'
});
var channel = pusher.subscribe('<channel name that you wrote in server');
channel.bind('<event that you wrote in sever',(data)=> {
alert(JSON.stringify(data)); // This will be the data entries coming as soon as they enter DB then you can update your state by using spread operators to maintain what you have and also add new contents
});
//Very important to have a clean-up function to render this once
return ()=>{
pusher.unbind();
pusher.unsubscribe_all();
}
})
Now like this you have everything being realtime
I am having major performance problems with MongoDB. Simple find() queries are sometimes taking 2,000-3,000 ms to complete in a database with less than 100 documents.
I am seeing this both with a MongoDB Atlas M10 instance and with a cluster that I setup on Digital Ocean on VMs with 4GB of RAM. When I restart my Node.js app on Heroku, the queries perform well (less than 100 ms) for 10-15 minutes, but then they slow down.
Am I connecting to MongoDB incorrectly or querying incorrectly from Node.js? Please see my application code below. Or is this a lack of hardware resources in a shared VM environment?
Any help will be greatly appreciated. I've done all the troubleshooting I know how with Explain query and the Mongo shell.
var Koa = require('koa'); //v2.4.1
var Router = require('koa-router'); //v7.3.0
var MongoClient = require('mongodb').MongoClient; //v3.1.3
var app = new Koa();
var router = new Router();
app.use(router.routes());
//Connect to MongoDB
async function connect() {
try {
var client = await MongoClient.connect(process.env.MONGODB_URI, {
readConcern: { level: 'local' }
});
var db = client.db(process.env.MONGODB_DATABASE);
return db;
}
catch (error) {
console.log(error);
}
}
//Add MongoDB to Koa's ctx object
connect().then(db => {
app.context.db = db;
});
//Get company's collection in MongoDB
router.get('/documents/:collection', async (ctx) => {
try {
var query = { company_id: ctx.state.session.company_id };
var res = await ctx.db.collection(ctx.params.collection).find(query).toArray();
ctx.body = { ok: true, docs: res };
}
catch (error) {
ctx.status = 500;
ctx.body = { ok: false };
}
});
app.listen(process.env.PORT || 3000);
UPDATE
I am using MongoDB Change Streams and standard Server Sent Events to provide real-time updates to the application UI. I turned these off and now MongoDB appears to be performing well again.
Are MongoDB Change Streams known to impact read/write performance?
Change Streams indeed affect the performance of your server. As noted in this SO question.
As mentioned in the accepted answer there,
The default connection pool size in the Node.js client for MongoDB is 5. Since each change stream cursor opens a new connection, the connection pool needs to be at least as large as the number of cursors.
const mongoConnection = await MongoClient.connect(URL, {poolSize: 100});
(Thanks to MongoDB Inc. for investigating this issue.)
You need to increase your pool size to get back your normal performance.
I'd suggest you do more log works. Slow queries after restarted for a while might be worse than you might think.
For a modern database/web app running on a normal machine, it's not very easy to encounter with performance issues if you are doing right. There might be a memory leak or other unreleased resources, or network congestion.
IMHO, you might want to determine whether it's a network problem first, and by enabling slow query log on MongoDB and logging in your code where the query begins and ends, you could achieve this.
If the network is totally fine and you see no MongoDB slow queries, that means something goes wrong in your own application. Detailed logging might really help where query goes slow.
Hope this would help.
There are a lot of examples of graceful stop for expressjs, how can I achieve the same for koajs?
I would like to disconnect database connections as well
I have a mongoose database connection, and 2 oracle db connection (https://github.com/oracle/node-oracledb)
I created an npm package http-graceful-shutdown (https://github.com/sebhildebrandt/http-graceful-shutdown) some time ago. This works perfectly with http, express and koa. As you want to add also your own cleanup stuff, I modified the package, so that you now can add your own cleanup function, that will be called on shutdown. So basically this package handles all http shutdown things plus calling your cleanup function (if provided in the options):
const koa = require('koa');
const gracefulShutdown = require('http-graceful-shutdown');
const app = new koa();
...
server = app.listen(...); // app can be an express OR koa app
...
// your personal cleanup function - this one takes one second to complete
function cleanup() {
return new Promise((resolve) => {
console.log('... in cleanup')
setTimeout(function() {
console.log('... cleanup finished');
resolve();
}, 1000)
});
}
// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
{
signals: 'SIGINT SIGTERM',
timeout: 30000,
development: false,
onShutdown: cleanup,
finally: function() {
console.log('Server gracefulls shutted down.....')
}
}
);
I have answered a variation of "how to terminate a HTTP server" many times on different node.js support channels. Unfortunately, I couldn't recommend any of the existing libraries because they are lacking in one or another way. I have since put together a package that (I believe) is handling all the cases expected of graceful HTTP server termination.
https://github.com/gajus/http-terminator
The main benefit of http-terminator is that:
it does not monkey-patch Node.js API
it immediately destroys all sockets without an attached HTTP request
it allows graceful timeout to sockets with ongoing HTTP requests
it properly handles HTTPS connections
it informs connections using keep-alive that server is shutting down by setting a connection: close header
it does not terminate the Node.js process
Usage with Koa:
import Koa from 'koa';
import {
createHttpTerminator,
} from 'http-terminator';
const app = new Koa();
const server = app.listen();
const httpTerminator = createHttpTerminator({
server,
});
await httpTerminator.terminate();
To make sure the Oracle DB connections are closed nicely, you can use a connection pool and call pool.close() with a drainTime of 0 or greater. This will let the app relatively cleanly interrupt any operation that is currently using a connection. It allows freeing the DB end of the connections without the DB waiting for whatever timeout period to expire before it cleans itself up. Even with two connections this is a solution I'd look at, since it doesn't matter that the pool is small. You may need to set the Oracle Net out-of-band break detection as well, see Connections and High Availability.
Modern versions of node have support for AbortController, so no need for external libraries. A Simple example:
const app = new Koa();
const server = http.createServer(app.callback());
const controller = new AbortController();
server.listen({
host: 'localhost',
port: 80,
signal: controller.signal
});
// middleware... etc.
app.use(async (ctx) => {
ctx.body = 'Hello World';
});
// Later, when you want to close the server.
controller.abort();
I am building a SOA with deepstream and I want to use a deepstream client server to perform API-KEY based look ups that the user should not know. How do I actually set up an RPC client provider? I have looked in the deepstream docs and on google, but there is not a full code example on how to do this. I have created a file like below and run it with node. The output I get is below it:
var deepstream = require('deepstream.io-client-js')
const client = deepstream('localhost:6020').login()
console.log('Starting up')
client.on('error', (error,event,topic) => {
console.log(error, event, topic);
})
client.on('connectionStateChanged', connectionState => {
console.log(connectionState);
})
client.login({username: 'USER', password: 'PASSWORD'}, (success, data) => {
if (success) {
client.rpc.provide('the-rpc', function( data, response ){
response.send(data);
});
} else {
console.log(data);
}
})
--
Starting up
AWAITING_CONNECTION
As you can see it runs the code, but does not actually connect to the deepstream server. I already have the deepstream server running, and a browser client that connects to it, so the config is correct. Please help!
I think your issue is based on the fact your trying to connect node via the webport. Try using port 6021 instead for tcp ( used by the node client ).
const client = deepstream('localhost:6021').login()
You should also only call .login() once, so the line would be:
const client = deepstream('localhost:6021')
We are working on a 2.0 release coming out very soon which will remove tcp entirely and only require a single port to make life easier in terms of deployment and performance.