PM2/NodeJS API with mongoDB index problems [duplicate] - node.js

This question already has an answer here:
mongoose index already exists with different options
(1 answer)
Closed 4 years ago.
I'm trying to deploy multiple NodeJS rest APIs in the same Ubuntu VPS using PM2, those APIs are used to serve data from different MongoDB database in the same local hosted db server.
I do encounter some port problems, but i do change it in every application to solve the problem. After that a strange mongoDB error appears.
API listen on port 3001
Connected!
MongoError: Index with pattern: { _fts: "text", _ftsx: 1 } already exists with different options
at /var/app/my-project/node_modules/mongodb-core/lib/connection/pool.js:593:63
at authenticateStragglers (/var/app/my-project/node_modules/mongodb-core/lib/connection/pool.js:516:16)
at Connection.messageHandler (/var/app/my-project/node_modules/mongodb-core/lib/connection/pool.js:552:5)
at emitMessageHandler (/var/app/my-project/node_modules/mongodb-core/lib/connection/connection.js:309:10)
at Socket.<anonymous> (/var/app/my-project/node_modules/mongodb-core/lib/connection/connection.js:452:17)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:597:20)
Now, the API keeps restarting and slows the all system. Please can anyone guide me threw this, note that i'm just maintaining the app. i'm not the developer and i'm not so familiar with mongo databases.
Thanks

check on which field you have your text index defined. Right now mongodb allows only one text index per collection
db.collection.getIndexes()
You will get some thing like this
{
"v" : 1,
"key" : {
"_fts" : "text",
"_ftsx" : 1
},
"name" : "desc_text",
"ns" : "some.ns",
"weights" : {
"title" : 1
},
"default_language" : "english",
"language_override" : "language",
"textIndexVersion" : 2
}
now if you want to scope in other columns also to use this index simply drop this index
db.collection.dropIndex('desc_text');
and then recreate it by including all columns you want to be covered by text index,
db.collection.createIndex({
title:'text'
...
});

Related

Problems implementing reconnection strategy for mongodb using mongoose on a replica set

I have an experimental app that uses a NodeJS backend which connects to a mongodb instance using mongoose. Mongodb is configured to run as single-node replica set to give me access to change streams which I use for real-time updates over a websocket connection.
For several reasons I'd like to be able to start the backend without the DB service running and have it connect as soon as it becomes available (primary reason: Docker startup). Also, I would like it to reconnect if the db server becomes unavailable for some reason.
A promising fix for reconnecting to mongodb seems to be https://stackoverflow.com/a/41859984/8447743. However, it does not seem to work in my case. It does work for delayed connection (mongod service starting after app), but gives an error on reconnection:
C:\Users\<Path to app>\node_modules\mongoose\node_modules\mongodb\src\cmap\connection.ts:740
callback(new MongoServerError(document));
^
MongoServerError: Cannot run getMore on cursor 8780071429437342673, which was created in session 993f2cbd-db02-4565-af56-f0c9a88ba012 - 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=, without an lsid
at MessageStream.messageHandler (C:\Users\<Path to app>\node_modules\mongoose\node_modules\mongodb\src\cmap\connection.ts:740:20)
at MessageStream.emit (node:events:526:28)
at MessageStream.emit (node:domain:475:12)
at processIncomingData (C:\Users\<Path to app>\node_modules\mongoose\node_modules\mongodb\src\cmap\message_stream.ts:167:12)
at MessageStream._write (C:\Users\<Path to app>\node_modules\mongoose\node_modules\mongodb\src\cmap\message_stream.ts:64:5)
at writeOrBuffer (node:internal/streams/writable:389:12)
at _write (node:internal/streams/writable:330:10)
at MessageStream.Writable.write (node:internal/streams/writable:334:10)
at Socket.ondata (node:internal/streams/readable:754:22)
at Socket.emit (node:events:526:28) {
ok: 0,
code: 50737,
codeName: 'Location50737',
'$clusterTime': {
clusterTime: Timestamp { low: 2, high: 1650366308, unsigned: true },
signature: { hash: [Binary], keyId: 0 }
},
operationTime: Timestamp { low: 2, high: 1650366308, unsigned: true }
}
[nodemon] app crashed - waiting for file changes before starting...
According to https://www.mongodb.com/docs/manual/reference/command/lockInfo/ I might able to get this lsid that seems to be missing, but I haven't found a way to make that known to mongoose for reconnecting. Is that possible somehow? Is there another way to avoid the error?

CentOS + Node.js (v8.15.0) + Sequelize (v4.38.0) to Azure SQL Intermittent Connection Issues

On our production Azure Hosted CentOS API server we are having intermittent Sequelize connection issues to our Azure SQL database.
Our connection settings are as follows:
const sequelize = new Sequelize(dbDatabase, dbUser, dbPassword, {
host: dbHost,
dialect: 'mssql',
operatorsAliases: false,
pool: {
max: 5092,
min: 0,
acquire: 10000,
idle: 10000,
evict: 500,
},
dialectOptions: {
encrypt: false, // Use this if you're on Windows Azure
requestTimeout: 60000 * 2,
},
}
We have also set the ulimit for open files by the process to the system max.
We are using PM2 to run the server.
The two errors that sometimes appear in our logs roughly every 10-20 minutes are:
SequelizeConnectionError
at Connection.connection.on (/home/AZ-admin/XXX/node_modules/sequelize/lib/dialects/mssql/connection-manager.js:75:16)
at emitNone (events.js:106:13)
at Connection.emit (events.js:208:7)
at Connection.cleanupConnection (/home/AZ-admin/XXX/node_modules/tedious/lib/connection.js:568:16)
at Connection.enter (/home/AZ-admin/XXX/node_modules/tedious/lib/connection.js:1961:12)
at Connection.transitionTo (/home/AZ-admin/XXX/node_modules/tedious/lib/connection.js:993:26)
at Connection.socketEnd (/home/AZ-admin/XXX/node_modules/tedious/lib/connection.js:1036:12)
at Socket.<anonymous> (/home/AZ-admin/XXX/node_modules/tedious/lib/connection.js:877:18)
at emitNone (events.js:111:20)
at Socket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)
or
SequelizeHostNotFoundError: Failed to connect to xxx-prod.database.windows.net:1433 - getaddrinfo ENOTFOUND xxx-prod.database.windows.net
at Connection.connection.on.err (/home/XXX/node_modules/sequelize/lib/dialects/mssql/connection-manager.js:98:22)
at emitOne (events.js:116:13)
at Connection.emit (events.js:211:7)
at Connection.socketError (/home/XXX/node_modules/tedious/lib/connection.js:1016:14)
at /home/XXX/node_modules/tedious/lib/connection.js:861:25
at GetAddrInfoReqWrap.callback (/home/XXX/node_modules/tedious/lib/connector.js:69:18)
at GetAddrInfoReqWrap.onlookupall [as oncomplete] (dns.js:79:17)
We think this could anything from an issue with connections/files being opened and not closed and reaching a limit to a problem with our DNS looking up the domain name.
It does appear to be somewhat tied to an increase in traffic, but the correlation is not 100% clear.
I saw your sample code missing options in dialectOptions.
It should be like below.
"dialectOptions": {
options: {
encrypt: true,
}
}
Before I have create a sample project to connect mssql and mysql. Hope it can help you. For more details, please check my answer in below post.
Azure Database for MySQL - webapp nodejs

Error! MongoNetworkError: failed to connect to server - after connect to another wifi network

I created an entire, working application with backend in Node.js and Mongodb (Mean stack). Everything is fine, but when I connect to another wifi network there is an error. Please help me.
If I have to send more code from the app, write it.
Error! MongoNetworkError: failed to connect to server [eventsdb-shard-00-00-ydx5k.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to eventsdb-shard-00-00-ydx5k.mongodb.net:27017 closed
at TLSSocket.<anonymous>
at Object.onceWrapper (events.js:300:26)
at TLSSocket.emit (events.js:210:5)
at net.js:659:12
at TCP.done (_tls_wrap.js:481:7) {
name: 'MongoNetworkError',
[Symbol(mongoErrorContextSymbol)]: {}
}]
...\App\server\node_modules\mongodb\lib\core\connection\connection.js:372:9 :
function closeHandler(conn) {
return function(hadError) {
if (connectionAccounting) deleteConnection(conn.id);
if (conn.logger.isDebug()) {
conn.logger.debug(`connection ${conn.id} with for [${conn.address}] closed`);
}
if (!hadError) {
conn.emit(
'close',
new MongoNetworkError(`connection ${conn.id} to ${conn.address} closed`), // <------ 372
conn
);
}
};
}
I solved my problem. On the Mongodb website, where I have a database, I entered "Network Access" -> "Add IP Address", I added my new ip address and there was no error.

LoopbackJS version 3.x `Connection fails: MongoError: Authentication failed.`

I am trying to follow up the tutorial, I created a new app with lb, added CoffeeShop mode, then added a datasource, mongodb.
My MongoDB instance is on my local mac, and authorization is turned off.
I just run it with mongod command, no extra params, and there are no addition configs.
this is my datasources.json
{
"corp1": {
"host": "127.0.0.1",
"port": 27017,
"url": "",
"database": "devdb",
"password": "devpassword",
"name": "corp1",
"user": "devuser",
"connector": "mongodb"
}
}
I even created devdb database, and given devdb user to all databases as admin.
And still getting this error.
Connection fails: MongoError: Authentication failed.
It will be retried for the next request.
/Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb/lib/mongo_client.js:462
throw err
^
MongoError: Authentication failed.
at Function.MongoError.create (/Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb-core/lib/error.js:31:11)
at /Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb-core/lib/connection/pool.js:497:72
at authenticateStragglers (/Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb-core/lib/connection/pool.js:443:16)
at Connection.messageHandler (/Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb-core/lib/connection/pool.js:477:5)
at Socket.<anonymous> (/Users/hazimdikenli/learn/loopback-getting-started/node_modules/mongodb-core/lib/connection/connection.js:333:22)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:597:20)
It am thinking that this is a newbie issue but I cannot find the problem. So please help me resolve this issue.
Although I am not running mongod with --auth option, it still turned out to be an authentication issue, it worked after I executed following.
db.createUser({
user : "devuser",
pwd : "devpassword",
roles: [ {role: "dbOwner", db: "devdb"},
{role: "readWrite", db: "devdb"}]
})
before this, I granted roles on "admin" with this script, but looks like it was not enough.
role: "userAdminAnyDatabase", db: "admin"

Mongoose bulkWrite - Wrong type for 'q'

For my test suite, I want to bulkWrite test info in the database, and then bulk delete any of the test info entered throughout the test to come back to a clean slate. I do so by running a bulkWrite on the db to which I pass the content of a JSON file loaded via nodeJS's require statement.
The problem is that for the dataset
[ { deleteOne: { username: 'test-author' } } ]
passed to models[key].collection.bulkWrite(action[key]), where key is the name of the model of interest and action is the JSON file,I get the following error:
{ MongoError: Wrong type for 'q'. Expected a object, got a null.
at Function.MongoError.create (/var/www/website/server/node_modules/mongodb-core/lib/error.js:31:11)
at /var/www/website/server/node_modules/mongodb-core/lib/connection/pool.js:483:72
at authenticateStragglers (/var/www/website/server/node_modules/mongodb-core/lib/connection/pool.js:429:16)
at Connection.messageHandler (/var/www/website/server/node_modules/mongodb-core/lib/connection/pool.js:463:5)
at Socket.<anonymous> (/var/www/website/server/node_modules/mongodb-core/lib/connection/connection.js:339:20)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)
at Socket.Readable.push (_stream_readable.js:134:10)
at TCP.onread (net.js:548:20)
name: 'MongoError',
message: 'Wrong type for \'q\'. Expected a object, got a null.',
ok: 0,
errmsg: 'Wrong type for \'q\'. Expected a object, got a null.',
code: 14,
codeName: 'TypeMismatch' }
I have done some research and have been unable to find a solution to this problem. The error itself is pretty meaningless, so I can't grasp much out of it. Any idea how to solve the problem?
Any help would be greatly appreciated!
Cheers!
As per the MongoDB API, the deleteOne, deleteMany, updateOne, updateMany, replaceOne, and replaceMany operation requires to have a property filter which acts as the filter for the query.
However Mongoose's API shows the following (mistaken) example:
Character.bulkWrite([
...
{
deleteOne: {
{ name: 'Eddard Stark' }
}
}
]).then(handleResult);
Hence, the data sent over changes from:
[{
"deleteOne": { "username": "test-author" }
}]
to
[{
"deleteOne": { "filter": { "username": "test-author" }}
}]
I'll make sure to pass the message along to the mongoosejs dev group.

Resources