Working with express(nodejs), mongoose, and Azure Cosmos DB to return objects.
When I connect to my local mongodb, the following code correctly returns a list of commit objects that exist in local mongodb.
Commit
.find({}, function(err, commits) {
if (err) {
res.render('search/index', {});
} else {
res.json(commits);
}
});
However, when connecting to Azure Cosmos DB using a PRIMARY CONNECTION STRING shown on my Azure portal website, the code just returns an empty list.
I checked that the mongoose.connection.readyState value is 1.
In addition, I can connect to the Azure Cosmos DB using Robo 3T.
Mongoose was designed to work with MongoDB. If your local testing with a real MongoDB server yields the expected result, then the fault is unlikely to be in mongoose or your code. Since CosmosDB only attempts to mimic MongoDB's API, there is no guarantee that it will work the same way. In your case, apparently it doesn't.
Being able to connect to CosmosDB using tools designed to work with MongoDB doesn't necessarily mean that CosmosDB will return the correct result.
If you require a cloud-based MongoDB deployment, using MongoDB Atlas is likely the best solution at this point in time.
Finally I could solve this problem by myself.
The latest version (v3.1.1) of the library doesn't work for connecting to Azure Cosmos DB.
You should use mongodb 2.2.33.
I found the solution from a comment on https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-samples.
Related
We are creating a NodeJS based solution that makes use of MongoDB, by means of Mongoose. We recently started adding support for Atlas, but we would like to be able to fallback to non-Atlas based queries, when Atlas is not available, for a given connection.
I can't assume the software will be using MongoDB Cloud. Although I could make assumptions based on the URL, I'd still need to have a way to be able to do something like:
const available: boolean = MyModel.connection.isAtlasAvailable()
The reason we want this is because if we make an assumption on Atlas being available and then the client uses a locally hosted MongoDB, the following code will break, since $search is Atlas specific:
const results = await Person.aggregate([
{
$search: {
index: 'people_text_index',
deleted: { $ne: true },
'text': {
'query': filter.query,
'path': {
wildcard: '*'
}
},
count: {
type: 'total'
}
}
},
{
$addFields: {
'mongoMeta': '$$SEARCH_META'
}
},
{ $skip : offset },
{ $limit: limit }
]);
I suppose I could surround this with a try/catch and then fall back to a non-Atlas search, but I'd rather check something is doable before trying an operation.
Is there any way to check whether MongoDB Atlas is available, for a given connection? As an extension to the question, does Mongoose provide a general pattern for checking for feature availability, such as if the connection supports transactions?
I suppose I could surround this with a try/catch and then fall back to a non-Atlas search, but I'd rather check something is doable before trying an operation.
As an isAtlasCluster() check, it would be more straightforward to use a regex match to confirm the hostname in the connection URI ends in mongodb.net as used by MongoDB Atlas clusters.
However, it would also be much more efficient to set a feature flag based on the connection URI when your application is initialised rather than using try/catch within the model on every request (which will add latency of at least one round trip failure for every search request).
I would also note that checking for an Atlas connection is not equivalent to checking if Atlas Search is configured for a collection. If your application requires some initial configuration of search indexes, you may want to have a more explicit feature flag configured by an app administrator or enabled as part of search index creation.
There are a few more considerations depending on the destination cluster tier:
Atlas free & shared tier clusters support fewer indexes so a complex application may have a minimum cluster tier requirement.
Atlas Serverless Instances (currently in preview) does not currently have support for Atlas Search (see Serverless Instance Limitations).
As an extension to the question, does Mongoose provide a general pattern checking for feature availability, such as if the connection supports transactions?
Multi-document transactions are supported in all non-EOL versions of MongoDB server (4.2+) as long as you are connected to a replica set or sharded cluster deployment using the WiredTiger storage engine (default for new deployments since MongoDB 3.2). MongoDB 4.0 also supports multi-document transactions, but only for replica set deployments using WiredTiger.
If your application has a requirement for multi-document transaction support, I would also check that on startup or make it part of your application deployment prerequisites.
Overall this feels like complexity that should be covered by prerequisites and set up of your application rather than runtime checks which may cause your application to behave unexpectedly even if the initial deployment seems fine.
I am trying to watch for changes in my collection but I am getting the following error:
"MongoError: Majority read concern requested, but it is not supported by the storage engine."
The answer seems to be: "To use watch you need to use replica set which is not part of mLab".
But i have a paid plan with a replica set. My connection to mlab looks like this.
mongoose.connect('mongodb://<dbuser>:<dbpassword>#ds327925-a0.mlab.com:27925,ds327925-a1.mlab.com:27925/<dbname>?replicaSet=rs-ds327925');
const taskCollection = db.collection('tasks');
const changeStream = taskCollection.watch();
changeStream.on('change', (change) => {
});
Majority read concern requires WiredTiger storage engine.
Availability of WiredTiger in mlab is limited to "dedicated" plans, apparently.
Besides upgrading your plan, you could also consider migrating to MongoDB Atlas.
In MongoDB 4.2+ change streams do not require majority read concern, but I don't imagine 4.2 is available in mlab either.
In our project we often have a problem when mongo doesn't save its state into disk, and after rebooting the application we lose data. I could not determine when and why this happens - somehow and somewhen :). Does anybody know how to synchronize mongodb storage to disk with some api? We use mongorito ODM. PLeasure to hear any variants.
Some details.
Mongo version 3.2.
Application - it is an electron application. Under the hood it uses mongo as storage - we use mongo on client side and install it as a windows service advantagely. Application starts, makes different transactions, read/write data from/to mondo db - nothing strange. When we close this application and reopen next time - we cannot find last rows (documents) in some collections that were succesfully (according to mongo answers) saved. We have no errors.
Can anyone explain what the write concern is and how to setup it not to wait 60 seconds before flushing the data - may be this is the reason?
Some code of db connect/disconnect. app means an electron application:
const {Database} = require('mongorito');
const db = new Database(__DBPATH__);
db.connect();
db.register(__MONGORITO_MODEL__);
app.on('window-all-closed', () => {
db.disconnect();
});
I'd take a look at the write concern setting within your application and make sure it's set to the requirements of your business - https://docs.mongodb.com/manual/reference/write-concern/
Also, make sure you're running a replica set in your production environment 👍
Thanks to everyboy, I've solved the problem. The reason was the journaling. I turn on the journaling for mongodb service and the problem has gone.
mongod.exe --journal
I'm building a webapp using the following the architecture:
a postgresql database (called DB),
a NodeJS service (called DBService) using Sequelize to manipulate the DB and Epilogue to expose a REST interface via Express,
a NodeJS service called Backend serving as a backend and using DBService threw REST calls
an AngularJS website called Frontend using Backend
Here are the version I'm using:
PostgreSQL 9.3
Sequelize 2.0.4
Epilogue 0.5.2
Express 4.13.3
My DB schema is quite complex containing 36 tables and some of them contains few hundreds of records. The DB is not meant to write data very often, but mostly to read them.
But recently I created a script in Backend to make a complete check up of datas contained inside the DB: basically this script retrieve all datas of all tables and do some basic checks on datas. Currently the script only does reading on database.
In order to achieve my script I had to remove the pagination limit of Epilogue by using the option pagination: false (see https://github.com/dchester/epilogue#pagination).
But now when I launch my script I randomly obtained that kind of error:
The request failed when trying to retrieve a uniquely associated objects with URL:http://localhost:3000/CallTypes/178/RendererThemes.
Code : -1
Message : Error: connect ECONNRESET 127.0.0.1:3000
The error randomly appears during the script execution: then it's not always this URL which is returned, and even not always the same tables or relations. The error message before code is a custom message returned by Backend.
The URL is a reference to the DBService but I don't see any error in it, even using logging: console.log in Sequelize and DEBUG=express:* to see what happens in Express.
I tried to put some setTimeout in my Backend script to slow it, without real change. I also tried to manipulate different values like PostgreSQL max_connections limit (I set the limit to 1000 connections), or Sequelize maxConcurrentQueries and pool values, but without success yet.
I did not find where I can customize the pool connection of Express, maybe it should do the trick.
I assume that the error comes from DBService, from the Express configuration or somewhere in the configuration of the DB (either in Sequelize/Epilogue or even in the postgreSQL server itself), but as I did not see any error in any log I'm not sure.
Any idea to help me solve it?
EDIT
After further investigation I may have found the answer which is very similar to How to avoid a NodeJS ECONNRESET error?
: I'm using my own object RestClient to do my http request and this object was built as a singleton with this method:
var NodeRestClient : any = require('node-rest-client').Client;
...
static getClient() {
if(RestClient.client == null) {
RestClient.client = new NodeRestClient();
}
return RestClient.client;
}
Then I was always using the same object to do all my requests and when the process was too fast, it created collisions... So I just removed the test if(RestClient.client == null) and for now it seems to work.
If there is a better way to manage that, by closing request or managing a pool feel free to contribute :)
I wrote a node web app and created a mongoDb database on my local system. I was using the following code to connect to local mongodb from node js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/db_name'); //local
And everything was working fine on my local machine. So I went on and created an mlab account and created a database. But when I tried to run the code by changing the connection string, connections are still established I believe. But the find and save requests are not invoking the callbacks, even no errors shows up. All requests are getting timed out.
var mongoose = require('mongoose');
mongoose.connect("mongodb://user:pass#ds036789.mlab.com:36789/db_name"); //mlab
Another thing I noticed is that I cannot ping ds036789.mlab.com. But TCP connections are succeeding when I tried the nc command
nc -w 3 -v ds036789.mlab.com 36789
I even tried deploying to azure. Which doesn't work either. Any help is much appreciated. Thanks.
EDIT:
Not being able to ping was due to the fact that I used azure hosting. It is expected. And I also found out that I get this error while trying to connect :
connection error: { [MongoError: auth failed] name: 'MongoError', ok: 0, errmsg: 'auth failed', code: 18 }
Credentials are correct though.
From the error mesasge it seems like you are using invalid auth details
This is most likely happen when you do not create username and password for individual database i.e, db_name in you case.
Check mLabs account and create username and password for db_name database and update your connection string.
According to the error information, as #Astro said, it seems to be caused by using invalid auth user/password which be created for database.
Did you create a new user for connecting the database, not account user for mlab? Such as the figures below shown.
Fig 1. A database user is required for connecting
Fig 2. Users list for the database
Hope it helps.
I figured out the issue, it wasn't an issue with the credentials. It was an issue with the mongoose version. The mongoose version I used didn't support the authentication. I had to remove the package and reinstall the latest version. with
node install mongoose#latest
Hope it helps someone. And thanks for the answers :)