I am trying to connect node.js app to MongoDB having replica set but it's throwing an error when any write operations are performed.
It throws MongoError: not master.
It tries to write on secondary mongo instances.
I have the options as { db: { readPreference: secondaryPreferred } } and passing it to the function MongoClient.connect in the node.js code using native Mongo Driver.
The URL used to connect looks like mongodb://admin:pass#host_one:27017,host_two:27017,host_three:27017/dbName
Any help would be really appreciated.
Did you add in your replicaSet name?
mongodb://admin:pass#host_one:27017,host_two:27017,host_three:27017/dbName?replicaSet=my-replica-set
replicaSet=name
The driver verifies that the name of the replica set it connects to
matches this name. Implies that the hosts given are a seed list, and
the driver will attempt to find all members of the set. No default
value.
If this is not set it will be treated as a standalone node.
Maybe your replica set configuration is not correct.
To check the configuration run the rs.conf() command in your mongo servers. You need to have a mongo host running as primary member.
MongoError: Not master
This error seems like your primary member of replica set is not configured properly.
You can confirm this by entering into mongo shell of the host_one. If mongo shell prompt doesn't show PRIMARY, then it's not configured properly.
Mongo shell prompt of host_two and host_three should show SECONDARY after proper configuration.
Important : Run rs.initiate() on just one and only one mongod instance for the replica set.
You can execute this command on the primary member to make the configuration work properly.
rs.initiate();
cfg = {
_id: 'rs0',
members: [{
_id: 0,
host: 'host_one:27017',
priority: 2
}, {
_id: 1,
host: 'host_two:27017',
priority: 1
}, {
_id: 2,
host: 'host_three:27017',
priority: 1
}]
};
cfg.protocolVersion = 1;
rs.reconfig(cfg, {
force: true
});
Please note that priority value indicates the relative eligibility of a member to become a primary.
Specify higher values to make a member more eligible to become primary, and lower values to make the member less eligible. A member with a priority of 0 is ineligible to become primary.
You can again check your replica set configuration using this command
rs.conf()
Read preference is not applicable to writes. Writes must always be performed on the primary.
You should be connecting to replica set instead of directly to an individual node. See node.js mongodb how to connect to replicaset of mongo servers
Related
We are creating a NodeJS based solution that makes use of MongoDB, by means of Mongoose. We recently started adding support for Atlas, but we would like to be able to fallback to non-Atlas based queries, when Atlas is not available, for a given connection.
I can't assume the software will be using MongoDB Cloud. Although I could make assumptions based on the URL, I'd still need to have a way to be able to do something like:
const available: boolean = MyModel.connection.isAtlasAvailable()
The reason we want this is because if we make an assumption on Atlas being available and then the client uses a locally hosted MongoDB, the following code will break, since $search is Atlas specific:
const results = await Person.aggregate([
{
$search: {
index: 'people_text_index',
deleted: { $ne: true },
'text': {
'query': filter.query,
'path': {
wildcard: '*'
}
},
count: {
type: 'total'
}
}
},
{
$addFields: {
'mongoMeta': '$$SEARCH_META'
}
},
{ $skip : offset },
{ $limit: limit }
]);
I suppose I could surround this with a try/catch and then fall back to a non-Atlas search, but I'd rather check something is doable before trying an operation.
Is there any way to check whether MongoDB Atlas is available, for a given connection? As an extension to the question, does Mongoose provide a general pattern for checking for feature availability, such as if the connection supports transactions?
I suppose I could surround this with a try/catch and then fall back to a non-Atlas search, but I'd rather check something is doable before trying an operation.
As an isAtlasCluster() check, it would be more straightforward to use a regex match to confirm the hostname in the connection URI ends in mongodb.net as used by MongoDB Atlas clusters.
However, it would also be much more efficient to set a feature flag based on the connection URI when your application is initialised rather than using try/catch within the model on every request (which will add latency of at least one round trip failure for every search request).
I would also note that checking for an Atlas connection is not equivalent to checking if Atlas Search is configured for a collection. If your application requires some initial configuration of search indexes, you may want to have a more explicit feature flag configured by an app administrator or enabled as part of search index creation.
There are a few more considerations depending on the destination cluster tier:
Atlas free & shared tier clusters support fewer indexes so a complex application may have a minimum cluster tier requirement.
Atlas Serverless Instances (currently in preview) does not currently have support for Atlas Search (see Serverless Instance Limitations).
As an extension to the question, does Mongoose provide a general pattern checking for feature availability, such as if the connection supports transactions?
Multi-document transactions are supported in all non-EOL versions of MongoDB server (4.2+) as long as you are connected to a replica set or sharded cluster deployment using the WiredTiger storage engine (default for new deployments since MongoDB 3.2). MongoDB 4.0 also supports multi-document transactions, but only for replica set deployments using WiredTiger.
If your application has a requirement for multi-document transaction support, I would also check that on startup or make it part of your application deployment prerequisites.
Overall this feels like complexity that should be covered by prerequisites and set up of your application rather than runtime checks which may cause your application to behave unexpectedly even if the initial deployment seems fine.
I'm attempting to connect to a new Aurora PostgreSQL instance with Babelfish enabled.
NOTE: I am able to connect to the instance using the pg library through the normal port 5432 (the Postgres TDAS endpoint).
However, for this test, I am attempting to connect through the Babelfish TDS endpoint (1433) using the standard mssql package.
If I specify a database name (it is correct), I receive the error 'database "postgres" does not exist':
var config = {
server: 'xxx.us-east-1.rds.amazonaws.com',
database: 'postgres',
user: 'xxx',
password: 'xxx'
};
and the connection closes since the connection fails.
if I omit the database property in the config, like:
var config = {
server: 'xxx.us-east-1.rds.amazonaws.com',
user: 'xxx',
password: 'xxx'
};
It will connect. Also, I can use that connection to query basic things like SELECT CURRENT_TIMESTAMP and it works!
However, I can't access any tables.
If I run:
SELECT COUNT(1) FROM PERSON
I receive an error 'relation "person" does not exist'.
If I dot-notate it:
SELECT COUNT(1) FROM postgres.dbo."PERSON"
I receive an error "Cross DB query is not supported".
So, I can't connect to the specific database directly and if I connect without specifying a database, I can't cross-query to the table.
Any one done this yet?
Or, if not, any ideas on helping me figure out what to try next? I'm out of ideas.
Babelfish databases (that you connect to on port 1433) have nothing to do with PostgreSQL databases (port 5432). Essentially, all of Babelfish lives within a single PostgreSQL database (parameter babelfishpg_tsql.database_name).
You seem to have a single-db setup, because Cross DB query is not supported. With such a setup, you can only have a single database via port 1433 (apart from master and tempdb). You have to use CREATE DATABASE to create that single database (if it isn't already created; ask sys.databases).
I can't tell if it is supported to create a table in PostgreSQL (port 5432) and use it on port 1433 (the other way around is fine), but if so, you have to create it in a schema that you created with CREATE SCHEMA while connected on port 1433.
The answer was that I should be connecting to database "master".
Even though there is no database titled master in the instance, you still do connect to it.
Once connected, running the following:
select current_database();
This will indicate you are connected to database "babelfish_db".
I don't know how that works or why a database would have an undocumented alias.
The bigger answer here is that cross-DB object references are not currently supported in Babelfish, outside your current SQL Server database.
This is currently being worked on. Stay tuned.
I made an test account in datastax (https://astra.datastax.com/) and want to test cassandra.
In there homepage is an cqlsh console. If I select datas is goes very fast maybe 1ms.
If I use it with nodejs and cassandra driver it takes 2-3 seconds. And I have only ONE row.
Why it takes time? Its my code fault?
const { Client } = require("cassandra-driver");
async function run() {
const client = new Client({
cloud: {
secureConnectBundle: "secure-connect-weinf.zip",
},
keyspace: 'wf_db',
credentials: {
username: "admin",
password: "password",
},
});
await client.connect();
// Execute a query
const rs = await client.execute("SELECT * FROM employ_by_id;");
console.log(`${rs}`);
await client.shutdown();
}
// Run the async function
run();
Unfortunately, it's not an apples-for-apples comparison.
Every time your app connects to a Cassandra cluster (Astra or otherwise), the driver executes these high-level steps:
Unpack the secure bundle to get cluster info
Open a TCP connection over the internet
Create a control connection to one of the nodes in the cluster
Obtain schema from the cluster using the control connection
Discover the topology of the cluster using the control connection
Open connections to the nodes in the cluster
Compute query plan (list of hosts to connect to based on load-balancing policy)
And finally, run the query
In contrast when you access the CQL Console on the Astra dashboard, the UI automatically connects + authenticates to the cluster and when you type a CQL statement it goes through the following steps:
Skipped (you're already authenticated to the cluster)
Skipped (it's already connected to a node within the same local VPC)
Skipped (already connected to cluster)
Skipped (already connected to cluster)
Skipped (already connected to cluster)
Skipped (already connected to cluster)
Skipped (already connected to cluster)
And finally, run the query
As you can see, the CQL Console does not have the same overhead as running an app repeatedly which only has 1 CQL statement in it.
In reality, your app will be reusing the same cluster session to execute queries throughout the life of the app so it doesn't have the same overhead as just re-running the app you have above. The initialisation phase (steps 1 to 6 above) are only done when the app is started. Once it's already running, it only has to do steps 7 and 8. Cheers!
I have to deploy a replica set using nodeJS script.If I've to add or remove some member from replica set on a later instance of time then I should be able to do so using script.
The problem is that I couldn't find any thing like db.initiate(config) similar to rs.initiate(config).will it be db.admin().initiate(config)?
If you want to send replica set configuration commands via Node.js, you will need to use the replica set admin commands:
replSetInitiate to initialise the replica set
replSetReconfig for any subsequent changes (i.e. adding or removing members)
The test/tools/replicaset_manager.js file in the MongoDB Node driver test suite includes some example code for managing replica sets that should be useful, in particular reStartAndConfigure().
The general steps for replica set config changes, as per reStartAndConfigure(), include:
connect to the local database
var connection = new Db("local", ... )
get the current replica set configuration from system.replset collection:
db.collection('system.replset').findOne({}, function(err, doc) { ...
iterate over all the member docs and apply any config changes
for (var i = 0; i < doc.members.length; i++) { ... }
increment the replica set config version
doc.version = doc.version + 1;
reconfigure using replSetReconfig command
db.admin().command({replSetReconfig: doc}, ... )
You can also see examples of how the replSetReconfig command is used in the mongo shell by invoking the rs.add and rs.remove shell helpers without the brackets.
A common connection string for mongoose connecting to a replica set is something like follows
var connection = mongoose.createConnection("mongodb://db_1:27017/client_test,mongodb://db_2:27017/client_test", {
replSet : { rs_name : "rs0", poolSize : 5, socketOptions : { keepAlive : 1 } }
}, function(err) {
if (err) { throw err; }
});
The problem with that is if one of the two hosts is down, then it will fail to connect. If you only specify one host, then no requests end up getting sent to secondaries.
Here's my proof for that claim. If you specify one host, and setup your replica set so that there is one primary and an arbiter and then perform a query such as
myApi.find({}).slaveOk().read("s").exec(function(err, docs) {
console.log(docs)
})
It will return results. Well, since I am specifying "s" (secondary), this query should throw an error because there are no running secondaries. In addition, if you bring the secondary online and then do db.currentOp(true), you will never see any actual queries sent it's way.
The moment you alter the connection string to specify every host then you will see connections go to the secondary. The dilemma is that now, because you had to specify the additional host in the connection string, in the event a secondary was offline, it would fail to connect and we've now lost failover (or the entire point to replica sets)
I can't determine if this is a configuration mistake on my part, a bug in Mongoose, or a conceptual flaw in my understanding of the way replica sets function. From some of the docs, they seem to state that reading from secondaries is basically a bad idea, but the reason for doing so is usually issues with stale data. My issue doesn't have anything to do with stale date, I can't figure out a way to setup the system so that I can get queries to secondaries without losing failover capacity.
1.connection string just defines seed servers, mongodb driver tries to connect to these servers and get information about other servers in replicaSet( by calling rs.status()). You could have replicaSet with 5 nodes, but specify only one in connection string, but driver would be able to find four others if server from connection string is available.
2.My proposal is to use secondaryPreferred instead of just secondary, so that in case there is no secondary available, request would be done to primary.
Ok, I believe I have solved all of my problems. Here is what I learned.
Specify all possible replica nodes in your connection string, otherwise Mongoose will never send requests there. Mongoose has a specific format for this which is different than the node-mongodb-native driver. Example below.
In order to prevent it from hanging forever if one of the nodes is down on bootup you need to specify connectTimeoutMS in the 'replset' options, then it will only wait that long for responses from each nodes on initial connection. If the node comes online at a later date, it will still be available.
The host name entries in your mongodb replica setup need to match the hostname entries in the connection string from your application and all hostnames need to be accessible from all parties (mongo to mongo and application to mongo). In my case I had aliased the hostnames from mongo to mongo as mongo1:27017, mongo2:27017, and mongo3:27017. My application server used a connection string with IPs. Mongoose was attempting to re-initate the connection using the mongo1:27017 hostname (which my application server could not reach) rather than the IP address I specified in the connection string. This resulted in it never re-connecting to a node it lost contact with. It is possible had I used hostnames that the application could reach it still would have worked, but I think it's a best practice to make the connection string and the replica setup identical to remove possibly places for error.
On the mongodb node that you rs.initiate() you might need to update the hostname to be a value that all boxes (other mongodbs and application server can reach). By default it will likely end up with a hostname like localhost, which means something different on each machine. This can be from that boxes mongo shell like so.
Example:
// from mongo shell
conf = rs.conf()
conf.members[0].host = "mongo1:27017"
rs.reconfig(conf)
Final functioning connection string which successfully fails over between nodes, including throwing errors if a query is destined for a secondary but there aren't secondaries.
var connection = mongoose.createConnection("mongodb://mongo1:27017/client_test,mongo2:27017/client_test,mongo3:27017/client_test", {
replset : { rs_name : "rs0", poolSize : 5, socketOptions : { keepAlive : 1, connectTimeoutMS : 1000 } },
}, function(err) {
if (err) { throw err; }
});
Working replica setup
{
"_id" : "rs0",
"version" : 4,
"members" : [
{
"_id" : 0,
"host" : "mongo1:27017"
},
{
"_id" : 1,
"host" : "mongo2:27017"
},
{
"_id" : 2,
"host" : "mongo3:27017",
"arbiterOnly" : true
}
]
}
I had some issue similar to yours while dealing with replica, in my case I had 1 primary node with a priority of 10, 1 secondary priority of 0(for analytics) and an arbiter.
My writes would fail after reconnecting the primary instance and I went through a lot trying to fix it here's the most important thing I learnt:
When my primary is down or unreacheable, there has to be another member eligible to become primary.(At least 2members in my set has to have a priority >= 1).
If I have only arbiters, hidden, or members with a priority of 0,
queries will get stuck even after I reconnect my primary, my client is
unable to complete write queries. Read queries would still work, but
write wouldn't.
This is what I faced with mongoose, even with keepalive, autoreconnect and all the socket and connection timeout MS set.
Hopefully this helps.