I'm trying to prevent concurrent requests to a specific record, see the following example:
function addMoney(orderID,orderID){
const status = Database.getOrder(orderID);
if (status === 1){
return "Money Already Added";
}
Database.udateOrder(orderID, {status: 1});
Database.addMoney(userID, 300);
return true;
}
Assume someone made this request exactly at the same time, therefore the "status" check passed, they'd be able to get Database.addMoney run twice.
Using MySQL, I'd start a transction to lock the row but not sure how to do so using MongoDB.
You can do the transactions in mongodb like MySQL. Consider having an order document with id:123 and status:0. Then you can check for status in a transaction and return if it's already paid or fall through in order to add money document and update order status.
If you face any issue like Transaction numbers are only allowed on a replica set member or mongos this link might help.
In order to use transactions, you need a MongoDB replica set, and
starting a replica set locally for development is an involved process.
The new run-rs npm module makes starting replica sets easy.
const uri = 'mongodb://localhost:27017';
const dbName = 'myDB';
const MongoClient = require('mongodb').MongoClient;
async function main() {
const client = new MongoClient(uri);
await client.connect();
const session = client.startSession();
try {
await session.withTransaction(async () => {
const orders = client.db(dbName).collection('orders');
const money = client.db(dbName).collection('money');
let doc = await orders.findOne({orderID: 123});
if (doc && doc.status === 1) {
console.log("Money Already Added");
return
}
await orders.updateOne({orderID: 123}, {'$set': {status: 1}});
await money.insertOne({orderID: 123, userID: 100, amount: 300}, {session});
console.log("Money added");
});
await session.commitTransaction();
} catch (e) {
console.log(e);
} finally {
await session.endSession();
await client.close();
}
}
main()
The code above may need improvement because I couldn't test it on MongoDB with replica set.
Related
I'm just starting to use Mongodb without mongoose (to get away from the schemas), and wanted to create a simple module with various exported functions to use in the rest of my app. I've pasted the code below.
The problem I'm having is that the databasesList.databases comes back as undefined, and I'm not sure why. There should be 2 databases on my cluster, and one collection in each database.
As a tangential question, I thought maybe I would check the collections instead (now commented out), but though I found this page (https://docs.mongodb.com/manual/reference/method/db.getCollectionNames/) the function getCollectionNames seems not to exist. Now I'm wondering if I'm using the wrong documentation and that is why my databases are coming back undefined.
const client = new MongoClient(uri)
const connection = client.connect( function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server');
}
});
module.exports = {
getDatabaseList: function() {
console.log('start ' + client);
databasesList = client.db().admin().listDatabases();
//collectionList = client.db().getCollectionNames();
//console.log("Collections: " + collectionList);
console.log("Databases: " + databasesList.databases);
//databasesList.databases.forEach(db => console.log(` - ${db.name}`));
}
}```
your code is correct Just need to change few things.
module.exports = {
getDatabaseList: async function() {
console.log('start ' + client);
databasesList = await client.db().admin().listDatabases();
//collectionList = await client.db().getCollectionNames();
//console.log("Collections: " + collectionList);
console.log("Databases: " + databasesList.databases);
databasesList.databases.forEach(db => console.log(` - ${db.name}`));
}
}
You have to declare async function and use await also.
The async and await keywords enable asynchronous, promise-based behaviour to be written in a cleaner style, avoiding the need to explicitly configure promise chains.
You can use this modular approach to build your database access code:
index.js: Run your database application code, like list database names, collection names and read from a collection.
const connect = require('./database');
const dbFunctions = require('./dbFunctions');
const start = async function() {
const connection = await connect();
console.log('Connected...');
const dbNames = await dbFunctions.getDbNames(connection);
console.log(await dbNames.databases.map(e => e.name));
const colls = await dbFunctions.getCollNames(connection, 'test');
console.log(await colls.map(e => e.name));
console.log(await dbFunctions.getDocs(connection, 'test', 'test'));
};
start();
database.js:: Create a connection object. This connection is used for all your database access code. In general, a single connection creates a connection pool and this can be used throughout a small application
const { MongoClient } = require('mongodb');
const url = 'mongodb://localhost:27017/';
const opts = { useUnifiedTopology: true };
async function connect() {
console.log('Connecting to db server...');
return await MongoClient.connect(url, opts );
}
module.exports = connect;
dbFunctions.js:: Various functions to access database details, collection details and query a specific collection.
module.exports = {
// return list of database names
getDbNames: async function(conn) {
return await conn.db().admin().listDatabases( { nameOnly: true } );
},
// return collections list as an array for a given database
getCollNames: async function(conn, db) {
return await conn.db(db).listCollections().toArray();
},
// return documents as an array for a given database and collection
getDocs: async function(conn, db, coll) {
return await conn.db(db).collection(coll).find().toArray();
}
}
Here I am building the rental REST API and I want to perform transaction but it not rolling back the changes ,and it not giving any error problem is only changes are not rolling back.
const session = await startSession();
const { error } = validateRental(req.body);
if (error?.details[0].message)
return res.status(400).send(error.details[0].message);
const movie = await Movies.findById(req.body.movieID, null, {
$session: session,
}); // here i am adding the session
const customer = await Customer.findById(req.body.customerID, null, {
$session: session,
}); // same
if (!movie || !customer)
return res.status(400).send("Please check the customer or Movie ID");
try {
session.startTransaction();
movie.numberOfStock--;
customer.numberOfRental++;
await movie.save();
await customer.save();
await session.commitTransaction();
} catch (e) {
console.log(e);
await session.abortTransaction();
res.status(500).send("Internal System error");
} finally {
session.endSession();
}
After spending much time i got to know that my mongodb server is standalone , for write property we need to convert into Replica Set.
Follow the official documentation
Convert Standalone to a Replica set
Main point you should first close your server
My OS is window so - net stop mongob
For ios you can find at out yourself
I'm using shortid package to shorten my URLs.
Currently, user have this kind of url: https://bucard.co.il/digitalCard/5edd4112eb6ba017d8a4595c (the long string is the _id),
and I want to make it like this: https://bucard.co.il/digitalCard/Y2i1_53Vc
So, I added ShortID field, and as in the documantion, I did this in models/VisitCard.js:
const mongoose = require('mongoose');
const shortid = require('shortid');
const VisitCardsSchema = mongoose.Schema({
ShortID: {
type: String,
default: shortid.generate
},
....
});
module.exports = mongoose.model('VisitCards', VisitCardsSchema);
And my get request in routes/VisitCard.js:
// Get a specific visit card
router.get('/:visitCardId', async (req, res) => {
try {
let cardShortId = req.params.visitCardId;
let allVisitCards = await VisitCard.find({}); // That's how I saw that all the values changed after every get request.
let visitCard = await VisitCard.findOne({ ShortID: cardShortId }); // Never found the card by the short id - even after coping the short id from above, after the next try it changes.
if (!visitCard) {
return res.status(404).json({
message: 'Not existing card.'
});
} else {
return res.status(200).json(visitCard);
}
} catch (error) {
console.log(error);
return res.status(404).json({
message: 'Some server issue accured...'
});
}
});
Now, the proplem is where after every refresh of the browser or another get request, all the ShortID's of all cards are changing (generated again). I want instead that the short url will not be refreshes after every restart of the server, and it will be stored in the Database.
How can I do that after each card gets it's shortID (by default) it will directlly be stored in the DB ?
By the way, I could just have that after every submits of visit card to put some random string to be stored with the other paramters, but I already have visit cards of users in my service.
THANK YOU !!!
Tried reproducing your issue:
const mongoose = require("mongoose");
const shortid = require("shortid");
mongoose.connect("mongodb://localhost/test9999", {
useNewUrlParser: true
});
const db = mongoose.connection;
db.on("error", console.error.bind(console, "connection error:"));
db.once("open", async function() {
await mongoose.connection.db.dropDatabase();
// we're connected!
console.log("Connected");
const VisitCardsSchema = mongoose.Schema({
name: String,
ShortID: {
type: String,
default: shortid.generate
}
});
const VisitCard = mongoose.model("VisitCard", VisitCardsSchema);
const v1 = new VisitCard({name: "abc"});
const v2 = new VisitCard({name: "cde"});
await v1.save();
await v2.save();
await VisitCard.find(function(err, vcs) {
if (err) return console.error(err);
console.log(vcs);
});
console.log("VCS second call:");
await VisitCard.find(function(err, vcs) {
if (err) return console.error(err);
console.log(vcs);
});
});
Apparently, it works perfectly fine. I even commented part with dropping db for a moment and values are persisted correctly.
The problem must be somewhere else - you sure you do not drop the db or the collection with each GET request somewhere else in the code?
This one: await mongoose.connection.db.dropDatabase(); drops entire db, mongooseconnection.connection.db.dropCollection drops a collection. Check if you can find such lines somewhere in your code.
I have the following code, and can't understand why my process hangs on the line that tries to close the connection to mongodb, here is my code:
async function save(clientCredentials, newFieldsToUpdate){
const url = `mongodb://${clientCredentials.username}:${clientCredentials.password}#my.server.ip:22222/${clientCredentials.database}`
const client = await MongoClient.connect(url, {useNewUrlParser:true, useUnifiedTopology: true})
.catch(err => { console.log(err); });
const db = client.db(clientName);
const collection = await db.collection("products");
let execute = false;
const updateOps = [];
for(let objectIdentifier in newFieldsToUpdate){
let updateOperation = {};
updateOperation['$set'] = newFieldsToUpdate[objectIdentifier];
let id = mongodb.ObjectID(objectIdentifier);
execute = true;
updateOps.push({ updateOne: { filter: {_id: id}, update: {$set: newFieldsToUpdate[objectIdentifier]}, upsert:true } })
}
if(execute){
try {
console.log('executing'); // I see this line
let report = await collection.bulkWrite(updateOps);
console.log('executed'); // I see this line
await client.close();
console.log('closed conn'); // I don't see this line! why? it's weird
return report;
} catch(ex){
console.error(ex);
}
} else {
console.log('not executing');
}
}
Thank you in advance for any help!
EDIT: The bulk operation is of about 200 documents only, if I try with a single document it works, this is weird. Mongodb driver for node version 3.3.2
EDIT2: I notice that using the parameter poolSize:1 on the mongo connect it closes the connection with success, but using the default poolSize of 5 it doesnt close, any suggestions why this might be happening?
Make sure you are using latest version of mongodb driver (delete node_modules just to be sure) as older version has a bug in which bulkWrite method fails silently because of some bug mentioned here - https://stackoverflow.com/a/46700933/1021796
Before trying to close the connection, you shou also check if it is connected or not. so code will be
if (client.isConnected) {
await client.close();
}
Hope this helps
8 out of ten times everything connects well. That said, I sometimes get a MongoClient must be connected before calling MongoClient.prototype.db error. How should I change my code so it works reliably (100%)?
I tried a code snippet from one of the creators of the Now Zeit platform.
My handler
const { send } = require('micro');
const { handleErrors } = require('../../../lib/errors');
const cors = require('../../../lib/cors')();
const qs = require('micro-query');
const mongo = require('../../../lib/mongo');
const { ObjectId } = require('mongodb');
const handler = async (req, res) => {
let { limit = 5 } = qs(req);
limit = parseInt(limit);
limit = limit > 10 ? 10 : limit;
const db = await mongo();
const games = await db
.collection('games_v3')
.aggregate([
{
$match: {
removed: { $ne: true }
}
},
{ $sample: { size: limit } }
])
.toArray();
send(res, 200, games);
};
module.exports = handleErrors(cors(handler));
My mongo script that reuses the connection in case the lambda is still warm:
// Based on: https://spectrum.chat/zeit/now/now-2-0-connect-to-database-on-every-function-invocation~e25b9e64-6271-4e15-822a-ddde047fa43d?m=MTU0NDkxODA3NDExMg==
const MongoClient = require('mongodb').MongoClient;
if (!process.env.MONGODB_URI) {
throw new Error('Missing env MONGODB_URI');
}
let client = null;
module.exports = function getDb(fn) {
if (client && !client.isConnected) {
client = null;
console.log('[mongo] client discard');
}
if (client === null) {
client = new MongoClient(process.env.MONGODB_URI, {
useNewUrlParser: true
});
console.log('[mongo] client init');
} else if (client.isConnected) {
console.log('[mongo] client connected, quick return');
return client.db(process.env.MONGO_DB_NAME);
}
return new Promise((resolve, reject) => {
client.connect(err => {
if (err) {
client = null;
console.error('[mongo] client err', err);
return reject(err);
}
console.log('[mongo] connected');
resolve(client.db(process.env.MONGO_DB_NAME));
});
});
};
I need my handler to be 100% reliable.
if (client && !client.isConnected) {
client = null;
console.log('[mongo] client discard');
}
This code can cause problems! Even though you're setting client to null, that client still exists, will continue connecting to mongo, will not be garbage collected, and its callback connection code will still run, but in its callback client will refer to the next client that's created that is not necessarily connected.
A common pattern for this kind of code is to only ever return a single promise from the getDB call:
let clientP = null;
function getDb(fn) {
if (clientP) return clientP;
clientP = new Promise((resolve, reject) => {
client = new MongoClient(process.env.MONGODB_URI, {
useNewUrlParser: true
});
client.connect(err => {
if (err) {
console.error('[mongo] client err', err);
return reject(err);
}
console.log('[mongo] connected');
resolve(client.db(process.env.MONGO_DB_NAME));
});
});
return clientP;
};
I had the same issue. In my case it was caused by calling getDb() before a previous getDb() call had returned. In this case, I believe that 'client.isConnected' returns true, even though it is still connecting.
This was caused by forgetting to put an 'await' before the getDb() call in one location. I tracked down which by outputting a callstack from getDb using:
console.log(new Error().stack);
I don't see the same issue in the sample code in the question, though it could be triggered by another bit of code that isn't shown.
I have written this article talking about serverless, lambda e db connections. There are some good concepts which could help you to find the root cause of your problem. There are also example and use cases of how to mitigate connection pool issues.
Just by looking your code I can tell it is missing this:
context.callbackWaitsForEmptyEventLoop = false;
Serverless: Dynamodb x Mongodb x Aurora serverless