I want to run a bulk operation in mongo but at the same time run it inside a multi document transaction. I am using Meteor running the NodeJs driver of MongoDb.
According to the MongoDB documentation (https://docs.mongodb.com/manual/reference/method/Bulk/) it should be possible to combine Bulk and multi-document transactions. However, I have not been able to solve this.
My problem is how to pass the session object to the Bulk operations. For non-bulk operations we just pass it as an options object const options = {session: session}. I have tried to pass it in several ways but nothing seems to work.
How should the session object be used in Bulk operations?
Below is a simple example of what I am trying to achive.
const getMongoClient = function() {
const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
return client;
}
const session = client.startSession();
try {
session.startTransaction();
const bulk = someCollectionToBulkWrite.rawCollection().initializeOrderedBulkOp(); // Pass the session here?
const dataInsert = someCollection.insert(someObject, {session}).await().value;
const dataInsertOtherDocument = someOtherCollection.insert(someOtherObject, {session}).await().value;
bulk.find( { _id: someId } ).update( { $set: { testField: 3}}); //Or pass the session here?
bulk.find( { _id: someOtherId } ).update( { $set: { testField: 3}});
bulk.execute().await(); // Or pass the session here?
session.commitTransaction().await();
} finally {
session.endSession();
}
I checked the code for the MongoDB driver for NodeJS and more specifically for the Bulk API (https://github.com/mongodb/node-mongodb-native/blob/master/lib/bulk/common.js)
The definition of the execute methods looks as follows:
execute(_writeConcern, options, callback);
The session should thus be passed as the second argument to execute. In the example provided in the question that would look like this:
bulk.execute(null, {session}).await();
Related
i am building hubspot api, i having trouble paginating the contacts records.
i am using #hubspot/api-client - npm for integration with hubspot and this is the docs for that https://github.com/HubSpot/hubspot-api-nodejs
hubspotClient.crm.contacts.basicApi
.getPage(limit, after, properties, propertiesWithHistory, associations, archived)
.then((results) => {
console.log(results)
})
.catch((err) => {
console.error(err)
})
in this code there is after argument, we can provide contacts id in it, and it will provide the records including and after that particular id.
How do i use this for pagination. or there is any other way.
Take a look at API Endpoints documentation for GET /crm/v3/objects/contacts and the data you receive. The getPage method returns the following data:
{
"results": [
{
// contact detail here
}
],
"paging": {
"next": {
"after": "NTI1Cg%3D%3D",
"link": "?after=NTI1Cg%3D%3D"
}
}
}
The pagination information is available in paging.next.after (if there is a consecutive page). So you can do something like this to iterate through each page:
async function doSomethingWithEachPage() {
let after = undefined;
const limit = 10;
const properties = undefined;
const propertiesWithHistory = undefined;
const associations = undefined;
const archived = false;
do {
const response = await hubspotClient.crm.contacts.basicApi.getPage(
limit,
after,
properties,
propertiesWithHistory,
associations,
archived
);
// do something with results
console.log(response.results); // contacts list
// pick after from response and store it outside of current scope
after = response.paging?.next?.after;
} while (after);
}
I rewrote the sample code to use async/await so it better works with do...while loop and omitted error handling.
If you are dealing with reasonable small amount of data and have enough of memory, you can also skip the pagination and use the getAll method to load all the data. (In fact, this method does internally a loop similar to the one above.)
I am trying to access a MongoDB Collection and I want to console.log a Object from this Collection.
I connect to my MongoDB as follows:
async function main(){
/**
* Connection URI. Update <username>, <password>, and <your-cluster-url> to reflect your cluster.
* See https://docs.mongodb.com/ecosystem/drivers/node/ for more details
*/
const uri = process.env.DB_Port;
const client = new MongoClient(uri,{ useNewUrlParser: true, useUnifiedTopology: true});
try {
// Connect to the MongoDB cluster
await client.connect();
// Make the appropriate DB calls
await listDatabases(client);
} catch (e) {
console.error(e);
} finally {
await client.close();
}
}
main().catch(console.error);
async function listDatabases(client){
databasesList = await client.db().admin().listDatabases();
console.log("Databases:");
databasesList.databases.forEach(db => console.log(` - ${db.name}`));
};
This console logs:
Databases:
- radiototem
- test
- admin
- local
Now I want to access the collection test and console log everything that is inside it. How can I achieve this?
Best regards
In order to get items from MongoDB, you first need to access the collection in which they are stored. Collections in MongoDB are basically the same as tables in SQL.
To get a collection, call the .collection() function on your DB object you get from client.db() with the name of your collection like this:
client.db().collection('test'); // This gives you the collection "test"
If you want to get items from a collection, you can use the .find() method. You can pass it a query parameter which is an object where you define, which items should be selected based on their properties.
Example, get all users named peter from the users collection:
const db = client.db();
const users = db.collection('users');
const usersNamedPeterCursor = users.find({ name: 'Peter' });
Now if you want to get all items from a collection, you can simply use the find method without the query parameter. This will return all items from the collection.
The find method returns a Cursor object which lets you interact with the returned data. You can call methods on the Cursor object like max(), min(), toArray(), each() and many more.
So, if you want to console.log every item in your collection you can do it like this:
client.db().collection('test').find().each(function(error, item) {
// console .log your item or do something else with it
});
I'm working on caching strategies for an application that uses knex.js for all sql related stuff.
Is there a way to intercept the query to check if it can be fetched from a cache instead of querying the database?
Briefly looked into knex.js events, which has a query event.
Doc:
A query event is fired just before a query takes place, providing data about the query, including the connection's __knexUid / __knexTxId properties and any other information about the query as described in toSQL. Useful for logging all queries throughout your application.
Which means that it's possible to do something like (also from docs)
.from('users')
.on('query', function(data) {
app.log(data);
})
.then(function() {
// ...
});
But is it possible to make the on query method intercept and do some logic before actually executing the query towards the database?
I note that this suggestion is attached to a Knex GitHub issue (credit to Arian Santrach) which seems relevant:
knex.QueryBuilder.extend('cache', async function () {
try {
const cacheKey = this.toString()
if(cache[cacheKey]) {
return cache[cacheKey]
}
const data = await this
cache[cacheKey] = data
return data
} catch (e) {
throw new Error(e)
}
});
This would allow:
knex('tablename').where(criteria).cache()
to check for cached data for the same query. I would think a similar sort of structure could be used for whatever your caching solution was, using the query's string representation as the key.
What's the best practice for sharing DB query code between multiple Node.js Express controller methods? I’ve searched but the samples I’ve found don’t really get into this.
For example, I have this getUser method (using Knex for MySQL) that makes a call to get user info. I want to use it in other methods but I don't need all the surrounding stuff like the response object.
export let getUser = (req: Request, res: Response, next: NextFunction) => {
try {
knex.select().where('email', req.params.email)
.table('users')
.then( (dbResults) => {
const results: IUser = dbResults[0];
res
.status(200)
.set({ 'Content-Type': 'application/json', 'Connection': 'close' })
.send(results);
});
} catch (err) {
res.send({ error: "Error getting person" + req.params.email });
return next(err);
}
};
It seems wrong to repeat the query code somewhere else where I need to get the user. Should I turn my DB query code into async functions like this example and then call them from within the controller methods that use the query? Is there a simpler way?
/**
* #param {string} email
*/
async function getUserId(email: string) {
try {
return await knex.select('id')
.where('email', email)
.table('users');
} catch (err) {
return err;
}
}
You can for example create "service" modules, which contain helpers for certain type of queries. Or you could use ORM and implement special queries in each model that is called "fat model" design. Pretty much anything goes as long as you remember to not create new knex instance in every helper module, but you just pass knex (containing its connection pool) for the helper methods so that all queries will share the same connection pool.
ORM's like objection.js also provides a way to extend query builder API so you can inherit custom query builder with any special query helper that you need.
I would like to update my API call that creates a new document, to create the new document, populate it, and then return the json response via express.
I have my model set up as follows:
import mongoose from 'mongoose';
import constants from '../../constants/enums';
const DepartmentBudgetSchema = new mongoose.Schema({
department : {
type : String,
enum : constants.departments
},
owner : {
type : mongoose.Schema.Types.ObjectId,
ref : 'Executive'
},
budget : Number
});
export default mongoose.model('DepartmentBudget', DepartmentBudgetSchema);
I have my API calls setup with my express paths, which make a call to my logic, which in turn makes a call to my controller, where I handle all the database work.
My express api call looks like the following:
import express from 'express';
import DepartmentsBudgetsLogic from './departmentBudget.logic.js';
import constants from '../../constants/enums';
const router = express.Router();
const code = constants.statusCode;
router.post('/', newDepartmentBudget);
async function newDepartmentBudget( req, res ) {
try {
const result = await new DepartmentsBudgetsLogic().newDepartmentBudget(req.body);
console.log(result);
res.status(code.created).json(result);
}
catch ( error ) {
console.error('Unable to create department budget.', error);
res.sendStatus(code.serverError);
}
}
Then my logic function simply calls my controller passing the budget object:
import DepartmentBudgetController from './departmentBudget.controller';
export default class DepartmentBudgetLogic {
newDepartmentBudget( budget ) {
return new DepartmentBudgetController().newDepartmentBudget(budget);
}
}
Finally, and this is where I think I am having my problem, the controller handles the mongoose work:
import DepartmentBudget from './departmentBudget.model';
export default class DepartmentBudgetController {
newDepartmentBudget( budget ) {
DepartmentBudget.create(budget).then(function( budget ) {
return budget.populate('owner').execPopulate();
});
}
}
The document gets created in the database correctly, but I am really struggling with the code to populate the owner field and then return the populated doc as my response.
I know from mongoose's docs that execPopulate() returns a promise that resolves to the populated document, but when I console.log the result in my express api call I get "undefined".
I'm still learning here, so I may have overlooked something very obvious.
Anyone have any ideas on what I'm doing wrong? Or is there a different direction I should go in so that I can accomplish the overall objective of being able to create a new document, populate it, and then return it in my response object.
Thank you!