i've been trying to use cache with redis in my nodejs mongodb application but i didn't find any tutorial on how to do that except few which are using logic that seems to not be explained in mongoose documentation
const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async
function (){
// our caching logic
return await exec.apply(this, arguments);
}
where does arguments come from ? because it seems to be undefined yet it is used
mongoose.Query.prototype.exec = async function(){
const collectionName = this.mongooseCollection.name;
if(this.cacheMe){
// You can't insert json straight to redis needs to be a string
const key = JSON.stringify({...this.getOptions(),
collectionName : collectionName, op : this.op});
const cachedResults = await redis.HGET(collectionName,key);
// this.op is the method which in our case is "find"
if (cachedResults){
// if you found cached results return it;
const result = JSON.parse(cachedResults);
return result;
}
//else
// get results from Database then cache it
const result = await exec.apply(this,arguments);
redis.HSET(collectionName, key, JSON.stringify(result) , "EX",this.cacheTime);
//Blogs - > {op: "find" , ... the original query} -> result we got from database
return result;
}
clearCachedData(collectionName, this.op);
return exec.apply(this,arguments);
}
and what is this.getOptions()?
i would be thankful if any one can explain me this logic, because i did not find any help in the documentations nor internet blogs and articls
The arguments object is a local variable that is available inside every function in JavaScript and contains the values of the arguments passed to the function.
this.getOptions() is the local method that returns the options to the query.
// A key example for mongoose Redis integration
const key = JSON.stringify({
collectionName: this.mongooseCollection.name,
op: this.op,
options: this.getOptions(),
filter: this.getFilter(),
projection: this.projection(),
populatedPaths: this.getPopulatedPaths(),
});
There are a lot of similar packages on NPM, but I highly recommend standard mongoose and redis ones to get up and running. I assumed your initialization point was similar to this post. This can also be a relevant source.
Related
I'm trying to implement an automatic caching system where any normal call to mongodb using mongoose uses redis as a middleman to cache it. And the problem is, I've gotten storing it, fetching it, but I can't use a callback. I've dug forever and can't find out how to call it. I've found some chunks ( How mongoose calls ) but I can't find the piece that is the actual callback I specified. Through some investigation, I found that returning doesn't do anything, and the only way I could figure out how to callback was using mongooseExec.apply(this, arguments) but that doesn't help when I need to call it once fetching because calling .apply also fetches from mongodb which I don't want if it's already cached. Here's the reduced code:
const mongooseExec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async function () {
const key = await JSON.stringify({
...this.getQuery(),
collection: this.mongooseCollection.name,
op: this.op,
options: this.options,
});
let cached = await self.redisClient.get(key);
let result;
if (cached) {
console.log("CACHED");
return await JSON.parse(cached)
} else {
result = await mongooseExec.apply(this, arguments);
}
};
The callback is passed as an argument to the exec function. For example, in find:
this.exec(callback);
Your exec function needs to accept a callback function as its first argument.
I have some async code that makes calls to a mongo database and inserts/fetches items. When I am developing locally, the code below works fine. However, when I make the mongoose instance connect to MongoDb Atlas, issues arise. In particular, it seems that my code does not work properly unless I console.log the promise, which makes no sense to me. For example, with the console.log statement, all my tests pass as expected. Without it, 35 tests fail... This is because the promise I am expecting returns null, when it should return some JSON object from the database. Is my code not blocking properly?
It feels like I'm dealing with Schrodinger's cat... Any help would be appreciated. Thanks in advance.
Below is an example promise/function call. I then pass it into _executeQuery. I have await on relevant functions, so I don't think it's because I'm missing the word await somewhere.
async _inSomeAsyncFunction = () => {
const dbQueryPromise = this._dbModel.findById(_id, modelView).lean();
await this._executeQuery({ dbQueryPromise, isAccessPermitted: true })
}
_executeQuery basically gets the result of the promise if the user has access.
private _executeQuery = async (props: {
isAccessPermitted: boolean;
dbQueryPromise: Promise<any>;
}): Promise<any> => {
const { isAccessPermitted, dbQueryPromise } = props;
if (!isAccessPermitted) {
throw new Error('Access denied.');
}
console.log(dbQueryPromise, 'promise'); // without this line, dbQueryResult would be null...
const dbQueryResult = await dbQueryPromise;
return dbQueryResult;
};
After some more testing, I found out that the first API call works but any calls after that returns null...
EDIT:
this._dbModel is some mongoose schema. For example,
const dbSchema= new Schema({
name: String,
});
const dbModel = mongoose.model('DbSchema', dbSchema);
Try replacing your dbQueryPromise as follows:
const dbQueryPromise = this._dbModel.findById(_id, modelView).lean().exec();
Mongoose queries do not get executed unless you pass a callBack function or use exec()
For anyone else having similar problems, here's how I solved it:
I changed
const dbQueryResult = await dbQueryPromise;
to
const dbQueryResult = await dbQueryPromise.then((doc) => {
return doc;
});
I'm using Mockingoose to mock my mongoose calls when running tests with Jest. I tried this but I get an error
mockingoose.Account.toReturn(
["593cebebebe11c1b06efff0372","593cebebebe11c1b06efff0373"],
"distinct"
);
Error:
ObjectParameterError: Parameter "obj" to Document() must be an object, got 593cebebebe11c1b06efff0372
So then I try passing it an array of document objects but it just returns the documents. How di I get it to return just an array or strings?
Here's the code inside the function I'm testing:
const accountIDs = await Account.find({
userID: "test",
lastLoginAttemptSuccessful: true
}).distinct("_id");
I'm open to other ways of mocking my mongoose calls if someone knows of a better way. Thanks!
You can't.
My bad. I looked into mockingoose implementation and realized, it kind of "supports" distinct by implementing a mock, but it actually returned just the given documents, as for the other operations.
Opened a pull request for this issue and added a test, so you're example should be valid and working.
I think the answer is to not use mockingoose. You can do it pretty easily with jest alone.
You can use jest.spyOn() and then mockImplementation() to mock the first call like find() and update(). Here's an example of findOneAndUpdate() where we're checking to make sure the correct object is passed:
// TESTING:
// await Timeline.findOneAndUpdate(query, obj);
//
const Timeline = require("./models/user.timeline");
...
const TimelineFindOneAndUpdateMock = jest.spyOn(Timeline, "findOneAndUpdate");
const TimelineFindOneAndUpdate = jest.fn((query, obj) => {
expect(obj.sendDateHasPassed).toBeFalsy();
expect(moment(obj.sendDate).format()).toBe(moment("2018-11-05T23:00:00.000Z").format());
});
TimelineFindOneAndUpdateMock.mockImplementation(TimelineFindOneAndUpdate);
If you want to mock a chained function you can have it return an object with the next chained function you want to call. Here's an example of how to mock a chained distinct() call.
// TESTING:
// let accountIDs = await Account.find(query).distinct("_id");
//
// WILL RETURN:
// ["124512341234","124512341234","124512341234"]
//
const Account = require("./models/user.account");
...
const AccountFindMock = jest.spyOn(Account, "find");
const AccountFindDistinctResult = ["124512341234","124512341234","124512341234"];
const AccountFindDistinctResult = jest.fn(() => AccountFindDistinctResult);
const AccountFindResult = {
distinct: AccountFindDistinct
};
const AccountFind = jest.fn(() => AccountFindResult);
AccountFindMock.mockImplementation(AccountFind);
And after your test runs, if you want to check how many times a function is called like how many times distinct() was called you can add this:
expect(AccountFindDistinct).toHaveBeenCalledTimes(0);
Straight and simple, I have the following function, using Google Cloud Datastore Node.js API:
fetchAll(query, result=[], queryCursor=null) {
this.debug(`datastoreService.fetchAll, queryCursor=${queryCursor}`);
if (queryCursor !== null) {
query.start(queryCursor);
}
return this.datastore.runQuery(query)
.then( (results) => {
result=result.concat(results[0]);
if (results[1].moreResults === _datastore.NO_MORE_RESULTS) {
return result;
} else {
this.debug(`results[1] = `, results[1]);
this.debug(`fetch next with queryCursor=${results[1].endCursor}`);
return this.fetchAll(query, result, results[1].endCursor);
}
});
}
The Datastore API object is in the variable this.datastore;
The goal of this function is to fetch all results for a given query, notwithstanding any limits on the number of items returned per single runQuery call.
I have not yet found out about any definite hard limits imposed by the Datastore API on this, and the documentation seems somewhat opaque on this point, but I only noticed that I always get
results[1] = { moreResults: 'MORE_RESULTS_AFTER_LIMIT' },
indicating that there are still more results to be fetched, and the results[1].endCursor remains stuck on constant value that is passed on again on each iteration.
So, given some simple query that I plug into this function, I just go on running the query iteratively, setting the query start cursor (by doing query.start(queryCursor);) to the endCursor obtained in the result of the previous query. And my hope is, obviously, to obtain the next bunch of results on each successive query in this iteration. But I always get the same value for results[1].endCursor. My question is: Why?
Conceptually, I cannot see a difference to this example given in the Google Documentation:
// By default, google-cloud-node will automatically paginate through all of
// the results that match a query. However, this sample implements manual
// pagination using limits and cursor tokens.
function runPageQuery (pageCursor) {
let query = datastore.createQuery('Task')
.limit(pageSize);
if (pageCursor) {
query = query.start(pageCursor);
}
return datastore.runQuery(query)
.then((results) => {
const entities = results[0];
const info = results[1];
if (info.moreResults !== Datastore.NO_MORE_RESULTS) {
// If there are more results to retrieve, the end cursor is
// automatically set on `info`. To get this value directly, access
// the `endCursor` property.
return runPageQuery(info.endCursor)
.then((results) => {
// Concatenate entities
results[0] = entities.concat(results[0]);
return results;
});
}
return [entities, info];
});
}
(except for the fact, that I don't specify a limit on the size of the query result by myself, which I have also tried, by setting it to 1000, which does not change anything.)
Why does my code run into this infinite loop, stuck on each step at the same "endCursor"? And how do I correct this?
Also, what is the hard limit on the number of results obtained per call of datastore.runQuery()? I have not found this information in the Google Datastore documentation thus far.
Thanks.
Looking at the API documentation for the Node.js client library for Datastore there is a section on that page titled "Paginating Records" that may help you. Here's a direct copy of the code snippet from the section:
var express = require('express');
var app = express();
var NUM_RESULTS_PER_PAGE = 15;
app.get('/contacts', function(req, res) {
var query = datastore.createQuery('Contacts')
.limit(NUM_RESULTS_PER_PAGE);
if (req.query.nextPageCursor) {
query.start(req.query.nextPageCursor);
}
datastore.runQuery(query, function(err, entities, info) {
if (err) {
// Error handling omitted.
return;
}
// Respond to the front end with the contacts and the cursoring token
// from the query we just ran.
var frontEndResponse = {
contacts: entities
};
// Check if more results may exist.
if (info.moreResults !== datastore.NO_MORE_RESULTS) {
frontEndResponse.nextPageCursor = info.endCursor;
}
res.render('contacts', frontEndResponse);
});
});
Maybe you can try using one of the other syntax options (instead of Promises). The runQuery method can take a callback function as an argument, and that callback's parameters include explicit references to the entities array and the info object (which has the endCursor as a property).
And there are limits and quotas imposed on calls to the Datastore API as well. Here are links to official documentation that address them in detail:
Limits
Quotas
I have the following queries, which starts with the GetById method firing up, once that fires up and extracts data from another document, it saves into the race document.
I want to be able to cache the data after I save it for ten minutes. I have taken a look at cacheman library and not sure if it is the right tool for the job. what would be the best way to approach this ?
getById: function(opts,callback) {
var id = opts.action;
var raceData = { };
var self = this;
this.getService().findById(id,function(err,resp) {
if(err)
callback(null);
else {
raceData = resp;
self.getService().getPositions(id, function(err,positions) {
self.savePositions(positions,raceData,callback);
});
}
});
},
savePositions: function(positions,raceData,callback) {
var race = [];
_.each(positions,function(item) {
_.each(item.position,function(el) {
race.push(el);
});
});
raceData.positions = race;
this.getService().modelClass.update({'_id' : raceData._id },{ 'positions' : raceData.positions },callback(raceData));
}
I have recently coded and published a module called Monc. You could find the source code over here. You could find several useful methods to store, delete and retrieve data stored into the memory.
You may use it to cache Mongoose queries using simple nesting as
test.find({}).lean().cache().exec(function(err, docs) {
//docs are fetched into the cache.
});
Otherwise you may need to take a look at the core of Mongoose and override the prototype in order to provide a way to use cacheman as you original suggested.
Create a node module and force it to extend Mongoose as:
monc.hellocache(mongoose, {});
Inside your module you should extend the Mongoose.Query.prototype
exports.hellocache = module.exports.hellocache = function(mongoose, options, Aggregate) {
//require cacheman
var CachemanMemory = require('cacheman-memory');
var cache = new CachemanMemory();
var m = mongoose;
m.execAlter = function(caller, args) {
//do your stuff here
}
m.Query.prototype.exec = function(arg1, arg2) {
return m.execAlter.call(this, 'exec', arguments);
};
})
Take a look at Monc's source code as it may be a good reference on how you may extend and chain Mongoose methods
I will explain with npm redis package which stores key/value pairs in the cache server. keys are queries and redis stores only strings.
we have to make sure that keys are unique and consistent. So key value should store query and also name of the model that you are applying the query.
when you query, inside the mongoose library, there is
function Query(conditions, options, model, collection) {} //constructor function
responsible for query. inside this constructor,
Query.prototype.exec = function exec(op, callback) {}
this function is responsible executing the queries. so we have to manipulate this function and have it execute those tasks:
first check if we have any cached data related to the query
if yes respond to request right away and return
if no we need to respond to request and update our cache and then respond
const redis = require("client");
const redisUrl = "redis://127.0.0.1:6379";
const client = redis.createClient(redisUrl);
const util = require("util");
//client.get does not return promise
client.get = util.promisify(client.get);
const exec = mongoose.Query.prototype.exec;
//mongoose code is written using classical prototype inheritance for setting up objects and classes inside the library.
mongoose.Query.prototype.exec = async function() {
//crate a unique and consistent key
const key = JSON.stringify(
Object.assign({}, this.getQuery(), {
collection: this.mongooseCollection.name
})
);
//see if we have value for key in redis
const cachedValue = await redis.get(key);
//if we do return that as a mongoose model.
//the exec function expects us to return mongoose documents
if (cachedValue) {
const doc = JSON.parse(cacheValue);
return Array.isArray(doc)
? doc.map(d => new this.model(d))
: new this.model(doc);
}
const result = await exec.apply(this, arguments); //now exec function's original task.
client.set(key, JSON.stringify(result),"EX",6000);//it is saved to cache server make sure capital letters EX and time as seconds
};
if we store values as array of objects we need to make sure that each object is individullay converted to mongoose document.
this.model is a method inside the Query constructor and converts object to a mongoose document.
note that if you are storing nested values instead of client.get and client.set, use client.hset and client.hget
Now we monkey patched
Query.prototype.exec
so you do not need to export this function. wherever you have a query operation inside your code, mongoose will execute above code