I'm trying to implement an automatic caching system where any normal call to mongodb using mongoose uses redis as a middleman to cache it. And the problem is, I've gotten storing it, fetching it, but I can't use a callback. I've dug forever and can't find out how to call it. I've found some chunks ( How mongoose calls ) but I can't find the piece that is the actual callback I specified. Through some investigation, I found that returning doesn't do anything, and the only way I could figure out how to callback was using mongooseExec.apply(this, arguments) but that doesn't help when I need to call it once fetching because calling .apply also fetches from mongodb which I don't want if it's already cached. Here's the reduced code:
const mongooseExec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async function () {
const key = await JSON.stringify({
...this.getQuery(),
collection: this.mongooseCollection.name,
op: this.op,
options: this.options,
});
let cached = await self.redisClient.get(key);
let result;
if (cached) {
console.log("CACHED");
return await JSON.parse(cached)
} else {
result = await mongooseExec.apply(this, arguments);
}
};
The callback is passed as an argument to the exec function. For example, in find:
this.exec(callback);
Your exec function needs to accept a callback function as its first argument.
Related
i've been trying to use cache with redis in my nodejs mongodb application but i didn't find any tutorial on how to do that except few which are using logic that seems to not be explained in mongoose documentation
const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async
function (){
// our caching logic
return await exec.apply(this, arguments);
}
where does arguments come from ? because it seems to be undefined yet it is used
mongoose.Query.prototype.exec = async function(){
const collectionName = this.mongooseCollection.name;
if(this.cacheMe){
// You can't insert json straight to redis needs to be a string
const key = JSON.stringify({...this.getOptions(),
collectionName : collectionName, op : this.op});
const cachedResults = await redis.HGET(collectionName,key);
// this.op is the method which in our case is "find"
if (cachedResults){
// if you found cached results return it;
const result = JSON.parse(cachedResults);
return result;
}
//else
// get results from Database then cache it
const result = await exec.apply(this,arguments);
redis.HSET(collectionName, key, JSON.stringify(result) , "EX",this.cacheTime);
//Blogs - > {op: "find" , ... the original query} -> result we got from database
return result;
}
clearCachedData(collectionName, this.op);
return exec.apply(this,arguments);
}
and what is this.getOptions()?
i would be thankful if any one can explain me this logic, because i did not find any help in the documentations nor internet blogs and articls
The arguments object is a local variable that is available inside every function in JavaScript and contains the values of the arguments passed to the function.
this.getOptions() is the local method that returns the options to the query.
// A key example for mongoose Redis integration
const key = JSON.stringify({
collectionName: this.mongooseCollection.name,
op: this.op,
options: this.getOptions(),
filter: this.getFilter(),
projection: this.projection(),
populatedPaths: this.getPopulatedPaths(),
});
There are a lot of similar packages on NPM, but I highly recommend standard mongoose and redis ones to get up and running. I assumed your initialization point was similar to this post. This can also be a relevant source.
My goal is to create an abstracted POST function for an express running on Node similar to Django's inbuilt REST methods. As a part of my abstracted POST function I'm checking the database (mongo) for valid foreign keys, duplicates, etc..., which is where we are in the process here. The function below is the one that actually makes the first calls to mongo to check that the incoming foreign keys actually exist in the tables/collections that they should exist in.
In short, the inbuilt response functionality inside the native .map function seems to be causing an early return from the called/subsidiary functions, and yet still continuing on inside the called/subsidiary functions after the early return happens.
Here is the code:
const db = require(`../../models`)
const findInDB_by_id = async params => {
console.log(`querying db`)
const records = await Promise.all(Object.keys(params).map(function(table){
return db[table].find({
_id: {
$in: params[table]._ids
}
})
}))
console.log('done querying the db')
// do stuff with records
return records
}
// call await findIndDB_by_id and do other stuff
// eventually return the response object
And here are the server logs
querying db
POST <route> <status code> 49.810 ms - 9 //<- and this is the appropriate response
done querying the db
... other stuff
When I the function is modified so that the map function doesn't return anything, (a) it doesn't query the database, and (b) doesn't return the express response object early. So by modifying this:
const records = await Promise.all(Object.keys(params).map(function(table){
return db[table].find({ // going to delete the `return` command here
_id: {
$in: params[table]._ids
}
})
}))
to this
const records = await Promise.all(Object.keys(params).map(function(table){
db[table].find({ // not returning this out of map
_id: {
$in: params[table]._ids
}
})
}))
the server logs change to:
querying db
done querying the db
... other stuff
POST <route> <status code> 49.810 ms - 9 // <-appropriate reponse
But then I'm not actually building my query, so the query response is empty. I'm experiencing this behavior with the anonymous map function being an arrow function of this format .map(table => (...)) as well.
Any ideas what's going on, why, or suggestions?
Your first version of your code is working as expected and the logs are as expected.
All async function return a promise. So findInDB_by_id() will always return a promise. In fact, they return a promise as soon as the first await inside the function is encountered as the calling code continues to run. The calling code itself needs to use await or .then() to get the resolved value from that promise.
Then, some time later, when you do a return someValue in that function, then someValue becomes the resolved value of that promise and that promise will resolve, allowing the calling code to be notified via await or .then() that the final result is now ready.
await only suspends execution of the async function that it is in. It does not cause the caller to be blocked at all. The caller still has to to deal with a promise returned from the async function.
The second version of your code just runs the db queries open loop with no control and no ability to collect results or process errors. This is often referred to as "fire and forget". The rest of your code continues to execute without regard for anything happening in those db queries. It's highly unlikely that the second version of code is correct. While there are a very few situations where "fire and forget" is appropriate, I will always want to at least log errors in such a situation, but since it appears you want results, this cannot be correct
You should try something like as when it encounter first async function it start executing rest of the code
let promiseArr = []
Object.keys(params).map(function(table){
promiseArr.push(db[table].find({
_id: {
$in: params[table]._ids
}
}))
})
let [records] = await Promise.all(promiseArr)
and if you still want to use map approach
await Promise.all(Object.keys(params).map(async function(table){
return await db[table].find({
_id: {
$in: params[table]._ids
}
})
})
)
I want to force Node to wait for the promise to complete (either by failure or success). So far, I'm not seeing what I wanted. I'm trying to wait for two web services to complete before merging, and especially before the function ends.
I've tried these two approaches below. In both cases, Node refuses to wait.
Approach 1:
function getStrategy() {
// this is a web service that takes a few ms to run,
// but so far I haven't seen any evidence that it bothers.
}
function getConfig() {
// Both strategy and jwt are set to Promises
const strategy = getStrategy();
const jwt = getJwt();
const lodash = require('lodash');
var config = {};
try {
Promise.all([strategy,jwt])
.then(data => { config = lodash.merge(config,data)})
} catch(error) {
console.log(' Print error message');
}
return config;
}
Approach 2:
function getStrategy() {
// this is a web service that takes a few ms to run,
// but so far I haven't seen any evidence that it bothers.
}
async function getConfig() {
// Both strategy and jwt are set to Promises
const strategy = getStrategy();
const jwt = getJwt();
const lodash = require('lodash');
var config = {};
try {
var promiseResult = await Promise.all([strategy, jwt]);
const lodash = require('lodash');
config = lodash.merge(config, strategy[0]);
config = lodash.merge(config, jwt);
} catch (reason) {
console.error('------------------------------------------------------------------------------------');
console.error(reason);
console.error("in getConfig(): Could not fetch strategy or jwt");
console.error('------------------------------------------------------------------------------------');
}
return config;
}
I wish approach 2 worked, but it does not. It will not print any console.log statements after the call to Promise.all. So within that function it does wait. Except that it because I told it to "await", I have to make the function async. That allows Node to say, "oh, it's an async function, I can just go off and do something else and completely ignore the await keyword." It does this by returning to the function calling getConfig().
In the first approach, neither the "then" handler, nor any exceptions are thrown. It just impatiently leaves the function and goes back to the caller.
How do I get the thread that calls the getConfig() function to wait for the result. I mean really wait, not like, partially "await". Or, throw an exception and let me handle that. I'm finding that in Node, as soon as something because asynchronous, I have no idea how to get the await, or the then handler to work.
Updated attempt:
I separated the two service calls to individually control each service. I now have
async function getSynchronousStrategy(isVaultAvailable) {
const secretsVaultReader = require('./src/configuration/secretsVaultReader');
const secretsConfigReader = require('./src/configuration/secretsConfigReader');
const strategy = isVaultAvailable
? secretsVaultReader.getStrategy()
: secretsConfigReader.getStrategy();
await strategy;
console.log('## strategy after await=' + strategy);
return strategy; ''
}
...
const strategy = await getSynchronousStrategy(isVaultAvailable);
console.log('### strategy =' + JSON.stringify(strategy));
...
In the above case, Node sees the await strategy, but then prints
"## strategy after await=[object Promise]"
However, the second await seems to work, and it prints the desired strategy. I'm guessing the promise was eventually settled and it was able to print the result. I don't mind the time, I just want it to wait.
Obviously, it did not wait. It
I would suggest simply using await with both the function calls directly, in this manner it will wait for both the functions to return responses before proceeding further.
I'm still somewhat new to working with Node and def new to working asynchronously and with promises.
I have an application that is hitting a REST endpoint, then calling a chain of functions. The end of this chain is calling hgetall and I need to wait until I get the result and pass it back. I'm testing with Postman and I'm getting {} back instead of the id. I can console.log the id, so I know that this is because some of the code isn't waiting for the result of hgetall before continuing.
I'm using await to wait for the result of hgetall, but that's only working for the end of the chain. do I need to do this for the entire chain of functions, or is there a way to have everything wait for the result before continuing on? Here's the last bit of the logic chain:
Note: I've removed some of the logic from the below functions and renamed a few things to make it a bit easier to see the flow and whats going on with this particular issue. So, some of it may look a bit weird.
For this example, it will call GetProfileById().
FindProfile(info) {
var profile;
var profileId = this.GenerateProfileIdkey(info); // Yes, this will always give me the correct key
profile = this.GetProfileById(profileId);
return profile;
}
This checks with the Redis exists, to verify if the key exists, then tries to get the id with that key. I am now aware that the Key() returns true instead of what Redis actually returns, but I'll fix that once I get this current issue resolved.
GetProfileById(profileId) {
if ((this.datastore.Key(profileId) === true) && (profileId != null)) {
logger.info('GetProfileById ==> Profile found. Returning the profile');
return this.datastore.GetId(profileId);
} else {
logger.info(`GetProfileById ==> No profile found with key ${profileId}`)
return false;
}
}
GetId() then calls the data_store to get the id. This is also where I started to use await and async to try and wait for the result to come through before proceeding. This part does wait for the result, but the functions prior to this don't seem to wait for this one to return anything. Also curious why it only returns the key and not the value, but when I print out the result in hgetall I get the key and value?
async GetId(key) {
var result = await this.store.RedisGetId(key);
console.log('PDS ==> Here is the GetId result');
console.log(result); // returns [ 'id' ]
return result;
}
and finally, we have the hgetall call. Again, new to promises and async, so this may not be the best way of handling this or right at all, but it is getting the result and waiting for the result before it returns anything
async RedisGetId(key) {
var returnVal;
var values;
return new Promise((resolve, reject) => {
client.hgetall(key, (err, object) => {
if (err) {
reject(err);
} else {
resolve(Object.keys(object));
console.log(object); // returns {id: 'xxxxxxxxxxxxxx'}
return object;
}
});
});
}
Am I going to need to async every single function that could potentially end up making a Redis call, or is there a way to make the app wait for the Redis call to return something, then continue on?
Short answer is "Yes". In general, if a call makes an asynchronous request and you need to await the answer, you will need to do something to wait for it.
Sometimes, you can get smart and issue multiple calls at once and await all of them in parallel using Promise.all.
However, it looks like in your case your workflow is synchronous, so you will need to await each step individually. This can get ugly, so for redis I typically use something like promisify and make it easier to use native promises with redis. There is even an example on how to do this in the redis docs:
const {promisify} = require('util');
const getAsync = promisify(client.get).bind(client);
...
const fooVal = await getAsync('foo');
Makes your code much nicer.
i tried to add a trigger function on my cloud functions.
When a document changed i want to perform some work. I have already the code for the work to do in another file (i separate my work into files, so it's get easier for me), so i have a function that performs work and when it is finished the callback is called.
This is what my index.js look like:
// requires...
// Express, i also use express for simulating an API
const app = express()
// So there is my Listener
const listenerForChange = functions.firestore
.document('myPath/documentName')
.onWrite((change, context) => {
const document = change.after.exists ? change.after.data() : null;
if (document != null) {
const oldDocument = change.before.data();
const newDocument = change.after.data()
// Here i call my function that is in the worker-listeners.js
listenerWorker.performStuff(newDocument, function() {
return 0
})
}
else {
console.error("Listener for classic was triggered but doc does not exist")
return 0
}
});
const api = functions.https.onRequest(app)
module.exports = {
api,
listenerForChange
}
...
There is my listenerWorker.js :
module.exports = {
performStuff: function(data, complete) {
performStuff(data, complete)
}
};
function performStuff(data, complete) {
// do stuff here ...
complete()
}
Problem is that i always an error in the Firebase console saying : Function returned undefined, expected Promise or value
Even if i do not do anything inside my worker and calling the callback as soon as i can i get the error.
So i understand the functions needs a response, promise or value. But it's like i'm not in the scope anymore but i do not want to return before the work is finished !
Any help ? thank you guys
All asynchronous work that you perform should be represented by a promise. If you have a utility object that does async work, it should return a promise so that the caller can decide what to do next with it. Right now you're just using callbacks, and that's going to make things much more difficult than necessary (because you'll have to "promisify" those callbacks).
Just use promises everywhere and it'll be much more clear how your function should work.