I'm trying to figure out how to clear a bunch of keys in redis from the cache in the most efficient manner possible.
In this example I'm getting a string array of userIds. Each user has a watchlist of auctionIds in their user document inside mongodb
"watchlist": [
"5eb81dbaecb87a4060c3d5a7",
"5eb81e24ecb87a4060c3d5a8",
"5eb8302ebae779466c97c1d9"
],
It's an auction site so if a user places a bid, then I need the watchlists that have the listing inside that user's [string] watchlist to clear the cache for that key const watchlistKey = `watchlist-${req.query.userIdBidder}`;
but in production that could be 100s of ids. Who knows.
I've been using this library for other pattern delete examples: https://www.npmjs.com/package/redis-delete-wildcard?activeTab=readme but I don't think I can apply it to this use case. What will be the most cost effective way to clear the cache using string array as input data?
User.find({
watchlist: { $in: [req.query.listingId] },
}).select("_id").then((res) => {
console.log(res);
const watchlistKey = `watchlist-${req.query.userIdBidder}`;
client.delwild('main-listings-no-filter-page=*', function (error, numberDeletedKeys) {
console.log("Page Didn't Match During Bid. All Listings Pages have been removed from cache.");
console.log(numberDeletedKeys);
});
})
Console Output
[ { _id: 5eb8196eecb87a4060c3d5a5 },
{ _id: 5eb82d0c71b0a230b853bd6f } ]
Related
What I was hoping to do was store an array of objects using RedisJSON very simply and then query that array.
I have something similar to this:
const data = [
{
_id: '63e7d1d85ad7e2f69df8ed6e',
artist: {
genre: 'rock',
},
},
{
_id: '63e7d1d85ad7e2f69df8ed6f',
artist: {
genre: 'metal',
},
},
{
_id: '63e7d1d85ad7e2f69df8ed6g',
artist: {
genre: 'rock',
},
},
]
then I can easily store and retrieve this:
await redisClient.json.set(cacheKey, '$', data)
await redisClient.json.get(cacheKey)
works great. but now I want to also query this data, I've tried creating an index as below:
await redisClient.ft.create(
`idx:gigs`,
{
'$.[0].artist.genre': {
type: SchemaFieldTypes.TEXT,
AS: 'genre',
},
},
{
ON: 'JSON',
PREFIX: 'GIGS',
}
)
and when I try and search this index what I expect is it to return the 2 documents with the correct search filter, but instead it always returns the entire array:
const searchResult = await redisClient.ft.search(`idx:gigs`, '#genre:(rock)')
produces:
{
total: 1,
documents: [
{ id: 'cacheKey', value: [Array] }
]
}
I can't quite work out at which level I'm getting this wrong, but any help would be greatly appreciated.
Is it possible to store an array of objects and then search the nested objects for nested values with RedisJSON?
The Search capability in Redis stack treats each key containing a JSON document as a separate search index entry. I think what you are doing is perhaps storing your whole array of documents in a single Redis key, which means any matches will return the document at that key which contains all of your data.
I would suggest that you store each object in your data array as its own key in Redis. Make sure that these will be indexed by using the GIGS prefix in the key name, for example GIGS:63e7d1d85ad7e2f69df8ed6e and GIGS:63e7d1d85ad7e2f69df8ed6f.
You'd want to change your index definition to account for each document being an object too so it would look something like this:
await redisClient.ft.create(
`idx:gigs`,
{
'$.artist.genre': {
type: SchemaFieldTypes.TEXT,
AS: 'genre',
},
},
{
ON: 'JSON',
PREFIX: 'GIGS:',
}
)
Note I also updated your PREFIX to be GIGS: not GIGS - this isn't strictly necessary, but does stop your index from accidentally looking at other keys in Redis whose name begins GIGS<whatever other characters>.
I want to be able to update an array of objects where each object has a new unique value assigned to it.
Here is a simplified example of what I'm doing. items is an array of my collection items.
let items = [{_id: '903040349304', number: 55}, {_id: '12341244', number: 1166}, {_id: '667554', number: 51115}]
I want to assign a new number to each item, and then update it in collection:
items = items.map(item => {
item.number = randomInt(0, 1000000);
return item;
})
What would be the best way to update the collection at once? I know that I could do it in forEach instead of map, how ever this seems as a dirty way of doing it, as it won't do the bulk update.
items.forEach(async (item) => {
await this.itemModel.update({_id: item._id}, {number: randomInt(0, 1000000)})
});
I've checked the updateMany as well but my understanding of it is that it's only used to update the documents with a same new value - not like in my case, that every document has a new unique value assigned to it.
After a bit of thinking, I came up with this solution using bulkWrite.
const updateQueries = [];
items.forEach(async (item) => {
updateQueries.push({
updateOne: {
filter: { _id: item._id },
update: { number: item.number },
},
});
});
await this.itemModel.bulkWrite(updateQueries);
About bulkWrite
Sends multiple insertOne, updateOne, updateMany, replaceOne,
deleteOne, and/or deleteMany operations to the MongoDB server in one
command. This is faster than sending multiple independent operations
(like) if you use create()) because with bulkWrite() there is only one
round trip to MongoDB.
You can call an aggregate() to instantly update them without needing to pull them first:
Step1: get a random number with mongoDb build in $rand option which returns a number between 0 and 1
Step2: $multiply this number by 1000000 since that is what you defined ;)
Step3: use another $set with $floor to remove the decimal portion
YourModel.aggregate([
{
'$set': {
'value': {
'$multiply': [
{
'$rand': {}
}, 1000000
]
}
}
}, {
'$set': {
'value': {
'$floor': '$value'
}
}
}
])
Here a picture of how that looks in mongo Compass as a proof of it working:
I have a Mongoose abTest document that has two fields:
status. This is a string enum and can be of type active, inactive or draft.
validCountryCodes. This is an array of strings enums (GB, EU, AU etc). By default, it will be empty.
In the DB, at any one time, I only want there to be one active abTest for each validCountryCode so I'm performing some validation prior to creating or editing a new abTest.
To do this, I've written a function that attempts to count the number of documents that have a status of active and that contain one of the countryCodes.
The function will then return if the count is more than one. If so, I will throw a validation error.
if (params.status === 'active') {
const activeTestForCountryExists = await checkIfActiveAbTestForCountry(
validCountryCodes,
);
if (params.activeTestForCountryExists) {
throw new ValidationError({
message: 'There can only be one active test for each country code.',
});
}
}
const abTest = await AbTest.create(params);
checkIfActiveAbTestForCountry() looks like this:
const checkIfActiveAbTestForCountry = async countryCodes => {
const query = {
status: 'active',
};
if (
!countryCodes ||
(Array.isArray(countryCodes) && countryCodes.length === 0)
) {
query.validCountryCodes = {
$eq: [],
};
} else {
query.validCountryCodes = { $in: [countryCodes] };
}
const count = await AbTest.countDocuments(query);
return count > 0;
};
The count query should count not only exact array matches, but for any partial matches.
If in the DB there is an active abTest with a validCountryCodes array of ['GB', 'AU',], the attempting to create a new abTest with ['GB' should fail. As there is already a test with GB as a validCountryCode.
Similarly, if there is a test with a validCountryCodes array of ['AU'], then creating a test with validCountryCodes of ['AU,'NZ'] should also fail.
Neither is enforced right now.
How can I do this? Is this possible write a query that checks for this?
I considered iterating over params.validCountryCodes and counting the docs that include each, but this seems like bad practice.
take a look at this MongoDB documantation.
As I understood what you need is to find out if there is any document that contains at least one of the specified countryCodes and it has active status. then your query should look like this:
{
status: 'active',
$or: [
{ validCountryCodes: countryCodes[0] },
{ validCountryCodes: countryCodes[1] },
// ...
]
}
note that counting documents is not an efficient manner to check if a document exists or not, instead use findOne with only one field being projected.
You are using the correct mongo-query for your requirement. Can you verify the actual queries executed from your application is the same? Check here
{ status: 'active', validCountryCodes: { $in: [ countryCodes ] } }
For eg; below query :
{ status: 'active', validCountryCodes: { $in: ['GB' ] } }
should match document :
{ status: 'active', validCountryCodes: ['AU','GB'] }
This is a Controller in which I'm trying to catch multiple candidates id(ObjectId) and try to store it in the database in the array Candidates. But data is not getting pushed in Candidates column of Array type.
routes.post('/Job/:id',checkAuthenticated,function(req,res){
var candidates=req.body.candidate;
console.log(candidates);
Job.update({_id:req.params.id},{$push:{Appliedby : req.user.username}},{$push:{Candidates:{$each:
candidates}}}
});
Console screens output
[ '5eb257119f2b2f0b4883558b', '5eb2ae1cff3ae7106019ad7e' ] //candidates
you have to do all the update operations ($set, $push, $pull, ...) in one object, and this object should be the second argument passed to the update method after the filter object
{$push:{Appliedby : req.user.username}},{$push:{Candidates:{$each: candidates}}
this will update the Appliedby array only, as the third object in update is reserved for the options (like upsert, new, ....)
you have to do something like that
{ $push: { Appliedby: req.user.username, Candidates: { $each: candidates } } }
then the whole query should be something like that
routes.post('/Job/:id', checkAuthenticated, function (req, res) {
var candidates = req.body.candidate;
console.log(candidates);
Job.update(
{ _id: req.params.id }, // filter part
{ $push: { Appliedby: req.user.username, Candidates: { $each: candidates } } } // update part in one object
)
});
this could do the trick I guess, hope it helps
I have a meteor app prototype that works well, but is very insecure as of now: I needed to display a list of matching users to the currently logged-in user. For starters, I decided to publish all users, limiting the fields to what I would need to filter the user list on the client side.
Meteor.publish('users', function () {
return Meteor.users.find({}, {
fields: {
'profile.picture': 1,
'profile.likes': 1,
'profile.friends': 1,
'profile.type': 1
}
});
});
Then in my router, I would do a request to only show what I wanted on the client side:
Router.map(function() {
this.route('usersList', {
path: '/users',
waitOn: function () {
return Meteor.subscribe('users');
},
data: function () {
var user = Meteor.user();
return {
users: Meteor.users.find({ $and: [
{_id: {$ne : user._id}},
{'profile.type': user.profile.interest}
]})
};
}
});
});
In the code above, I query all users who are not the current user and whose type correspond the current user's interest. I also display a certain border on the photos of users who have my user in their "profile.friends" array, using this client helper:
Template.userItem.helpers({
getClass: function(userId) {
var user = Meteor.user();
var lookedup = Meteor.users.findOne({_id: userId});
if ($.inArray(user._id, lookedup.profile.friends) !== -1)
return "yes";
return "no";
}
});
Now this all worked great, but with this setup, every client can query every users and get their type, picture, list of friends and number of likes. If I was in an MVC, this info would only be accessible on server side. So I decided my next iteration to be a security one. I would move my query from the router file to the publications file. That's where trouble began...
Meteor.publish('users', function () {
var user = Meteor.users.findOne({_id: this.userId});
var interest = user.profile.interest;
// retrieve all users, with their friends for now
allUsers = Meteor.users.find({ $and: [
{'_id': {$ne: user._id}},
{'profile.type':interest}
]},
{ fields: {'profile.picture': 1, 'profile.friends': 1}}
);
return allUsers;
});
And in the router:
Router.map(function() {
this.route('usersList', {
path: '/users',
waitOn: function () {
return Meteor.subscribe('users');
},
data: function () {
var user = Meteor.user();
return {users: Meteor.users.find({_id: {$ne : user._id}})};
}
});
});
(note that I still need to exclude the current user from the router query since the current user is always fully published)
This works, but:
the user list does not get updated when I change the user interest and then do a Router.go('usersList'). Only when I refresh the browser, my list is updated according to the user's new interest. No idea why.
this solution still publishes the users' friends in order to display my matching borders. I wish to add a temporary field in my publish query, setting it to "yes" if the user is in the user's friends and "no" otherwise, but... no success so far. I read I could use aggregate to maybe achieve that but haven't managed to so far. Also, aggregate doesn't return a cursor which is what is expected from a publication.
This problem makes me doubt about the praises about meteor being suitable for secure apps... This would be so easy to achieve in Rails or others!
EDIT: As requested, here is the code I have so far for the transition of my "matching" check to the server:
Meteor.publish('users', function () {
var user = Meteor.users.findOne({_id: this.userId});
var interest = user.profile.interest;
// retrieve all users, with their friends for now
allUsers = Meteor.users.find({ $and: [
{'_id': {$ne: user._id}},
{'profile.type':interest}
]},
{ fields: {'profile.picture': 1, 'profile.friends': 1}}
);
// ------------- ADDED ---------------
allUsers.forEach(function (lookedup) {
if (_.contains(lookedup.profile.friends, user._id))
lookedup.profile.relation = "yes";
else
lookedup.profile.relation = "no";
lookedup.profile.friends = undefined;
return lookedup;
});
// ------------- END ---------------
return allUsers;
});
Obviously this code doesn't work at all, since I cannot modify cursor values in a foreach loop. But it gives an idea of what I want to achieve: give a way to the client to know if a friend is matched or not, without giving access to the friend lists of all users to the client. (and also, avoiding having to do one request per each user during display to ask the server if this specific user matches this specific one)
You can add a transform function and modify a cursor docs on the fly
meteor Collection.find