I have a question, is it possible to create a LIKE operator search in Redis? Similar to relational (mysql/oracle) database.
I have complex json:
{"_id" : ObjectId("581c8b8854fdcd1ff8c944e0"),
"Objectcode" : "xxxxx",
"Objecttype" : "xxxx",
"docid" : "581c8b8554fdcd1ff8c93d10",
"description" : "Tags based search .... ",
"metaTags" : [
"tag1",
"tag2",
"tag3",
"tag5",
"tag6",
"tag7",
"tag8",
"tag9",
"tag10"
],
"__v" : 0
}
and i want to search on array of metaTags how can i do it?
Thanks
You can use MATCH commands to search data.
Eg: scan 0 MATCH *11*
Refer: http://redis.io/commands/scan
You can use Redis *SCAN commands http://redis.io/commands/scan, depending of your type of data to filter by a pattern:
SCAN iterates the set of keys in the currently selected Redis database.
SSCAN iterates elements of Sets types.
HSCAN iterates fields of Hash types and their associated values.
ZSCAN iterates elements of Sorted Set types and their associated scores.
Never use KEYS in app code, because it may ruin performance.
The two major nodejs redis client libraries node_redis and ioredis support it, with some syntax sugar:
const keys = [];
const redis = new Redis(); // ioredis
redis.mset('foo1', 1, 'foo2', 1, 'foo3', 1, 'foo4', 1, 'foo10', 1, () => {
const stream = redis.scanStream();
stream.on('data', data => {
keys = keys.concat(data);
});
stream.on('end', () => {
assert.equal(keys.sort(), ['foo1', 'foo10', 'foo2', 'foo3', 'foo4']);
});
});
Related
I am trying to build a "number of visitors" collection in mongoDb using Node.JS backend of my website. The frontend sends the following info to Node.JS backend as JSON.
isUniqueVisitor - 1 if yes, 0 if no
country - standard country code - "JP", "IN", "UK", etc
My database looks like following
{
"today": 2019-06-07,
"uniqueVisitors": {
"count": 230,
"countries": {
"JP": 102,
"IN": 88,
"UK": 30
}
}
}
It works well if I use $inc with fixed values
Eg. $inc: {count: 1} // for string/integers keys
Eg. $inc: {"uniqueVisitors.count": 1} // inside quotes to access key of a JSON
Main issue:
I am not able to access a document name using variable.
Eg. $inc: {`uniqueVisitors.countries[${req.body.country}]`}
This creates an error as backticks can't be used for Mongo.
I tried with
Eg. $inc: {uniqueVisitors["countries"][req.body.country]}
But even this creates error.
I followed the web and found that mongo $set using variables can be realized by passing the required JSON directly to $set. Hence I resorted to code it the following way.
mongoClient.connect(mongoURL, async function (err, db) {
if (err) throw err;
console.log("Database connected");
// Identifying my document with today's date
var myQuery = {
date: getTodayDate()
};
// Defining the JSON to be passed to uniqueVisitors $inc
var uniqueVisitorsInc = {
"uniqueVisitors": {
"count": 0,
"countries": {}
}
};
// Populating the JSON to be passed to uniqueVisitors $inc => essentially asking to increase count by 1 and increase that country's count by 1
uniqueVisitorsInc["uniqueVisitors"]["count"] = 1;
uniqueVisitorsInc["uniqueVisitors"]["countries"][myData.country] = 1;
var newValues = {
$inc: uniqueVisitorsInc
};
await db.collection("visitorStats").update(myQuery, newValues, {upsert: true});
db.close();
});
The above method worked well on editor but threw the following runtime error:
$inc requires numerical values
Basically asking me to pass values to $inc in {var1: 1, var2: 5} pattern.
Please help me bypass this weird situation.
I know I can do a two step process where I read the values first, increment in variable and $set it in Mongo.
But does anyone know how to overcome this situation using $inc?
If this update were hardcoded to update "JP" only, it'd need to look like:
$inc: { "uniqueVisitors.country.JP": 1 }
So you were almost there with the backtick method but change the syntax a bit and keep the : 1 part like so:
$inc: { [`uniqueVisitors.country.${req.body.country}`]: 1 }
I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};
In the following example, assume the document is in the db.people collection.
How to remove the 3rd element of the interests array by it's index?
{
"_id" : ObjectId("4d1cb5de451600000000497a"),
"name" : "dannie",
"interests" : [
"guitar",
"programming",
"gadgets",
"reading"
]
}
This is my current solution:
var interests = db.people.findOne({"name":"dannie"}).interests;
interests.splice(2,1)
db.people.update({"name":"dannie"}, {"$set" : {"interests" : interests}});
Is there a more direct way?
There is no straight way of pulling/removing by array index. In fact, this is an open issue http://jira.mongodb.org/browse/SERVER-1014 , you may vote for it.
The workaround is using $unset and then $pull:
db.lists.update({}, {$unset : {"interests.3" : 1 }})
db.lists.update({}, {$pull : {"interests" : null}})
Update: as mentioned in some of the comments this approach is not atomic and can cause some race conditions if other clients read and/or write between the two operations. If we need the operation to be atomic, we could:
Read the document from the database
Update the document and remove the item in the array
Replace the document in the database. To ensure the document has not changed since we read it, we can use the update if current pattern described in the mongo docs
You can use $pull modifier of update operation for removing a particular element in an array. In case you provided a query will look like this:
db.people.update({"name":"dannie"}, {'$pull': {"interests": "guitar"}})
Also, you may consider using $pullAll for removing all occurrences. More about this on the official documentation page - http://www.mongodb.org/display/DOCS/Updating#Updating-%24pull
This doesn't use index as a criteria for removing an element, but still might help in cases similar to yours. IMO, using indexes for addressing elements inside an array is not very reliable since mongodb isn't consistent on an elements order as fas as I know.
in Mongodb 4.2 you can do this:
db.example.update({}, [
{$set: {field: {
$concatArrays: [
{$slice: ["$field", P]},
{$slice: ["$field", {$add: [1, P]}, {$size: "$field"}]}
]
}}}
]);
P is the index of element you want to remove from array.
If you want to remove from P till end:
db.example.update({}, [
{ $set: { field: { $slice: ["$field", 1] } } },
]);
Starting in Mongo 4.4, the $function aggregation operator allows applying a custom javascript function to implement behaviour not supported by the MongoDB Query Language.
For instance, in order to update an array by removing an element at a given index:
// { "name": "dannie", "interests": ["guitar", "programming", "gadgets", "reading"] }
db.collection.update(
{ "name": "dannie" },
[{ $set:
{ "interests":
{ $function: {
body: function(interests) { interests.splice(2, 1); return interests; },
args: ["$interests"],
lang: "js"
}}
}
}]
)
// { "name": "dannie", "interests": ["guitar", "programming", "reading"] }
$function takes 3 parameters:
body, which is the function to apply, whose parameter is the array to modify. The function here simply consists in using splice to remove 1 element at index 2.
args, which contains the fields from the record that the body function takes as parameter. In our case "$interests".
lang, which is the language in which the body function is written. Only js is currently available.
Rather than using the unset (as in the accepted answer), I solve this by setting the field to a unique value (i.e. not NULL) and then immediately pulling that value. A little safer from an asynch perspective. Here is the code:
var update = {};
var key = "ToBePulled_"+ new Date().toString();
update['feedback.'+index] = key;
Venues.update(venueId, {$set: update});
return Venues.update(venueId, {$pull: {feedback: key}});
Hopefully mongo will address this, perhaps by extending the $position modifier to support $pull as well as $push.
I would recommend using a GUID (I tend to use ObjectID) field, or an auto-incrementing field for each sub-document in the array.
With this GUID it is easy to issue a $pull and be sure that the correct one will be pulled. Same goes for other array operations.
For people who are searching an answer using mongoose with nodejs. This is how I do it.
exports.deletePregunta = function (req, res) {
let codTest = req.params.tCodigo;
let indexPregunta = req.body.pregunta; // the index that come from frontend
let inPregunta = `tPreguntas.0.pregunta.${indexPregunta}`; // my field in my db
let inOpciones = `tPreguntas.0.opciones.${indexPregunta}`; // my other field in my db
let inTipo = `tPreguntas.0.tipo.${indexPregunta}`; // my other field in my db
Test.findOneAndUpdate({ tCodigo: codTest },
{
'$unset': {
[inPregunta]: 1, // put the field with []
[inOpciones]: 1,
[inTipo]: 1
}
}).then(()=>{
Test.findOneAndUpdate({ tCodigo: codTest }, {
'$pull': {
'tPreguntas.0.pregunta': null,
'tPreguntas.0.opciones': null,
'tPreguntas.0.tipo': null
}
}).then(testModificado => {
if (!testModificado) {
res.status(404).send({ accion: 'deletePregunta', message: 'No se ha podido borrar esa pregunta ' });
} else {
res.status(200).send({ accion: 'deletePregunta', message: 'Pregunta borrada correctamente' });
}
})}).catch(err => { res.status(500).send({ accion: 'deletePregunta', message: 'error en la base de datos ' + err }); });
}
I can rewrite this answer if it dont understand very well, but I think is okay.
Hope this help you, I lost a lot of time facing this issue.
It is little bit late but some may find it useful who are using robo3t-
db.getCollection('people').update(
{"name":"dannie"},
{ $pull:
{
interests: "guitar" // you can change value to
}
},
{ multi: true }
);
If you have values something like -
property: [
{
"key" : "key1",
"value" : "value 1"
},
{
"key" : "key2",
"value" : "value 2"
},
{
"key" : "key3",
"value" : "value 3"
}
]
and you want to delete a record where the key is key3 then you can use something -
db.getCollection('people').update(
{"name":"dannie"},
{ $pull:
{
property: { key: "key3"} // you can change value to
}
},
{ multi: true }
);
The same goes for the nested property.
this can be done using $pop operator,
db.getCollection('collection_name').updateOne( {}, {$pop: {"path_to_array_object":1}})
I have the following code:
var db = require("redis");
var dbclient1 = db.createClient();
dbclient1.zadd("myprivateset", 3, {"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"})
dbclient1.zadd("myprivateset", 2, {"guid":"abab-baba3", "data-persistent":"xxxx", "size":"20"})
dbclient1.zrangebyscore("myprivateset", 1, 4)
dbclient1.hgetall("myprivateset", function(err, rep){
console.log(rep);
});
I wish to store my objects (in JSON format) in a sorted set, which determine by the score (3 & 2 in our case).
For some reason, when I print this table (rep), I get undefined.
What I do wrong?
Issue 1 -- sorted set keys
Try stringifying the JSON you are using as the keys of your sorted set. For example,
dbclient1.zadd("myprivateset", 3, {"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"})
needs to be:
dbclient1.zadd("myprivateset", 3, JSON.stringify({"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"}))
Without stringifying the keys, every zadd will use the key [object Object] overwriting each time. That is, you'll only ever have one item in your sorted set that is unidentifiable (other than by [object Object]).
Issue 2 -- fetching data
Also, hgetall is not the redis command to use for retrieving data in a redis sorted set. You'll want to focus on sorted set specific commands. A list of redis commands are listed here: http://redis.io/commands
My two cents, building on comments by #leonid-beschastny and #cpentra1. I recommend using redis.multi(). It allows for several calls in a batch, and as you can see in the example, as soon as the three elements are added to the ordered set, we can perform a zrangebyscore in the same multi batch and get the expected results. Instructions can be created dynamically. The replies array when multi.exec() is invoked returns the results for each of the multi operations, in order.
var db = require("redis");
var dbclient1 = db.createClient();
var multi = dbclient1.multi();
// We use JSON.stringify() as suggested by #cpentra1
multi.zadd("myprivateset", 3, JSON.stringify({"guid":"abab-baba", "data-persistent":"xxxx", "size":"20"}));
multi.zadd("myprivateset", 2, JSON.stringify({"guid":"abab-baba3", "data-persistent":"xxxx", "size":"20"}));
multi.zadd("myprivateset", 2, JSON.stringify({"guid":"abab-dafa3", "data-persistent":"yyyy", "size":"21"}));
multi.zrangebyscore("myprivateset", 1, 4);
multi.zcard("myprivateset"); // The total number of elements in the set
multi.exec(function(err, replies) {
console.log(replies)
// Will output something like:
// [ 1,
// 1,
// 1,
// [ '{"guid":"abab-baba3","data-persistent":"xxxx","size":"20"}',
// '{"guid":"abab-dafa3","data-persistent":"yyyy","size":"21"}',
// '{"guid":"abab-baba","data-persistent":"xxxx","size":"20"}' ],
// 3 ]
});
Note: if you run the same example twice, instead of 1s in the first three elements of the replies array, you'll get 0s as the same member with the same score cannot be added twice.
Seems like with Redis 6.2 the format has changed to and object with score and value attributes or an array of those, like this:
async function sortedSet() {
let client;
try {
client = createClient();
client.on("error", (err) => console.log("Redis Client Error", err));
await client.connect();
console.log("connected");
await client.zAdd("user:0:followers", [{score: "1", value: "John"}, {score: "2", value: "Other John"}]);
console.log("sorted set added");
} finally {
await client.quit();
}
}
sortedSet("duto_guerra", "with hashes");
In case you are wondering, I figured this out by reading the source code for node-redis ZADD
I am using node and mongo with the native client.
I would like to add pagination to my application.
To get pagination, I need my responses to always return count alongside data
I would like to get something like:
{
count : 111,
data : [ { 'a' : 'only first item was requested' } ]
}
I can do this in mongo
> var guy = db.users.find({}).limit(1)
> guy.count()
11
> guy.toArray()
[
{
"_id" : ObjectId("5381a7c004fb02b10b557ee3"),
"email" : "myEmail#guy.com",
"fullName" : "guy mograbi",
"isAdmin" : true,
"password" : "fe20a1f102f49ce45d1170503b4761ef277bb6f",
"username" : "guy",
"validated" : true
}
]
but when I do the same with nodejs mongo client I get errors.
var cursor = collection.find().limit(1);
cursor.toArray( function(){ .. my callback .. });
cursor.count();
It seems that
count is not defined on cursor
that once I applied toArray on cursor, I cannot use the cursor again
How, using nodejs, can I accomplish the same thing I can with mongo directly?
As others have said, if you want to have a total count of the items and then the data you will need to have two queries, there is no other way. Why are you concerned with creating two queries?