Related
I'm trying to create multiple documents based on a filter: if document not found => create it.
After searching a bit I found that the correct way to do so is by using updateMany and setting upsert: true (docs).
This made somewhat sense from the documentation's example but as I understand it the filter modifier would be used for the newly-created document. As in the example:
try {
db.inspectors.updateMany(
{ "Sector" : { $gt : 4 }, "inspector" : "R. Coltrane" },
{ $set: { "Patrolling" : false } },
{ upsert: true }
);
} catch (e) {
print(e);
}
"Inspector" : "R. Coltrane" would be inserted to the newly-created document.
But what if my setOnInsert modifier contains the same field as the one in the filter?
What my code looks like:
//first find the already-created tags
await tagModel.find({"tagName": tags}).select('tagName -_id').exec()
.then(async (result: Tag[])=>{
//create the new tags
const newTags = tags.map((tag: any)=>new tagModel({tagName: tag}));
//now insert only the new tags, filtering out the already-created tags ("result")
await tagModel.updateMany(
{"tagName": result},
{$setOnInsert: newTags} ,
{upsert: true},
(err:any, res:any)=>{
...
At first, result is an empty Array ([]). What is created in my MongoDB database is a new Tag document, but its tagName is the result object. Meaning, it looks like:
{
"_id": {
"$oid": "61659c92c6267fe11963b236"
},
"tagName": {
"$in": []
}
}
So essentially my question is, what am I suppose to do in this case where my update modifier should replace my filter query? Perhaps it's just something bad in my code that makes the updateMany function to malfunction? Or should I replace it with a different function?
I have documents in a MongoDB as below -
[
{
"_id": "17tegruebfjt73efdci342132",
"name": "Test User1",
"obj": "health=8,type=warrior",
},
{
"_id": "wefewfefh32j3h42kvci342132",
"name": "Test User2",
"obj": "health=6,type=magician",
}
.
.
]
I want to run a query say health>6 and it should return the "Test User1" entry. The obj key is indexed as a text field so I can do {$text:{$search:"health=8"}} to get an exact match but I am trying to incorporate mathematical operators into the search.
I am aware of the $gt and $lt operators, however, it cannot be used in this case as health is not a key of the document. The easiest way out is to make health a key of the document for sure, but I cannot change the document structure due to certain constraints.
Is there anyway this can be achieved? I am aware that mongo supports running javascript code, not sure if that can help in this case.
I don't think it's possible in $text search index, but you can transform your object conditions to an array of objects using an aggregation query,
$split to split obj by "," and it will return an array
$map to iterate loop of the above split result array
$split to split current condition by "=" and it will return an array
$let to declare the variable cond to store the result of the above split result
$first to return the first element from the above split result in k as a key of condition
$last to return the last element from the above split result in v as a value of the condition
now we have ready an array of objects of string conditions:
"objTransform": [
{ "k": "health", "v": "9" },
{ "k": "type", "v": "warrior" }
]
$match condition for key and value to match in the same object using $elemMatch
$unset to remove transform array objTransform, because it's not needed
db.collection.aggregate([
{
$addFields: {
objTransform: {
$map: {
input: { $split: ["$obj", ","] },
in: {
$let: {
vars: {
cond: { $split: ["$$this", "="] }
},
in: {
k: { $first: "$$cond" },
v: { $last: "$$cond" }
}
}
}
}
}
}
},
{
$match: {
objTransform: {
$elemMatch: {
k: "health",
v: { $gt: "8" }
}
}
}
},
{ $unset: "objTransform" }
])
Playground
The second upgraded version of the above aggregation query to do less operation in condition transformation if it's possible to manage in your client-side,
$split to split obj by "," and it will return an array
$map to iterate loop of the above split result array
$split to split current condition by "=" and it will return an array
now we have ready a nested array of string conditions:
"objTransform": [
["type", "warrior"],
["health", "9"]
]
$match condition for key and value to match in the array element using $elemMatch, "0" to match the first position of the array and "1" to match the second position of the array
$unset to remove transform array objTransform, because it's not needed
db.collection.aggregate([
{
$addFields: {
objTransform: {
$map: {
input: { $split: ["$obj", ","] },
in: { $split: ["$$this", "="] }
}
}
}
},
{
$match: {
objTransform: {
$elemMatch: {
"0": "health",
"1": { $gt: "8" }
}
}
}
},
{ $unset: "objTransform" }
])
Playground
Using JavaScript is one way of doing what you want. Below is a find that uses the index on obj by finding documents that have health= text followed by an integer (if you want, you can anchor that with ^ in the regex).
It then uses a JavaScript function to parse out the actual integer after substringing your way past the health= part, doing a parseInt to get the int, and then the comparison operator/value you mentioned in the question.
db.collection.find({
// use the index on obj to potentially speed up the query
"obj":/health=\d+/,
// now apply a function to narrow down and do the math
$where: function() {
var i = this.obj.indexOf("health=") + 7;
var s = this.obj.substring(i);
var m = s.match(/\d+/);
if (m)
return parseInt(m[0]) > 6;
return false;
}
})
You can of course tweak it to your heart's content to use other operators.
NOTE: I'm using the JavaScript regex capability, which may not be supported by MongoDB. I used Mongo-Shell r4.2.6 where it is supported. If that's the case, in the JavaScript, you will have to extract the integer out a different way.
I provided a Mongo Playground to try it out in if you want to tweak it, but you'll get
Invalid query:
Line 3: Javascript regex are not supported. Use "$regex" instead
until you change it to account for the regex issue noted above. Still, if you're using the latest and greatest, this shouldn't be a limitation.
Performance
Disclaimer: This analysis is not rigorous.
I ran two queries against a small collection (a bigger one could possibly have resulted in different results) with Explain Plan in MongoDB Compass. The first query is the one above; the second is the same query, but with the obj filter removed.
and
As you can see the plans are different. The number of documents examined is fewer for the first query, and the first query uses the index.
The execution times are meaningless because the collection is small. The results do seem to square with the documentation, but the documentation seems a little at odds with itself. Here are two excerpts
Use the $where operator to pass either a string containing a JavaScript expression or a full JavaScript function to the query system. The $where provides greater flexibility, but requires that the database processes the JavaScript expression or function for each document in the collection.
and
Using normal non-$where query statements provides the following performance advantages:
MongoDB will evaluate non-$where components of query before $where statements. If the non-$where statements match no documents, MongoDB will not perform any query evaluation using $where.
The non-$where query statements may use an index.
I'm not totally sure what to make of this, TBH. As a general solution it might be useful because it seems you could generate queries that can handle all of your operators.
In the following example, assume the document is in the db.people collection.
How to remove the 3rd element of the interests array by it's index?
{
"_id" : ObjectId("4d1cb5de451600000000497a"),
"name" : "dannie",
"interests" : [
"guitar",
"programming",
"gadgets",
"reading"
]
}
This is my current solution:
var interests = db.people.findOne({"name":"dannie"}).interests;
interests.splice(2,1)
db.people.update({"name":"dannie"}, {"$set" : {"interests" : interests}});
Is there a more direct way?
There is no straight way of pulling/removing by array index. In fact, this is an open issue http://jira.mongodb.org/browse/SERVER-1014 , you may vote for it.
The workaround is using $unset and then $pull:
db.lists.update({}, {$unset : {"interests.3" : 1 }})
db.lists.update({}, {$pull : {"interests" : null}})
Update: as mentioned in some of the comments this approach is not atomic and can cause some race conditions if other clients read and/or write between the two operations. If we need the operation to be atomic, we could:
Read the document from the database
Update the document and remove the item in the array
Replace the document in the database. To ensure the document has not changed since we read it, we can use the update if current pattern described in the mongo docs
You can use $pull modifier of update operation for removing a particular element in an array. In case you provided a query will look like this:
db.people.update({"name":"dannie"}, {'$pull': {"interests": "guitar"}})
Also, you may consider using $pullAll for removing all occurrences. More about this on the official documentation page - http://www.mongodb.org/display/DOCS/Updating#Updating-%24pull
This doesn't use index as a criteria for removing an element, but still might help in cases similar to yours. IMO, using indexes for addressing elements inside an array is not very reliable since mongodb isn't consistent on an elements order as fas as I know.
in Mongodb 4.2 you can do this:
db.example.update({}, [
{$set: {field: {
$concatArrays: [
{$slice: ["$field", P]},
{$slice: ["$field", {$add: [1, P]}, {$size: "$field"}]}
]
}}}
]);
P is the index of element you want to remove from array.
If you want to remove from P till end:
db.example.update({}, [
{ $set: { field: { $slice: ["$field", 1] } } },
]);
Starting in Mongo 4.4, the $function aggregation operator allows applying a custom javascript function to implement behaviour not supported by the MongoDB Query Language.
For instance, in order to update an array by removing an element at a given index:
// { "name": "dannie", "interests": ["guitar", "programming", "gadgets", "reading"] }
db.collection.update(
{ "name": "dannie" },
[{ $set:
{ "interests":
{ $function: {
body: function(interests) { interests.splice(2, 1); return interests; },
args: ["$interests"],
lang: "js"
}}
}
}]
)
// { "name": "dannie", "interests": ["guitar", "programming", "reading"] }
$function takes 3 parameters:
body, which is the function to apply, whose parameter is the array to modify. The function here simply consists in using splice to remove 1 element at index 2.
args, which contains the fields from the record that the body function takes as parameter. In our case "$interests".
lang, which is the language in which the body function is written. Only js is currently available.
Rather than using the unset (as in the accepted answer), I solve this by setting the field to a unique value (i.e. not NULL) and then immediately pulling that value. A little safer from an asynch perspective. Here is the code:
var update = {};
var key = "ToBePulled_"+ new Date().toString();
update['feedback.'+index] = key;
Venues.update(venueId, {$set: update});
return Venues.update(venueId, {$pull: {feedback: key}});
Hopefully mongo will address this, perhaps by extending the $position modifier to support $pull as well as $push.
I would recommend using a GUID (I tend to use ObjectID) field, or an auto-incrementing field for each sub-document in the array.
With this GUID it is easy to issue a $pull and be sure that the correct one will be pulled. Same goes for other array operations.
For people who are searching an answer using mongoose with nodejs. This is how I do it.
exports.deletePregunta = function (req, res) {
let codTest = req.params.tCodigo;
let indexPregunta = req.body.pregunta; // the index that come from frontend
let inPregunta = `tPreguntas.0.pregunta.${indexPregunta}`; // my field in my db
let inOpciones = `tPreguntas.0.opciones.${indexPregunta}`; // my other field in my db
let inTipo = `tPreguntas.0.tipo.${indexPregunta}`; // my other field in my db
Test.findOneAndUpdate({ tCodigo: codTest },
{
'$unset': {
[inPregunta]: 1, // put the field with []
[inOpciones]: 1,
[inTipo]: 1
}
}).then(()=>{
Test.findOneAndUpdate({ tCodigo: codTest }, {
'$pull': {
'tPreguntas.0.pregunta': null,
'tPreguntas.0.opciones': null,
'tPreguntas.0.tipo': null
}
}).then(testModificado => {
if (!testModificado) {
res.status(404).send({ accion: 'deletePregunta', message: 'No se ha podido borrar esa pregunta ' });
} else {
res.status(200).send({ accion: 'deletePregunta', message: 'Pregunta borrada correctamente' });
}
})}).catch(err => { res.status(500).send({ accion: 'deletePregunta', message: 'error en la base de datos ' + err }); });
}
I can rewrite this answer if it dont understand very well, but I think is okay.
Hope this help you, I lost a lot of time facing this issue.
It is little bit late but some may find it useful who are using robo3t-
db.getCollection('people').update(
{"name":"dannie"},
{ $pull:
{
interests: "guitar" // you can change value to
}
},
{ multi: true }
);
If you have values something like -
property: [
{
"key" : "key1",
"value" : "value 1"
},
{
"key" : "key2",
"value" : "value 2"
},
{
"key" : "key3",
"value" : "value 3"
}
]
and you want to delete a record where the key is key3 then you can use something -
db.getCollection('people').update(
{"name":"dannie"},
{ $pull:
{
property: { key: "key3"} // you can change value to
}
},
{ multi: true }
);
The same goes for the nested property.
this can be done using $pop operator,
db.getCollection('collection_name').updateOne( {}, {$pop: {"path_to_array_object":1}})
I'm trying to figure out how Mongoose and MongoDB works... I'm really new to them, and I can't seem to figure how to return values based on a find statement, where some of the given parameters in the query possible are null - is there an attribute I can set for this or something?
To explain it further, I have a web page that has different input fields that are used to search for a company, however they're not all mandatory.
var Company = mongoose.model('Company');
Company.find({companyName: req.query.companyName, position: req.query.position,
areaOfExpertise: req.query.areaOfExpertise, zip: req.query.zip,
country: req.query.country}, function(err, docs) {
res.json(docs);
});
By filling out all the input fields on the webpage I get a result back, but only that specific one which matches. Let's say I only fill out country, it returns nothing because the rest are empty, but I wish to return all rows which are e.g. in Germany. I hope I expressed myself clearly enough.
You need to wrap the queries with the $or logic operator, for example
var Company = mongoose.model('Company');
Company.find(
{
"$or": [
{ "companyName": req.query.companyName },
{ "position": req.query.position },
{ "areaOfExpertise": req.query.areaOfExpertise },
{ "zip": req.query.zip },
{ "country": req.query.country }
]
}, function(err, docs) {
res.json(docs);
}
);
Another approach would be to construct a query that checks for empty parameters, if they are not null then include it as part of the query. For example, you can just use the req.query object as your query assuming the keys are the same as your document's field, as in the following:
/*
the req.query object will only have two parameters/keys e.g.
req.query = {
position: "Developer",
country: "France"
}
*/
var Company = mongoose.model('Company');
Company.find(req.query, function(err, docs) {
if (err) throw err;
res.json(docs);
});
In the above, the req.query object acts as the query and has an implicit logical AND operation since MongoDB provides an implicit AND operation when specifying a comma separated list of expressions. Using an explicit AND with the $and operator is necessary when the same field or operator has to be specified in multiple expressions.
If you are after a query that satisfies both logical AND and OR i.e. return all documents that match the conditions of both clauses for example given a query for position AND country OR any other fields then you may tweak the query to:
var Company = mongoose.model('Company');
Company.find(
{
"$or": [
{ "companyName": req.query.companyName },
{
"position": req.query.position,
"country": req.query.country
},
{ "areaOfExpertise": req.query.areaOfExpertise },
{ "zip": req.query.zip }
]
}, function(err, docs) {
res.json(docs);
}
);
but then again this could be subject to what query parameters need to be joined as mandatory etc.
I simply ended up deleting the parameters in the query in case they were empty. It seems all the text fields in the submit are submitted as "" (empty). Since there are no such values in the database, it would return nothing. So simple it never crossed my mind...
Example:
if (req.query.companyName == "") {
delete req.query.companyName;
}
Consider following sample documents stored in CouchDB
{
"_id":....,
"rev":....,
"type":"orders",
"Period":"2013-01",
"Region":"East",
"Category":"Stationary",
"Product":"Pen",
"Rate":1,
"Qty":10,
"Amount":10
}
{
"_id":....,
"rev":....,
"type":"orders",
"Period":"2013-02",
"Region":"South",
"Category":"Food",
"Product":"Biscuit",
"Rate":7,
"Qty":5,
"Amount":35
}
Consider following SQL query
SELECT Period, Region,Category, Product, Min(Rate),Max(Rate),Count(Rate), Sum(Qty),Sum(Amount)
FROM Sales
GROUP BY Period,Region,Category, Product;
Is it possible to create map/reduce views in couchdb equivalent to the above SQL query and to produce output like
[
{
"Period":"2013-01",
"Region":"East",
"Category":"Stationary",
"Product":"Pen",
"MinRate":1,
"MaxRate":2,
"OrdersCount":20,
"TotQty":1000,
"Amount":1750
},
{
...
}
]
Up front, I believe #benedolph's answer is best-practice and best-case-scenario. Each reduce should ideally return 1 scalar value to keep the code as simple as possible.
However, it is true you'd have to issue multiple queries to retrieve the full resultset described by your question. If you don't have the option to run queries in parallel, or it is really important to keep the number of queries down it is possible to do it all at once.
Your map function will remain pretty simple:
function (doc) {
emit([ doc.Period, doc.Region, doc.Category, doc.Product ], doc);
}
The reduce function is where it gets lengthy:
function (key, values, rereduce) {
// helper function to sum all the values of a specified field in an array of objects
function sumField(arr, field) {
return arr.reduce(function (prev, cur) {
return prev + cur[field];
}, 0);
}
// helper function to create an array of just a single property from an array of objects
// (this function came from underscore.js, at least it's name and concept)
function pluck(arr, field) {
return arr.map(function (item) {
return item[field];
});
}
// rereduce made this more challenging, and I could not thoroughly test this right now
// see the CouchDB wiki for more information
if (rereduce) {
// a rereduce handles transitionary values
// (so the "values" below are the results of previous reduce functions, not the map function)
return {
OrdersCount: sumField(values, "OrdersCount"),
MinRate: Math.min.apply(Math, pluck(values, "MinRate")),
MaxRate: Math.max.apply(Math, pluck(values, "MaxRate")),
TotQty: sumField(values, "TotQty"),
Amount: sumField(values, "Amount")
};
} else {
var rates = pluck(values, "Rate");
// This takes a group of documents and gives you the stats you were asking for
return {
OrdersCount: values.length,
MinRate: Math.min.apply(Math, rates),
MaxRate: Math.max.apply(Math, rates),
TotQty: sumField(values, "Qty"),
Amount: sumField(values, "Amount")
};
}
}
I was not able to test the "rereduce" branch of this code at all, you'll have to do that on your end. (but this should work) See the wiki for information about reduce vs rereduce.
The helper functions I added at the top actually made the code overall much shorter and easier to read, they're largely influenced by my experience with Underscore.js. However, you can't include CommonJS modules in reduce functions, so it has to be written manually.
Again, best-case scenario is to have each aggregated field get it's own map/reduce index, but if that isn't on option to you, the above code should get you what you've described here in the question.
I will propose a very simple solution that requires one view per variable you want to aggregate in your "select" clause. While it is certainly possible to aggregate all variables in a single view, the reduce function would be far more complex.
The design document looks like this:
{
"_id": "_design/ddoc",
"_rev": "...",
"language": "javascript",
"views": {
"rates": {
"map": "function(doc) {\n emit([doc.Period, doc.Region, doc.Category, doc.Product], doc.Rate);\n}",
"reduce": "_stats"
},
"qty": {
"map": "function(doc) {\n emit([doc.Period, doc.Region, doc.Category, doc.Product], doc.Qty);\n}",
"reduce": "_stats"
}
}
}
Now, you can query <couchdb>/<database>/_design/ddoc/_view/rates?group_level=4 to get the statistics about the "Rate" variable. The result should look like this:
{"rows":[
{"key":["2013-01","East","Stationary","Pen"],"value":{"sum":4,"count":3,"min":1,"max":2,"sumsqr":6}},
{"key":["2013-01","North","Stationary","Pen"],"value":{"sum":1,"count":1,"min":1,"max":1,"sumsqr":1}},
{"key":["2013-01","South","Stationary","Pen"],"value":{"sum":0.5,"count":1,"min":0.5,"max":0.5,"sumsqr":0.25}},
{"key":["2013-02","South","Food","Biscuit"],"value":{"sum":7,"count":1,"min":7,"max":7,"sumsqr":49}}
]}
For the "Qty" variable, the query would be <couchdb>/<database>/_design/ddoc/_view/qty?group_level=4.
With the group_level property you can control over which levels the aggregation is to be performed. For example, querying with group_level=2 will aggregate up to "Period" and "Region".