I have collection with name products with almost 100k documents. I want to introduce a new key called secondaryKey with unique value uuid in all the documents.
I do this using nodejs.
Problem I am facing:-
When I try the below query,
db.collection('products').updateMany({},{"$set":{secondaryKey: uuid()}});
Here it updates all the documents with same uuid value,
I try with loop to update document one by one,but here issues is I don't have filter value in updateOne because I want to update all the documents.
Can anyone please help me here.
Thanks :)
If you are using MongoDB version >= 4.4 You can try this:
db.products.updateMany(
{},
[
{
$set: {
secondaryKey: {
$function: {
body: function() {
return UUID().toString().split('"')[1];
},
args: [],
lang: "js"
}
}
}
}
]
);
Output
[
{
"_id": ObjectId("..."),
"secondaryKey": "f41b15b7-a0c5-43ed-9d15-69dbafc0ed29"
},
{
"_id": ObjectId("..."),
"secondaryKey": "50ae7248-a92e-4b10-be7d-126b8083ff64"
},
{
"_id": ObjectId("..."),
"secondaryKey": "fa778a1a-371b-422a-b73f-8bcff865ad8e"
}
]
Since it's not the same value you want to put in each document you have to use the loop.
In your loop, you have to update the current document of the iteration. So you have to filter with the _id in the updateOne
The above reply didn't work for me. Plus, it compromises security when you enable javascript on your database (see here $function and javascript enabling on database). The best way is to not overload your server, do your work on local as below:
const { nanoid, customAlphabet } = require('nanoid')
async function asdf() {
const movies = await client.db("localhost").collection("productpost");
var result2 = []
let result = await movies.find({}).toArray()
result.forEach(element => {
const nanoid = customAlphabet('1234567890', 10)
console.log(element.price)
element.price = 4
element.id = nanoid()
result2.push(element)
});
console.log("out reult2", result2)
await movies.deleteMany({})
await movies.insertMany(result2)
})
It will delete any objects on your collections and update with the new ones. Using nanoid as uniqueids.
This is the database object array after adding unique id:
{ "_id": { "$oid": "334a98519a20b05c20574dd1" }, "attach": "[\"http://localhost:8000/be/images/2022/4/bitfinicon.png\"]", "title": "jkn jnjn", "description": "jnjn", "price": 4, "color": "After viewing I am 48.73025772956596% more satisfied with life.", "trademark": "", "category": "[]", "productstate": "Published", "createdat": { "$date": "2022-04-03T17:40:54.743Z" }, "language": "en"}
P.S: Please backup your collection before doing this or filter the array on your needs for not going through all collection.
Related
I am working on moving my database from sqlite3 to mongo. I went
through mongo university, yet I'm not sure I have found a really good
example of upsertting in bulk.
Use case : user uploads a data file with a list of players and their stats. The app needs to either update a player or add a new player if they do not already exist.
Current Implementation : Function takes a list of Players and creates SQL statement.
let template = '(Player.Rank, Player.Player_ID, Player.Player, Player.Score, TTP, 1),(Player.Rank, Player_ID, ...) ... (... TTP, 1)';
const stmt = `INSERT INTO playerStats (Rank, Player_ID, Player, Score, TPP, Games_Played)
VALUES ${template}
ON CONFLICT(Player_ID) DO UPDATE
SET Score = Score+excluded.Score, TPP=TPP+excluded.TPP, Games_Played=Games_Played+1`;
db.run(stmt, callback);
Im hoping to have each document be a league which contains players, games, and managers.
Mongo DB document template
{
"LEAGUE_ID": "GEO_SAM",
"Players": [
{
"id": "PlayerID",
"name": "Player",
"score": "Score",
"rank": "Rank",
"xPlayed": "Games_Played",
"ttp": "TTP"
}
],
"Managers": [
{...}
],
"Games": [
{...}
]
}
I am totally lost and not sure how to get this done. Do I need to create a loop and ask mongo to upsert on each iteration? I have searched through countless examples but all of them use static hard coded values.
Here is my testing example.
const query = { League_id : "GEO_SAM", Players.$.id: $in{ $Players }};
const update = { $inc: { score: $Player.Score}};
const options = { upsert: true };
collection.updateMany(query, update, options);
I also don't understand how to pass the entire player object to push to the array if the player_id isn't found.
My solution was to create a metaData field containing the league ID with a single player. If anyone else has a better solution I would love to hear from you.
{
MetaData: { "LEAGUE_ID": "GEO_SAM"},
Player: {
"id": "PlayerID",
"name": "Player",
"score": "Score",
"rank": "Rank",
"xPlayed": "Games_Played",
"ttp": "TTP"
}
}
Then I mapped over the values and inserted each one.
client.connect().then((client) => {
const db = client.db(dbName);
const results = Players.map((player) => {
db.collection('Players').updateOne(
{ Player_Name: player.Player_ID },
{
$setOnInsert: {
Player_ID: player.Player_ID,
Player: player.Player,
Rank: player.Rank,
},
$inc: { Score: player.Score, Games_Played: 1, TPP: player.TPP },
},
{ upsert: true, multi: true },
);
});
Document in mongodb collection 'users' is
{
"$oid": "5e612272bcb362513824ff9b",
"name": "abcd",
"email": "test#test.com",
"cart": {
"items": [{
"productId": {
"$oid": "5e614367cae25319c4388288"
},
"quantity": {
"$numberInt": "1"
}
}]
}
}
For a particular users._id and a productId in cart.items, I need to increase the quantity by 1
My nodejs code is
IncrCartItem(prodId){
const db=getDb()
return db
.collection('users').
updateOne({_id: new mongodb.ObjectId(this._id),'this.cart.items.productId': new mongodb.ObjectId(prodId)},{$inc : {'this.cart.items.quantity':1}})
}
Is the query right for checking multiple conditions in updateOne()??
You're kinda there, to "and" them all together you just keep appending them, however the field is wrong and you need to use the positional operator ($) - https://docs.mongodb.com/manual/reference/operator/update/positional/
const filter = {
_id: new mongodb.ObjectId(this._id),
'cart.items.productId': new mongodb.ObjectId(prodId)
};
const update = {
$inc : { 'this.cart.items.$.quantity': 1 }
};
IncrCartItem(prodId){
const db=getDb()
return db
.collection('users').
updateOne(filter,update)
}
I want to update multiple docs with different values.
My Database looks something like this.
[
{
"_id": 1,
"value": 50
},
{
"_id": 2,
"value": 100
}
]
This Query return an error because i'm passing an array instead of an object in the $set.
Model.update({_id: {$in: ids}}, {$set: ids.value}, {multi: true};
I want my database to look like this
[
{
"_id": 1,
"value": 4
},
{
"_id": 2,
"value": 27
}
]
Supposing you had an array of objects that you wanted to update in your collection on matching ids like
var soldItems = [
{
"_id": 1,
"value": 4
},
{
"_id": 2,
"value": 27
}
];
then you could use the forEach() method on the array to iterate it and update your collection:
soldItems.forEach(function(item)){
Model.update({"_id": item._id}, {"$set": {"value": item.value }}, callback);
});
or use promises as
var updates = [];
soldItems.forEach(function(item)){
var updatePromise = Model.update({"_id": item._id}, {"$set": {"value": item.value }});
updates.push(updatePromise);
});
Promise.all(updates).then(function(results){
console.log(results);
});
or using map()
var updates = soldItems.map(function(item)){
return Model.update({"_id": item._id}, {"$set": {"value": item.value }});
});
Promise.all(updates).then(function(results){
console.log(results);
});
For larger arrays, you could take advantage of using a bulk write API for better performance. For Mongoose versions >=4.3.0 which support MongoDB Server 3.2.x,
you can use bulkWrite() for updates. The following example shows how you can go about this:
var bulkUpdateCallback = function(err, r){
console.log(r.matchedCount);
console.log(r.modifiedCount);
}
// Initialise the bulk operations array
var bulkOps = soldItems.map(function (item) {
return {
"updateOne": {
"filter": { "_id": item._id } ,
"update": { "$set": { "value": item.value } }
}
}
});
// Get the underlying collection via the native node.js driver collection object
Model.collection.bulkWrite(bulkOps, { "ordered": true, w: 1 }, bulkUpdateCallback);
For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows
var bulk = Model.collection.initializeOrderedBulkOp(),
counter = 0;
soldItems.forEach(function(item) {
bulk.find({ "_id": item._id }).updateOne({
"$set": { "value": item.value }
});
counter++;
if (counter % 500 == 0) {
bulk.execute(function(err, r) {
// do something with the result
bulk = Model.collection.initializeOrderedBulkOp();
counter = 0;
});
}
});
// Catch any docs in the queue under or over the 500's
if (counter > 0) {
bulk.execute(function(err,result) {
// do something with the result here
});
}
First of all your update() query is not ok..
See documentation for this here
Model.update(conditions, update, options, callback);
Suppose your current db model contains docs like this (as you described in question as well):
[
{
"_id": 1,
"value": 50
},
{
"_id": 2,
"value": 100
}
];
and you've below array which contains objects (i.e., docs) to be modified with current db's docs to like this:
idsArray: [
{
"_id": 1,
"value": 4
},
{
"_id": 2,
"value": 27
}
];
From my experience with mongodb & mongoose, i don't think you can update all docs with single line query (that's you're trying to do).. (P.S. I don't know about that so I am not sure to this..)
But to make your code work, you will be doing something like this:
Idea: Loop over each doc in docs i.e, idsArray and call update() over it..
So, Here's code to this:
idsArray.forEach(function(obj) {
Model.update({_id: obj._id}, {$set: {value: obj.value}});
});
In above code, I am supposing you've _id values in db docs as they 're written above (i.e, "_id": 1).. But if they're like this "_id": ObjectId('1')
[
{
"_id": ObjectId('1'),
"value": 50
},
.....
.....
]
then you'll be need to convert _id to ObjectId(obj._id) in update() query..
so for that you'll be doing like this.
var ObjectId = require('mongodb').ObjectID;
idsArray.forEach(function(obj) {
Model.update({_id: ObjectId(obj._id)}, {$set: {value: obj.value}});
});
P.S. Just confirm it (i.e., _id) before go forward..
Hope this helps.
Cheers,
Multi update can be used only for updating multiple documents to the same value(s) or updating with the same update operation for all documents.
If you want to update to different values you have to use several update statements.
I have a Model called Post, witch contains an property array with user-ids for users that have liked this post.
Now, i need to query the post model, and mark the returned results with likedBySelf true/false for use in by client - is this possible?
I dont have to store the likedBySelf property in the database, just modify the results to have that property.
A temporary solution i found was to do 2 queries, one that finds the posts that is liked by user x, and the ones that have not been liked by user x, and en map (setting likedBySelf true/false) and combine the 2 arrays and return the combined array. But this gives some limitations to other query functions such as limit and skip.
So now my queries looks like this:
var notLikedByQuery = Post.find({likedBy: {$ne: req.body.user._id}})
var likedByQuery = Post.find({likedBy: req.body.user._id})
(I'm using the Mongoose lib)
PS. A typical post can look like this (JSON):
{
"_id": {
"$oid": "55fc463c83b2d2501f563544"
},
"__t": "Post",
"groupId": {
"$oid": "55fc463c83b2d2501f563545"
},
"inactiveAfter": {
"$date": "2015-09-25T17:13:32.426Z"
},
"imageUrl": "https://hootappprodstorage.blob.core.windows.net/devphotos/55fc463b83b2d2501f563543.jpeg",
"createdBy": {
"$oid": "55c49e2d40b3b5b80cbe9a03"
},
"inactive": false,
"recentComments": [],
"likes": 8,
"likedBy": [
{
"$oid": "558b2ce70553f7e807f636c7"
},
{
"$oid": "559e8573ed7c830c0a677c36"
},
{
"$oid": "559e85bced7c830c0a677c43"
},
{
"$oid": "559e854bed7c830c0a677c32"
},
{
"$oid": "559e85abed7c830c0a677c40"
},
{
"$oid": "55911104be2f86e81d0fb573"
},
{
"$oid": "559e858fed7c830c0a677c3b"
},
{
"$oid": "559e8586ed7c830c0a677c3a"
}
],
"location": {
"type": "Point",
"coordinates": [
10.01941398718396,
60.96738099591897
]
},
"updatedAt": {
"$date": "2015-09-22T08:45:41.480Z"
},
"createdAt": {
"$date": "2015-09-18T17:13:32.426Z"
},
"__v": 8
}
#tskippe you can use a method like following to process whether the post is liked by the user himself and call the function anywhere you want.
var processIsLiked = function(postId, userId, doc, next){
var q = Post.find({post_id: postId});
q.lean().exec(function(err,res){
if(err) return utils.handleErr(err, res);
else {
if(_.find(doc.post.likedBy,userId)){ //if LikedBy array contains the user
doc.post.isLiked = true;
} else {
doc.post.isLiked = false;
}
});
next(doc);
}
});
}
Because you are using q.lean() you dont need to actually persist the data. You need to just process it , add isLiked field in the post and send back the response. **note that we are manuplating doc directly. Also you chan tweek it to accept doc containing array of posts and iterating it and attach an isLiked field to each post.
I found that MongoDB's aggregation with $project tequnique was my best bet. So i wrote up an aggregation like this.
Explanation:
Since i want to keep the entire document, but $project purpose is to modify the docs, thus you have to specify the properties you want to keep. A simple way of keeping all the properties is to use "$$ROOT".
So i define a $project, set all my original properties to doc: "$$ROOT", then create a new property "likedBySelf", which is marked true / false if a specified USERID is in the $likedBy set.
I think that this is more clean and simple, than querying every single model after a query to set a likedBySelf flag. It may not be faster, but its cleaner.
Model.aggregate([
{ $project: {
doc: "$$ROOT",
likedBySelf: {
$cond: {
"if": { "$setIsSubset": [
[USERID],
"$likedBy"
]},
"then": true,
"else": false
}
}
}}
]);
I've been successfully using $in in my node webservice when my mongo arrays only held ids. Here is sample data.
{
"_id": {
"$oid": "52b1a60ce4b0f819260bc6e5"
},
"title": "Sample",
"team": [
{
"$oid": "52995b263e20c94167000001"
},
{
"$oid": "529bfa36c81735b802000001"
}
],
"tasks": [
{
"task": {
"$oid": "52af197ae4b07526a3ee6017"
},
"status": 0
},
{
"task": {
"$oid": "52af197ae4b07526a3ee6017"
},
"status": 1
}
]
}
Notice that tasks is an array, but the id is nested in "task", while in teams it is on the top level. Here is where my question is.
In my Node route, this is how I typically deal with calling a array of IDs in my project, this works fine in the team example, but obviously not for my task example.
app.get('/api/tasks/project/:id', function (req, res) {
var the_id = req.params.id;
var query = req.params.query;
models.Projects.findById(the_id, null, function (data) {
models.Tasks.findAllByIds({
ids: data._doc.tasks,
query: query
}, function(items) {
console.log(items);
res.send(items);
});
});
});
That communicates with my model which has a method called findAllByIds
module.exports = function (config, mongoose) {
var _TasksSchema = new mongoose.Schema({});
var _Tasks = mongoose.model('tasks', _TasksSchema);
/*****************
* Public API
*****************/
return {
Tasks: _Tasks,
findAllByIds: function(data, callback){
var query = data.query;
_Tasks.find({_id: { $in: data.ids} }, query, function(err, doc){
callback(doc);
});
}
}
}
In this call I have $in: data.ids which works in the simple array like the "teams" example above. Once I nest my object, as with "task" sample, this does not work anymore, and I am not sure how to specify $in to look at data.ids array, but use the "task" value.
I'd like to avoid having to iterate through the data to create an array with only id, and then repopulate the other values once the data is returned, unless that is the only option.
Update
I had a thought of setting up my mongo document like this, although I'd still like to know how to do it the other way, in the event this isn't possible in the future.
"tasks": {
"status0": [
{
"$oid": "52995b263e20c94167000001"
},
{
"$oid": "529bfa36c81735b802000001"
}
],
"status1": [
{
"$oid": "52995b263e20c94167000001"
},
{
"$oid": "529bfa36c81735b802000001"
}
]
}
You can call map on the tasks array to project it into a new array with just the ObjectId values:
models.Tasks.findAllByIds({
ids: data.tasks.map(function(value) { return value.task; }),
query: query
}, function(items) { ...
Have you try the $elemMatch option in find conditions ? http://docs.mongodb.org/manual/reference/operator/query/elemMatch/