I currently have an upsert function in my project which works but my main problem is that it creates another instance of the record, and updates the new instance instead. This is the code:
router.route('/carousel/update/:_id').put(function(req, res) {
var id;
if(req.params._id == 'undefined'){
id = crypto.randomBytes(12).toString('hex');
}
else {
id = ObjectId(req.params._id)
}
db.collection('home').updateOne({"_id": id},
{$set: req.body}, {upsert: true}, (err, results) => {
if (err) throw err;
res.send(results)
console.log(req.body)
});
});
The problem:
1. It mystifies me that mongoDB takes my crypto generated _id and takes it as the new _id for the upserted document. Why is that? When {upsert: true}, isn't mongoDB supposed to generate a new _id?
2. Because of the nature of problem 1, whenever I try to update the original document, it updates the upserted document instead since they have the same _id values even though their _ids are positioned at different document levels.
In conclusion, when given a 'home' document, how do I upsert correctly without adding a new record with the same values and _ids?
Thanks for your help!
EDIT
This is the JSON body content of the document with custom generated _id using crypto:
{
"_id": "1262d480eea83567181b3206",
"header": "hello",
"subheader": "hello"
}
Whereas, this is the body content of the upserted document.
{
"_id": {
"$oid": "1262d480eea83567181b3206"
},
"header": "helloasad",
"subheader": "helloasda"
}
As observed, after upserting, it takes the same _id value of the original document but on another document level.
A possible solution/explanation based on #Ashwanth Madhav information:
In your code 'id' was being sent to the update as a String type, but the id in MongoDB is an ObjectId type:
Code will be something like that:
var id;
if(req.params._id == 'undefined'){
// 'id' NEED TO BE AN ObjectId...
// 'id' WAS BEING SENT AS A 'String'
id = ObjectId(crypto.randomBytes(12).toString('hex'));
}
else {
id = ObjectId(req.params._id)
}
Related
I came from relational database whereas the primary key (this case, _id) is the same along its life, and so I was surprised when I saw this behavior in mongodb.
I'm using the mongoose's findOneAndUpdate plugin method in the below way:
User.findOneAndUpdate(
{ "products._id": _id, "_id": req.payload._id },
{
"$set": {
"products.$": {name: "New name"}
}
},
{
new: true ,
runValidators: true
},
function (err, doc) {
if (err != null) {
res.status(500).json({ message: "Error on updating. Please, try again later." });
} else if (doc == null) {
res.status(404).json({ message: "Product not found." });
} else {
res.status(200).json(doc.products)
}
}
);
Before start:
{_id: 58b5e637f9f904a800721abf, name: "Old name"}
After (_id changed):
{_id: 58b5e35a7f4ff38c433a5bc9, name: "New name"}
I just want to keep the same _id after an update, 'cause I think I could to face troubles when I implement a simultaneous updates for example.
I searched and I found out this mongoose method is called straightforward to the mongo's drivers without middlewares. Thus, I guess this question can be solved by experts in mongodb without knowledge in mongoose.
_id is attached to a document revision, not a document entity.
By passing new: true you're asking Mongo to return the id of the latest revision, which will have a different id than the original document (Upsert).
For document based storages it's recommended to implement your own UUID schema.
Either go deterministic with uuid:
var UUID = require('uuid-1345');
UUID.v3({
namespace: UUID.namespace.oid,
name: "abc" // Some formula to calculate you uuid, could be based on the document's legacy id or some other unique identifier.
}, function (err, id) {
console.log("Generated a name-based UUID using MD5:\n\t%s\n", id);
});
Or random with plain random HEX:
var crypto = require('crypto');
var id = crypto.randomBytes(20).toString('hex');
Include this in your document's body... and don't forget to index it!
So I have a fairly simple piece of code as follows
db.get().collection('bars').insert({
barID: req.body.button,
}, {
$push: {
usersfbID: req.body.profileUser[0].facebookID
}
}, function(err, doc) {
if (err) {
throw err;
}
if (doc) {
console.log('Had to create a new document for this bar');
console.log(doc);
//callback(null, doc);
}
});
So, I'm just checking to see if a document for a bar exists, and if it doesn't then I create that document. And I want to insert an array for the usersfbID field so that I can store all the users going to the bar.
However, when I run the code, I don't get an error and it says the document has inserted but when the document logs, it doesn't have the userfbID field.
So what am I doing wrong? Does the $push operator only work with the update method of db? If so, how do I insert an array for that field?
Yes, it does work with the update methods
Reference > Operators > Update Operators > Array Update Operators > $push
Inserting a new entry means feeding the fields. In that case, there's no $push operation, since the array of the entry is freshly created and can be explicitly set (usersfbID:[req.body.profileUser[0].facebookID], meaning that you expect several fbId for that bar). Updating an array in an element of a collection isn't an insertion, it's an update.
So, just to provide an answer to the question I was facing..
Yes, you can only use $push or $addToSet with an update operation on a mongoDB document
Here is the way I implemented the code.
db.get().collection('bars').update({
barID: req.body.button,
}, {
$addToSet: {
usersfbID: req.body.profileUser[0].facebookID,
usersDocID: req.body.profileUser[0]._id
}
}, {
upsert: true
}, function(err, doc) {
if (err) {
console.log('There is an error here');
throw err;
}
if (doc) {
console.log('Had to create a new document for this bar');
callback(null, doc);
}
});
The upsert: true makes sure to insert a new document if the update method couldn't find the specified document.
Due to a management decision, we are using userId for the users collection, postId for the posts collection, and topicId for the topics collection, instead of '_id' for each collection as the unique identifier.
This causes a few problems getting started - one of the problems I have encountered is with upserts -
Using Mongoose, we have a schema that restricts userId to be a unique value - but when doing an update on a user model, with upsert set to true, MongoDB appears to only look at the ObjectIds of a collection to see if the same one exists - it doesn't check to see if a model already exists with the same userId - therefore Mongo does an insert instead of an update.
let me illustrate this with some data:
let's say the user's collection has one document:
{
_id:'561b0fad638e99481ab6d84a'
userId:3,
name:'foo'
}
we then run:
User.update({userId:3},{"$set":{name:'bar'},{upsert:true},function(err,resp){
if(err){
// "errMessage": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: app42153482.users.$userId_1 dup key: { : 3 }",
}
});
one would think that MongoDB would find the existing document with userId:3 and udpate it, so there must be something I am doing wrong since it's giving me the duplicate key error?
Typically the default value ObjectId is more ideal for the _id. Here, in this situation you can either override the default _id or you can have your own field for id(like userId in your case).
Use a separate counters collection to track the last number sequence used. The _id field contains the sequence name and the seq field contains the last value of the sequence.
Insert into the counters collection, the initial value for the userid:
db.counters.insert( {
_id: "userid",
seq: 0 } )
Create a getNextSequence function that accepts a name of the sequence. The function uses the findAndModify() method to atomically increment the seq value and return this new value:
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
Use this getNextSequence() function during insert().
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Bob D."
}
)
This way you can maintain as many sequences as you want in the same counter collection. For the upsert issue, check out the Optimistic Loop block in this link Create an auto-increment sequence field.
The second approach is to use a mongoose middleware like mongodb-autoincrement.
Hope it helps.
I don't know which versions of MongoDB and Mongoose you are using, but I couldn't reproduce your problem with MongoDB 3.0 and Mongoose 4.1.10.
I made a sample for you which will create and save a new user, update (using upsert) it, and create another one through an upsert. Try running this code:
"use strict";
var mongoose=require("mongoose");
var Schema = require('mongoose').Schema;
var ObjectId = mongoose.Schema.Types.ObjectId;
// Connect to test
mongoose.connect("mongodb://localhost:27017/test");
// Lets create your schema
var userSchema = new Schema({
_id: ObjectId,
userId: {type: Number, unique: true },
name: String
});
var User = mongoose.model("User", userSchema, "Users");
User.remove() // Let's prune our collection to start clean
.then( function() {
// Create our sample record
var myUser = new User({
_id:'561b0fad638e99481ab6d84a',
userId:3,
name:'foo'
});
return myUser.save();
})
.then( function() {
// Now its time to update (upsert userId 3)
return User.update({userId:3},{"$set":{name:'bar'}},{upsert:true});
})
.then( function() {
// Now its time to insert (upsert userId 4)
return User.update({userId:4},{"$set":{name:'bee'}},{upsert:true});
})
.then( function() {
// Lets show what we have inserted
return User.find().then(function(data) {console.log(data)});
})
.catch( function(err) {
// Show errors if anything goes wrong
console.error("ERROR", err);
})
.then( function() {
mongoose.disconnect();
});
Following the documentation (of MongoDB 3.0) upsert:true will only not insert a non-existing document if your query conditions match on the _id field.
See: https://docs.mongodb.org/manual/reference/method/db.collection.update/#mongodb30-upsert-id
Why are you not using the user_name for a user as unique id?
Because auto-incrementing fields as ids are a bad practice to use in a mongodb environment, especially if you want to use sharding
=> all your inserts will occur on the latest shard
=> the mongodb cluster will have to rebalance often / redistribute the data around.
(Currently this will not occur on your system as you still use the generated _id field)
You can off course also create a unique index on the user_id field:
https://docs.mongodb.org/manual/core/index-unique/#index-type-unique
I'm getting a duplicate document when using the mongodb-native-driver to save an update to a document. My first call to save() correctly creates the document and adds a _id with an ObjectID value. A second call creates a new document with a text _id of the original ObjectID. For example I end up with:
> db.people.find()
{ "firstname" : "Fred", "lastname" : "Flintstone", "_id" : ObjectId("52e55737ae49620000fd894e") }
{ "firstname" : "Fred", "lastname" : "Flintstone with a change", "_id" : "52e55737ae49620000fd894e" }
My first call correctly created Fred Flinstone. A second call that added " with a change" to the lastname, created a second document.
I'm using MongoDB 2.4.8 and mongo-native-driver 1.3.23.
Here is my NodeJS/Express endpoint:
app.post("/contacts", function (req, res) {
console.log("POST /contacts, req.body: " + JSON.stringify(req.body));
db.collection("people").save(req.body, function (err, inserted) {
if (err) {
throw err;
} else {
console.dir("Successfully inserted/updated: " + JSON.stringify(inserted));
res.send(inserted);
}
});
});
Here is the runtime log messages:
POST /contacts, req.body: {"firstname":"Fred","lastname":"Flintstone"}
'Successfully inserted/updated: {"firstname":"Fred","lastname":"Flintstone","_id":"52e55737ae49620000fd894e"}'
POST /contacts, req.body: {"firstname":"Fred","lastname":"Flintstone with a change","_id":"52e55737ae49620000fd894e"}
'Successfully inserted/updated: 1'
Why doesn't my second update the existing record? Does the driver not cast the _id value to an ObjectID?
What you are posting back the 2nd time contains a field named "_id", and it's a string. That is the problem.
Look at the document, what the save method does is a "Simple full document replacement function". I don't use this function quit often so here's what I guess. The function use the _id field to find the document and then replace the full document with what you provided. However, what you provided is a string _id. Apparently it doesn't equal to the ObjectId. I think you should wrap it to an ObjectId before passing to the function.
Besides, the save method is not recommended according to the document. you should use update (maybe with upsert option) instead
I don't exactly know why a second document is created, but why don't you use the update function (maybe with the upsert operator)?
An example for the update operation:
var query = { '_id': '52e55737ae49620000fd894e' };
db.collection('people').findOne(query, function (err, doc) {
if (err) throw err;
if (!doc) {
return db.close();
}
doc['lastname'] = 'Flintstone with a change';
db.collection('people').update(query, doc, function (err, updated) {
if (err) throw err;
console.dir('Successfully updated ' + updated + ' document!');
return db.close();
});
});
And now with the upsert operator:
var query = { '_id': '52e55737ae49620000fd894e' };
var operator = { '$set': { 'lastname': 'Flintstone with a change' } };
var options = { 'upsert': true };
db.collection('people').update(query, operator, options, function (err, upserted) {
if (err) throw err;
console.dir('Successfully upserted ' + upserted + ' document!');
return db.close();
});
The difference is that the upsert operator will update the document if it exist, otherwise it will create a new one. When using the upsert operator you should keep in mind that this operation can be underspecified. That means if your query does not contain enough information to identify a single document, a new document will be inserted.
I would like to retrieve some data from a Mongoose setting in my Node.js application. I noticed that no matter what I write as field selection, I always get the _id field. Is there a way not to fetch it?
This is how I do right now:
Transaction.find({username : user.username}, ['uniqueId', 'timeout', 'confirmation_link', 'item_name'], function(err, txs){
console.log("user : " + user.username + " with txs: " + txs);
callback(txs);
});
And logs me the results which contain the _id field.
Another way is to use text argument with prefix - which will exclude this or that field from the result:
Entity.find({ ... }, '-_id field1 field2', function(err, entity) {
console.log(entity); // { field1: '...', field2: '...' }
});
_id must be specifically excluded. For example,
Transaction.find({username : user.username}, { '_id': 0, 'uniqueId' :1, 'timeout': 1, 'confirmation_link': 1, 'item_name': 1}, function(err, txs){
console.log("user : " + user.username + " with txs: " + txs);
callback(txs);
});
Another approach:
Augment the .toJSON() of the schema that it deletes the _id and the __v fields
Call .toJSON() on all DB objects sent to client
Extra benefit #1: you can use item.id === 'something' because typeof id === 'string', not ObjectId.
Extra benefit #2: When you got gan object back from the client and you want to search / update then you don't have to manually delete _id because there is none, just an id which is ignored.
Augmenting JSON:
mySchema.set('toJSON', {
virtuals: true,
transform: (doc, ret, options) => {
delete ret.__v;
ret.id = ret._id.toString();
delete ret._id;
},
});
So you can use:
let item = (await MyCollection.findOne({/* search */}).exec()).toJSON();
if (item.id === 'someString') return item;
I know it's ugly. But it's the best bad idea that I have so far.
In 5.2.13 version of Mongoose (Sept 2018)- using the query builder approach the same can be converted to
async function getUserDetails(user) {
try {
if (!user || !user.name) return;
const result = await Transaction.
find({username : user.username}).
select('uniqueId timeout confirmation_link item_name -_id');
// Adding minus sign before the _id (like -_id) in the select string unselects the _id which is sent by default.
console.log(result);
} catch(ex) {
return ex
}
}
The easiest thing you can do is something like this:
Transaction.find({username : user.username}, {_id: 0}, (err, txs) => {
// the return document won't contain _id field
// callback function body
}
Just remember that in the second object passed in the find()-
Pass 0 as the value to the specific key that you wish not to fetch
from the mongodb database.
Pass 1 as the value when you wish to
fetch from the mongodb database.