I have problem with duplicate key.
Long time can`t find answer. Please help me solve this problem or explain why i get duplicate key error.
Trace: { [MongoError: E11000 duplicate key error collection: project.monitor index: _id_ dup key: { : 24392490 }]
name: 'MongoError',
message: 'E11000 duplicate key error collection: project.monitor index: _id_ dup key: { : 24392490 }',
driver: true,
index: 0,
code: 11000,
errmsg: 'E11000 duplicate key error collection: project.monitor index: _id_ dup key: { : 24392490 }' }
at /home/project/app/lib/monitor.js:67:12
at callback (/home/project/app/node_modules/mongoose/lib/query.js:2029:9)
at Immediate.<anonymous> (/home/project/app/node_modules/kareem/index.js:160:11)
at Immediate._onImmediate (/home/project/app/node_modules/mquery/lib/utils.js:137:16)
at processImmediate [as _immediateCallback] (timers.js:368:17)
but in monitor i use upsert, so why i get duplicate error??
monitor.js:62-70
monitor schema
var monitorSchema = db.Schema({
_id : {type: Number, default: utils.minute},
maxTicks : {type: Number, default: 0},
ticks : {type: Number, default: 0},
memory : {type: Number, default: 0},
cpu : {type: Number, default: 0},
reboot : {type: Number, default: 0},
streams : db.Schema.Types.Mixed
}, {
collection: 'monitor',
strict: false
});
index
monitorSchema.index({_id: -1});
Monitor = db.model('Monitor', monitorSchema);
and increase by property
exports.increase = function (property, incr) {
var update = {};
update[property] = utils.parseRound(incr) || 1;
Monitor.update({_id: utils.minute()}, {$inc: update}, {upsert: true}, function (err) {
if (err) {
console.trace(err);
}
});
};
utils.js
exports.minute = function () {
return Math.round(Date.now() / 60000);
};
exports.parseRound = function (num, round) {
if (isNaN(num)) return 0;
return Number(parseFloat(Number(num)).toFixed(round));
};
An upsert that results in a document insert is not a fully atomic operation. Think of the upsert as performing the following discrete steps:
Query for the identified document to upsert.
If the document exists, atomically update the existing document.
Else (the document doesn't exist), atomically insert a new document that incorporates the query fields and the update.
So steps 2 and 3 are each atomic, but another upsert could occur after step 1 so your code needs to check for the duplicate key error and then retry the upsert if that occurs. At that point you know the document with that _id exists so it will always succeed.
For example:
var minute = utils.minute();
Monitor.update({ _id: minute }, { $inc: update }, { upsert: true }, function(err) {
if (err) {
if (err.code === 11000) {
// Another upsert occurred during the upsert, try again. You could omit the
// upsert option here if you don't ever delete docs while this is running.
Monitor.update({ _id: minute }, { $inc: update }, { upsert: true },
function(err) {
if (err) {
console.trace(err);
}
});
}
else {
console.trace(err);
}
}
});
See here for the related documentation.
You may still be wondering why this can happen if the insert is atomic, but what that means is that no updates will occur on the inserted document until it is completely written, not that no other insert of a doc with the same _id can occur.
Also, you don't need to manually create an index on _id as all MongoDB collections have a unique index on _id regardless. So you can remove this line:
monitorSchema.index({_id: -1}); // Not needed
Related
I have a unique index like this
code: {
type: String,
index: {
unique: true,
partialFilterExpression: {
code: { $type: 'string' }
}
},
default: null
},
state: { type: Number, default: 0 },
but When the state is 2 (archived) I want to keep the code, but it should be able to reuse the code, so it cannot be unique if state is 2.
Is there any away that I could accomplish this?
This is possible, though it's through a work around documented here https://jira.mongodb.org/browse/SERVER-25023.
In MongoDB 4.7 you will be able to apply different index options to the same field but for now you can add a non-existent field to separate the two indexes.
Here's an example using the work around.
(async () => {
const ItemSchema = mongoose.Schema({
code: {
type: String,
default: null
},
state: {
type: Number,
default: 0,
},
});
// Define a unique index for active items
ItemSchema.index({code: 1}, {
name: 'code_1_unique',
partialFilterExpression: {
$and: [
{code: {$type: 'string'}},
{state: {$eq: 0}}
]
},
unique: true
})
// Defined a non-unique index for non-active items
ItemSchema.index({code: 1, nonExistantField: 1}, {
name: 'code_1_nonunique',
partialFilterExpression: {
$and: [
{code: {$type: 'string'}},
{state: {$eq: 2}}
]
},
})
const Item = mongoose.model('Item', ItemSchema)
await mongoose.connect('mongodb://localhost:27017/so-unique-compound-indexes')
// Drop the collection for test to run correctly
await Item.deleteMany({})
// Successfully create an item
console.log('\nCreating a unique item')
const itemA = await Item.create({code: 'abc'});
// Throws error when trying to create with the same code
await Item.create({code: 'abc'})
.catch(err => {console.log('\nThrowing a duplicate error when creating with the same code')})
// Change the active code
console.log('\nChanging item state to 2')
itemA.state = 2;
await itemA.save();
// Successfully created a new doc with sama code
await Item.create({code: 'abc'})
.then(() => console.log('\nSuccessfully created a new doc with sama code'))
.catch(() => console.log('\nThrowing a duplicate error'));
// Throws error when trying to create with the same code
Item.create({code: 'abc'})
.catch(err => {console.log('\nThrowing a duplicate error when creating with the same code again')})
})();
This is not possible with using indexes. Even if you use a compound index for code and state there will still be a case where
new document
{
code: 'abc',
state: 0
}
archived document
{
code: 'abc',
state: 2
}
Now although you have the same code you will not be able to archive the new document or unarchive the archived document.
You can do something like this
const checkCode = await this.Model.findOne({code:'abc', active:0})
if(checkCode){
throw new Error('Code has to be unique')
}
else{
.....do something
}
I was trying to delete a specific value (a game) from my array in my schema, and this is the code:
User.update({ userName: user }, { $pull: { games: { _id: deleteItem } } }, function (err, val) {
console.log(err);
});
the schema:
const userSchema = new mongoose.Schema({
userName: { type: String, index: true, unique: true },
userPassword: String,
games: [gameSchema]
});
the error:
MongoError: E11000 duplicate key error collection: mountain.users index: games.password_1
errmsg: 'E11000 duplicate key error collection: mountain.users index: games.password_1 dup key: { games.password: null }',
[Symbol(mongoErrorContextSymbol)]: {}
}
why is the error apear and how can I solve it?/other way to delete a value from array inside on object
thanks for your help!
You have a unique index built on games. password array.
I assume the game you're trying to pull is the last game in the array. and that you already have a document with an empty games array.
Hence the value of both of those documents for the index is (null) the same as they both don't exist.
sparse indexes exist for this very reason, it allows you to benefit from the unique behaviour while only taking under account documents with values that exist.
So basically you have to re-built your index to be a unique + sparse one.
When I execute the following code (a larger example, boiled down to the essentials)
var mongoose = require("mongoose");
var LocationSchema = new mongoose.Schema({
userName: String,
loc: {
'type': { type: String, enum: "Point", default: "Point" },
coordinates: { type: [Number] }
}
})
LocationSchema.index({ category: 1, loc: "2dsphere" });
var Location = mongoose.model("location", LocationSchema);
var mongoDB = 'mongodb://user1:test#ds042417.mlab.com:42417/locationdemo';
mongoose.Promise = global.Promise;
mongoose.connect(mongoDB, { useMongoClient: true });
var testUser = Location({
userName: "Tester",
loc: { coordinates: [12.44, 55.69] }
});
testUser.save(function (err) {
if (err) {
return console.log("UPPPPs: " + err);
}
console.log("User Saved, Try to find him:");
let query = Location.find({
loc:
{
$near:
{
$geometry:
{
type: "Point",
coordinates: [12.50, 55.71]
},
$maxDistance: 600000
}
}
})
query.exec(function (err, docs) {
if (err) {
return console.log("Err: " + err);
}
console.log("Found: " + JSON.stringify(docs))
})
});
I get this error:
Err: MongoError: error processing query: ns=locationdemo.locationsTree: GEONEAR field=loc maxdist=600000 isNearSphere=0
Sort: {}
Proj: {}
planner returned error: unable to find index for $geoNear query
But the index is there (see line 10) and the screenshot from mlab below. What am I doing wrong?:
You are breaking a rule of how you can use a an index in general. Whilst it is true that there is no restriction that a "2dsphere" index be the "first" property in a compound index, it is however very important that your "queries" actually address the first property in order for the index to be selected.
This is covered in Prefixes from the manual on compound indexes. In excerpt:
{ "item": 1, "location": 1, "stock": 1 }
The index has the following index prefixes:
{ item: 1 }
{ item: 1, location: 1 }
For a compound index, MongoDB can use the index to support queries on the index prefixes. As such, MongoDB can use the index for queries on the following fields:
the item field,
the item field and the location field,
the item field and the location field and the stock field.
However, MongoDB cannot use the index to support queries that include the following fields since without the item field, none of the listed fields correspond to a prefix index:
the location field,
the stock field, or
the location and stock fields.
Because your query references "loc" first and does not include "category", the index does not get selected and MongoDB returns the error.
So in order to use the index you have defined, you need to actually query "category" as well. Amending your listing:
var mongoose = require("mongoose");
mongoose.set('debug',true);
var LocationSchema = new mongoose.Schema({
userName: String,
category: Number,
loc: {
'type': { type: String, enum: "Point", default: "Point" },
coordinates: { type: [Number] }
}
})
//LocationSchema.index({ loc: "2dsphere", category: 1 },{ "background": false });
LocationSchema.index({ category: 1, loc: "2dsphere" });
var Location = mongoose.model("location", LocationSchema);
var mongoDB = 'mongodb://localhost/test';
mongoose.Promise = global.Promise;
mongoose.connect(mongoDB, { useMongoClient: true });
var testUser = Location({
userName: "Tester",
category: 1,
loc: { coordinates: [12.44, 55.69] }
});
testUser.save(function (err) {
if (err) {
return console.log("UPPPPs: " + err);
}
console.log("User Saved, Try to find him:");
let query = Location.find({
category: 1,
loc:
{
$near:
{
$geometry:
{
type: "Point",
coordinates: [12.50, 55.71]
},
$maxDistance: 600000
}
}
})
query.exec(function (err, docs) {
if (err) {
return console.log("Err: " + err);
}
console.log("Found: " + JSON.stringify(docs))
})
});
As long as we include "category" everything is fine:
User Saved, Try to find him:
Mongoose: locations.find({ loc: { '$near': { '$geometry': { type: 'Point', coordinates: [ 12.5, 55.71 ] }, '$maxDistance': 600000 } }, category: 1 }, { fields: {} })
Found: [{"_id":"59f8f87554900a4e555d4e22","userName":"Tester","category":1,"__v":0,"loc":{"coordinates":[12.44,55.69],"type":"Point"}},{"_id":"59f8fabf50fcf54fc3dd01f6","userName":"Tester","category":1,"__v":0,"loc":{"coordinates":[12.44,55.69],"type":"Point"}}]
The alternate case is to simply "prefix" the index with the location. Making sure to drop previous indexes or the collection first:
LocationSchema.index({ loc: "2dsphere", category: 1 },{ "background": false });
As well as you probably should be in the habit of setting "background": true, else you start running into race conditions on unit tests where the index has not finished being created before unit test code attempts to use it.
My first solution to this problem was to create the index via the mLab web-interface which worked like a charm.
I have tried the solution suggested by Neil, but that still fails. The detailed instructions related to indexes, given by Neil however, did point me toward the solution to the problem.
It was a timing problem (which you not always see if you run the database locally) related to that my test code did the following:
Created the index, created a Location document (which first time will also create the collection), and then in the callback provided by save, I tried to find the user. It seems that the index was not yet created here, which is what gave the error.
If I delay the find method a second, using setTimeout it works fine.
But still, thanks to Neil for valuable information about the right way of using indexes (background) :-)
I'm trying to update a document which has a specific ID with the current date/time but the below code is not resulting in the DB getting updated and no errors. Any help would be great, thanks.
Schema:
var MerchantShema = new Schema({
merchant_id: Number,
merchant_aw_id: Number,
merchant_name: String,
merchant_url: String,
merchant_image: String,
product_feed: String,
product_feed_updated: Date,
created_at: {type: Date, default: Date.now},
updated_at: {type: Date, default: Date.now}
});
Update Query:
updateMerchantLastProductUpdate: function (mID) {
now = new Date();
var query = { "merchant_aw_id" : mID };
Merchants.update(query, { "product_feed_updated": now }, function (err) {
if (err) return console.error(err);
})
}
Route
app.get('/queries', function (req, res) {
queries.updateMerchantLastProductUpdate("2926");
});
Example document
{
"_id": {
"$oid": "55997638e4b01f0391cb99aa"
},
"merchant_id": "0003",
"merchant_aw_id": "2926",
"merchant_name": "Multipower",
"merchant_url": "www.multipower.com/uk/",
"merchant_image": "",
"product_feed": "aw",
"product_feed_updated": "",
"created_at": "",
"updated_at": ""
}
The merchant_aw_id field in your mongoose schema is expecting a number so you need to parse the string for integer by using the parseInt() method in your query. You also need the $set update operator which replaces the value of a field with the specified value to update your document, together with the {multi: true} option which if set to true, updates multiple documents that meet the query criteria. If set to false, updates one document. The default value is false:
updateMerchantLastProductUpdate: function (mID) {
var now = new Date(),
query = { "merchant_aw_id" : parseInt(mID) },
update = {
"$set": { "product_feed_updated": now }
},
options = { "multi": true };
Merchants.update(query, update, options, function (err) {
if (err) return console.error(err);
})
}
My error was caused by my model have the ID I was looking for in the format Number but my data in mongoDB was a String
Schema (../models/add.js)
var addSchema = new Schema({
name: {type: String, unique: true, sparse: true},
phone: Number,
email: String,
country: Number
});
module.exports = mongoose.model('Contact', addSchema);
add-manager.js
var Add = require('../models/add.js');
var AM = {};
var mongoose = require('mongoose');
module.exports = AM;
AM.notOwned = function(country, callback)
{
Add.update({country: country}, {country: country}, {upsert: true}, function(err, res){
if (err) callback (err);
else callback(null, res);
})
}
news.js
// if country # is not in the database
AM.notOwned(country, function(error, resp){
if (error) console.log("error: "+error);
else
{
// do stuff
}
})
error:
MongoError: E11000 duplicate key error index: bot.contacts.$name_1 dup key: { : null }
After seeing the error message, I googled around and learned that when the document is created, since name isn't set, its treated as null. See Mongoose Google Group Thread The first time AM.notOwned is called it will work as there isn't any documents in the collection without a name key. AM.notOwned will then insert a document with an ID field, and a country field.
Subsequent AM.notOwned calls fails because there is already a document with no name field, so its treated as name: null, and the second AM.notOwned is called fails as the field "name" is not set and is treated as null as well; thus it is not unique.
So, following the advice of the Mongoose thread and reading the mongo docs I looked at using sparse: true. However, its still throwing the same error. Further looking into it, I thought it may be the same issue as: this, but setting schema to name: {type: String, index: {unique: true, sparse: true}} doesn't fix it either.
This S.O. question/answer leads me to believe it could be caused by the index not being correct, but I'm not quite sure how to read the the db.collection.getIndexes() from Mongo console.
db.contacts.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "bot.contacts",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"name" : 1
},
"unique" : true,
"ns" : "bot.contacts",
"name" : "name_1",
"background" : true,
"safe" : null
}
]
What can I do to resolve this error?
You need to drop the old, non-sparse index in the shell so that Mongoose can recreate it with sparse: true the next time your app runs.
> db.contacts.dropIndex('name_1')