Mongodb error for duplicate indexes - node.js

I am new to MongoDB/Mongoose and am in the process of dumping my localdb in order to create a DB for heroku. When I run mongorestore I get the following error(s):
2016-08-29T22:05:00.411-0500 building a list of collections to restore from /Users/micahsherman/tmp/mongodump/Loc8r dir
2016-08-29T22:05:00.518-0500 reading metadata for heroku_n1kxxxxxx.locations from /Users/micahsherman/tmp/mongodump/Loc8r/locations.metadata.json
2016-08-29T22:05:00.518-0500 reading metadata for heroku_n1kxxxxxx.test from /Users/micahsherman/tmp/mongodump/Loc8r/test.metadata.json
2016-08-29T22:05:00.519-0500 restoring heroku_n1kxxxxxx.test from /Users/micahsherman/tmp/mongodump/Loc8r/test.bson
2016-08-29T22:05:00.519-0500 restoring heroku_n1kxxxxxx.locations from /Users/micahsherman/tmp/mongodump/Loc8r/locations.bson
2016-08-29T22:05:00.520-0500 restoring indexes for collection heroku_n1kxxxxxx.test from metadata
2016-08-29T22:05:00.574-0500 finished restoring heroku_n1kxxxxxx.test (0 documents)
2016-08-29T22:05:00.799-0500 error: multiple errors in bulk operation:
- E11000 duplicate key error index: heroku_n1kxxxxxx.locations.$_id_ dup key: { : ObjectId('57c334af05803d85c7b9e780') }
- E11000 duplicate key error index: heroku_n1kxxxxxx.locations.$_id_ dup key: { : ObjectId('57c3819605803d85c7b9e783') }
2016-08-29T22:05:00.799-0500 restoring indexes for collection heroku_n1kxxxxxx.locations from metadata
2016-08-29T22:05:00.852-0500 finished restoring heroku_n1kxxxxxx.locations (2 documents)
2016-08-29T22:05:00.852-0500 done
These prevent a connection from Heroku.
I have gone into the console and printed the indexes, here are the results:
rs-ds017886:PRIMARY> db.system.indexes.find();
{ "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "heroku_n1kxxxxxx.test" }
{ "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "heroku_n1kxxxxxx.locations" }
{ "v" : 1, "key" : { "coords" : "2dsphere" }, "name" : "coords_2dsphere", "ns" : "heroku_n1kxxxxxx.locations", "background" : true, "2dsphereIndexVersion" : 2 }
{ "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "heroku_n1kxxxxxx.objectlabs-system" }
{ "v" : 1, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "heroku_n1kxxxxxx.objectlabs-system.admin.collections" }
rs-ds017886:PRIMARY>
I'm not quite sure how to go about debugging this. Thoughts??

Double check your locations.bson file to ensure you don't have duplicated documents ('57c334af05803d85c7b9e780' & '57c3819605803d85c7b9e783'). Looks like you're adding data before you create your indexes.
Here's an example of what I believe to be happening (hard to tell without seeing the data)
> db.people.insert({name: "Matt"})
WriteResult({ "nInserted" : 1 })
> db.people.insert({name: "Matt"})
WriteResult({ "nInserted" : 1 })
> db.people.createIndex({name: 1}, {unique: true})
{
"ok" : 0,
"errmsg" : "E11000 duplicate key error index: test.people.$name_1 dup key: { : \"Matt\" }",
"code" : 11000
}
the _id field already contains the unique constraint

Related

$merge, $match and $update in one aggregate query

I have data in a collection ex:"jobs". I am trying to copy specific data from "jobs" after every 2 hours to a new collection (which may not exist initially) and also add a new key to the copied data.
I have been trying with this query to copy the data:
db.getCollection("jobs").aggregate([{ $match: { "job_name": "UploadFile", "created_datetime" : {"$gte":"2021-08-18 12:00:00"} } },{"$merge":{into: {coll : "reports"}}}])
But after this, the count in "reports" collection is 0. Also, how can I update the documents (with an extract key "report_name") without using an extra updateMany() query?
The data in jobs collection is as shown:
{
"_id" : ObjectId("60fa8e8283dc22799134dc6f"),
"job_id" : "408a5654-9a89-4c15-82b4-b0dc894b19d7",
"job_name" : "UploadFile",
"data" : {
"path" : "share://LOCALNAS/Screenshot from 2021-07-23 10-34-34.png",
"file_name" : "Screenshot from 2021-07-23 10-34-34.png",
"parent_path" : "share://LOCALNAS",
"size" : 97710,
"md5sum" : "",
"file_uid" : "c4411f10-a745-48d0-a55d-164707b7d6c2",
"version_id" : "c3dfd31a-80ba-4de0-9115-2d9b778bcf02",
"session_id" : "c4411f10-a745-48d0-a55d-164707b7d6c2",
"resource_name" : "Screenshot from 2021-07-23 10-34-34.png",
"metadata" : {
"metadata" : {
"description" : "",
"tag_ids" : [ ]
},
"category_id" : "60eed9ea33c690a0dfc89b41",
"custom_metadata" : [ ]
},
"upload_token" : "upload_token_c5043927484e",
"upload_url" : "/mnt/share_LOCALNAS",
"vfs_action_handler_id" : "91be4282a9ad5067642cdadb75278230",
"element_type" : "file"
},
"user_id" : "60f6c507d4ba6ee28aee5723",
"node_id" : "syeda",
"state" : "COMPLETED",
"priority" : 2,
"resource_name" : "Screenshot from 2021-07-23 10-34-34.png",
"group_id" : "upload_group_0babf8b7ce0b",
"status_info" : {
"progress" : 100,
"status_msg" : "Upload Completed."
},
"error_code" : "",
"error_message" : "",
"created_datetime" : ISODate("2021-07-23T15:10:18.506Z"),
"modified_datetime" : ISODate("2021-07-23T15:10:18.506Z"),
"schema_version" : "1.0.0",
}
Your $match stage contains a condition which takes created_datetime as string while in your sample data it is an ISODate. Such condtion won't return any document, try:
{
$match: {
"job_name": "UploadFile",
"created_datetime": {
"$gte": ISODate("2021-07-01T12:00:00.000Z")
}
}
}
Mongo Playground

Query an array in MongoDB

I have this collection in MongoDB:
{
"_id" : ObjectId("5df013b10a88910018267a89"),
"StockNo" : "33598",
"Description" : "some description",
"detections" : [
{
"lastDetectedOn" : ISODate("2020-01-29T04:36:41.191+0000"),
"lastDetectedBy" : "comp-t",
"_id" : ObjectId("5e3135f68c9e930017de8aec")
},
{
"lastDetectedOn" : ISODate("2019-12-21T18:12:06.571+0000"),
"lastDetectedBy" : "comp-n",
"_id" : ObjectId("5e3135f68c9e930017de8ae9")
},
{
"lastDetectedOn" : ISODate("2020-01-29T07:36:06.910+0000"),
"lastDetectedBy" : "comp-a",
"_id" : ObjectId("5e3135f68c9e930017de8ae7")
}
],
"createdAt" : ISODate("2019-12-10T21:52:49.788+0000"),
"updatedAt" : ISODate("2020-01-29T07:36:22.950+0000"),
"__v" : NumberInt(0)
}
I want to search by StockNo and get the name of the computer that last detected it (lastDetectedBy) only if lastDetectedOn was in the last 5 minutes with Mongoose in node.js with Express.
I also have this collection:
{
"_id" : ObjectId("5df113b10d35670018267a89"),
"InvoiceNo" : "1",
"InvoiceDate" : ISODate("2020-01-14T02:18:11.196+0000"),
"InvoiceContact : "",
"isActive" : true
},
{
"_id" : ObjectId("5df013c90a88910018267a8a"),
"InvoiceNo" : "2",
"InvoiceDate" : ISODate("2020-01-14T02:18:44.279+0000"),
"InvoiceContact : "Bob Smith",
"isActive" : true
},
{
"_id" : ObjectId("5e3096bb8c9e930017dc6e20"),
"InvoiceNo" : "3",
"InvoiceDate" : ISODate("2020-01-14T02:19:50.155+0000"),
"InvoiceContact : "",
"isActive" : true
}
And I want to update all the documents with empty InvoiceContact which has been issued in the last 30 seconds (or any date range between now and sometime in the past) with isActive equals true to isActive equals false. So for example, the first record has been issued in the last 30 seconds without InvoiceContact and isActive is true so this must be updated but the next two records will remain untouched for different reasons, the second record has InvoiceContact and the third record is out of range.
First Part
var mins5 = new Date(ISODate() - 1000* 60 * 5 )
db.getCollection('user').find({$and:[
{ "StockNo":"33598"},
{"detections.lastDetectedOn" : { $gte : mins5 }}
]})
.map(function(list){
var results = [];
list.detections.forEach(function (detections){
if(detections.lastDetectedOn > mins5){
results.push(detections.lastDetectedBy);
}
})
return results;
});
Second Part could be solved by a similar query using update instead of find.

MongoDB + NodeJS: Document failed validation & Data Types behaviour

I am new to MongoDB and NodeJS,
When i try to create the JsonSchema with data types, string, integer, date and bool, it is created but always throwing an error as document validation error while inserting the data, So i changed the bsonType of one data type to number, then it started creating collection records, but the observation is it is storing as Double datatype, I read somewhere in the stackoverflow, that it stores like that only, but my question is why is this behavior? WHY THE ERROR IS NOT BEING THROWN AT THE TIME OF CREATION OF THE JSONSCHEMA but it is throwing at the time of data insertion?
Also, if we have nested objects let us say, Customer object with Address as nested object, the main object's int/number values are stored as Double where as inside the address object's pincode storing as Int32. This is also very confusing. what is the difference between these objects but the structure of the schema is same.
What are the other ways to implement and having proper validated schema for MongoDB.
>
db.getCollectionInfos({name:"companysInt1s1"})
[
{
"name" : "companysInt1s1",
"type" : "collection",
"options" : {
"validator" : {
"$jsonSchema" : {
"bsonType" : "object",
"required" : [
"tin"
],
"properties" : {
"tin" : {
"bsonType" : "int",
"minLength" : 2,
"maxLength" : 11,
"description" : "must be a string and is not required, should be 11 characters length"
}
}
}
}
},
"info" : {
"readOnly" : false,
"uuid" : UUID("27cba650-7bd3-4930-8d3e-7e6cbbf517db")
},
"idIndex" : {
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "invoice.companysInt1s1"
}
}
]
> db.companysInt1s1.insertOne({tin:22222})
2019-02-14T15:04:28.712+0530 E QUERY [js] WriteError: Document failed validation :
WriteError({
"index" : 0,
"code" : 121,
"errmsg" : "Document failed validation",
"op" : {
"_id" : ObjectId("5c653624e382c2ec16c16893"),
"tin" : 22222
}
})
WriteError#src/mongo/shell/bulk_api.js:461:48
Bulk/mergeBatchResults#src/mongo/shell/bulk_api.js:841:49
Bulk/executeBatch#src/mongo/shell/bulk_api.js:906:13
Bulk/this.execute#src/mongo/shell/bulk_api.js:1150:21
DBCollection.prototype.insertOne#src/mongo/shell/crud_api.js:252:9
#(shell):1:1
Am i missing something or any other documentation should i be following? Appreciate your guidance...
You need to insert as NumberInt.
when you run this
db.companysInt1s1.insertOne({tin:22222})
you are actually inserting tin as float.
so the correct way to do it is
db.companysInt1s1.insertOne({tin: NumberInt(22222) })

mongodb taking too much time for old entries

i am new in mongodb and i am facing an issue, i have around millions of documents in my collectionand i am trying to find single entry using findOne({}) command and when i am trying to find recent entries then response comes in miliseconds but when i am trying to fetch older entries around 600 millionth document then it takes around 2 minutes on mongo shell and my node server gives
{ MongoErro : connection 1 to 127.0.0.1:27017 timed out }
and my nodejs server sends an empty response. can any one tell me what should i do to resolve this issueThanks in advance
explain gives me
db.contacts.find({"phoneNumber":"9165900137"}).explain("executionStats")
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "meanApp.contacts",
"indexFilterSet" : false,
"parsedQuery" : {
"phoneNumber" : {
"$eq" : "9165900137"
}
},
"winningPlan" : {
"stage" : "COLLSCAN",
"filter" : {
"phoneNumber" : {
"$eq" : "9165900137"
}
},
"direction" : "forward"
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 321188,
"totalKeysExamined" : 0,
"totalDocsExamined" : 495587806,
"executionStages" : {
"stage" : "COLLSCAN",
"filter" : {
"phoneNumber" : {
"$eq" : "9165900137"
}
},
"nReturned" : 1,
"executionTimeMillisEstimate" : 295230,
"works" : 495587808,
"advanced" : 1,
"needTime" : 495587806,
"needYield" : 0,
"saveState" : 3871779,
"restoreState" : 3871779,
"isEOF" : 1,
"invalidates" : 0,
"direction" : "forward",
"docsExamined" : 495587806
}
},
"serverInfo" : {
"host" : "li1025-15.members.linode.com",
"port" : 27017,
"version" : "3.2.16",
"gitVersion" : "056bf45128114e44c5358c7a8776fb582363e094"
},
"ok" : 1
}
As indicated in the explain plan results, the current query is doing Collection Scan. This means it has to scan every document in collection to produce the match and you have got about half a billion documents.
Try adding this index and it might take a bit to create it.
db.contacts.createIndex( { phoneNumber: 1 }, { background: true } )
Run the query once the index creation is successful, you must see a dramatic improvement in performance. To be certain whether index got picked up, try explain again and it should no longer say COLLSCAN.

how to prevent auto deletion collection records from Mongodb

Hello all I got stuck somewhere, I am working on mongodb with node.js where my collection data deleted automatically after 1 year on certain date and I want to stop that permanently how can I do that ? I have checked the available material on google but didn't got much success please help me friends ...
I have checked the index in one of my collection and it is showing data like this . Can you please tell me its is having TTL index or not
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "firstfive.teachers"
},
{
"v" : 1,
"key" : {
"_fts" : "text",
"_ftsx" : 1
},
"name" : "firstname_lastname_text",
"weights" : {
"firstName" : 1,
"lastName" : 1
},
"default_language" : "english",
"language_override" : "language",
"ns" : "firstfive.teachers",
"textIndexVersion" : 2
}
]
most likely you have TTL (time to limit) index defined on collection you're working with (https://docs.mongodb.com/v3.2/core/index-ttl/)
yu can check it by running db.your_collection.getIndexes() (it will be one with expireAfterSeconds) in mongo shell.
as any other index it can be removed - but do it carefully, apparently someone did it deliberately

Resources