Now I'd like to save my json data into mongoose but the duplicate value had to be filtered.
my_json = [
{"name":"michael","age":21,"sports":"basketball"},
{"name":"nick","age":31,"sports":"golf"},
{"name":"joan","age":41,"sports":"soccer"},
{"name":"henry","age":51,"sports":"baseball"},
{"name":"joe","age":61,"sports":"dance"},
];
Database data is :
{
"name":"joan","age":41,"sports":"soccer"
}
Is there some specific method to avoid duplicate data insert to mongoose directly? It might be saved 4 of values except "joan" value.
Once I suppose to try to use "for statement", it was fine.
However I just want to make a simple code for that what could happen in a variety possible code.
for(var i = 0; i < my_json.length; i++){
// to check out duplicate value
db.json_model.count({"name":my_json[i].name}, function(err, cat){
if(cat.length == 0){
my_json_vo.savePost(function(err) {
});
}
})
};
As you see I need to use count method whether the value is duplicated or not. I don't want to use count method but make it more simple..
Could you give me an advice for that?
You can mark field as unique in mongoose schema:
var schema = new Schema({
name: {type: String, required: true, unique: true}
//...
});
Also, you can add unique index for name field into your database:
db.js_model.createIndex( {"name": 1}, { unique: true, background: true } );
then, if new entity with the same name will be asked to save - mongo won't save it, and respond an error.
In Addition to #Alex answer about adding unique key on the name field.
You can use insertMany() method with ordered parameter set to
false. Like this...
let my_json = [
{"name":"michael","age":21,"sports":"basketball"},
{"name":"nick","age":31,"sports":"golf"},
{"name":"joan","age":41,"sports":"soccer"},
{"name":"henry","age":51,"sports":"baseball"},
{"name":"joe","age":61,"sports":"dance"},
];
User.insertMany(my_json ,{ordered :false});
This query will successfully run and insert unique documents, And also
produces error later after successful insertion. So You will come to
know that there were duplicate records But now in the database, all
records are unique.
Reference InsertMany with ordered parameter
Related
I have a collection of documents which are being added as a result of users' interactions.
Those docs already have an _id field, but I also wanna add a unique human readable ID for every existing and newly created object, in a form of D123456
What is the best way of adding such an ID and being sure that all those IDs are unique?
MongoDB doesn't have an auto-increment option like relational databases.
You can implement something yourself: before you save your document, generate an ID. First, create a database collection whose sole purpose is to hold a counter:
const Counter = mongoose.model('Counter', new mongoose.schema({
current: Number
}));
Second, before you save your object, find and increment the number in the collection:
const humanReadableDocumentId = await Counter.findOneAndUpdate(
// If you give this record a name, you can have multiple counters.
{ _id: 'humanReadableDocumentId' },
{ $inc: { current: 1 } },
// If no record exists, create one. Return the new value after updating.
{ upsert: true, returnDocument: 'after' }
);
const yourDocument.set('prettyId', format(humanReadableDocumentId.current));
function format(id) {
// Just an example.
return 'D' + id.toString().padStart(6, '0');
}
Note: I've tested the query in MongoDB (except for the 'returnDocument' option, which is Mongoose-specific, but this should work)
Formatting is up to you. If you have more than 999999 documents, the 'nice looking ID' in the example will just get longer and be 7+ characters.
I am designing an item inventory system for a website that I am building.
The user's inventory is loaded from a Web API. This information is then processed so that it is more suited to my web app. I am trying to combine all the item records into one MongoDB collection - so other user inventories will be cached in the same place. What I have to deal with is deleting old item records if they are missing from the user's inventory (i.e. they sold it to someone) and also upserting the new items. Please note I have looked through several Stack Overflow questions about bulk upserts but I was unable to find anything about conditional updates.
Each item has two unique identifiers (classId and instanceId) that allow me to look them up (I have to use both IDs to match it) which remain constant. Some information about the item, such as its name, can change and therefore I want to be able to update those records when I fetch new inventory information. I also want new items that my site hasn't seen before to be added to my database.
Once the data returned from the Web API has been processed, it is left in a large array of objects. This means I am able to use bulk writing, however, I am unaware of how to upsert with conditions with multiple records.
Here is part of my item schema:
const ItemSchema = new mongoose.Schema({
ownerId: {
type: String,
required: true
},
classId: {
type: String,
required: true
},
instanceId: {
type: String,
required: true
},
name: {
type: String,
required: true
}
// rest of item attributes...
});
User inventories typically contain 600 or more items, with a max count of 2500.
What is the most efficient way of upserting this much data? Thank you
Update:
I have had trouble implementing the solution to the bulk insert problem. I made a few assumptions and I don't know if they were right. I interpreted _ as lodash, response.body as the JSON returned by the API and myListOfItems also as that same array of items.
import Item from "../models/item.model";
import _ from 'lodash';
async function storeInventory(items) {
let bulkUpdate = Item.collection.initializeUnorderedBulkOp();
_.forEach(items, (data) => {
if (data !== null) {
let newItem = new Item(data);
bulkUpdate.find({
classId: newItem.classId,
instanceId: newItem.instanceId
}).upsert().updateOne(newItem);
items.push(newItem);
}
});
await bulkUpdate.execute();
}
Whenever I run this code, it throws an error that complains about an _id field being changed, when the schema objects I created don't specify anything to do with schemas, and the few nested schema objects don't make a difference to the outcome when I change them to just objects.
I understand that if no _id is sent to MongoDB it auto generates one, but if it is updating a record it wouldn't do that anyway. I also tried setting _id to null on each item but to no avail.
Have I misunderstood anything about the accepted answer? Or is my problem elsewhere in my code?
This is how I do it :
let bulkUpdate = MyModel.collection.initializeUnorderedBulkOp();
//myItems is your array of items
_.forEach(myItems, (item) => {
if (item !== null) {
let newItem = new MyModel(item);
bulkUpdate.find({ yyy: newItem.yyy }).upsert().updateOne(newItem);
}
});
await bulkUpdate.execute();
I think the code is pretty readable and understandable. You can adjust it to make it work with your case :)
Is it possible to add a default value if the field is empty/not present in the record?
ex:
if ( ! record.options ) { record.options = {}; }
After searching a lot, the response i found was i can do this using .toObject() or .toJSON(), But i am not calling any of these and i just want to add this conditions on the schema so it works directly.
Right now i am checking for this condition in the returned record/(s) which is bad as i am repeating the same logic for .find and .findOne
If you want to add empty default value, you need to do something like this:
var schema = new Schema({
info: { type: <YOUR_TYPE_HERE>, default: {} }
}, { minimize: false });
I am maintaining a collection of unique values that has a companion collection that has instances of those values. The reason I have it that way is the companion collection has >10 million records where the unique values collection only add up to 100K and I use those values all over the place and do partial match lookups.
When I upload a csv file it is usually 10k to 500k records at a time that I insert into the companion collection. What is the best way to insert only values that dont already exist into the unique values collection?
Example:
//Insert large quantities of objects into mongo
var bulkInsert = [
{
name: "Some Name",
other: "zxy",
properties: "abc"
},
{
name: "Some Name",
other: "zxy",
properties: "abc"
},
{
name: "Other Name",
other: "zxy",
properties: "abc"
}]
//Need to insert only values that do not already exist in mongo unique values collection
var uniqueValues = [
{
name:"Some Name"
},
{
name:"Other Name"
}
]
EDIT
I tried creating a unique index on the field, but once it finds a duplicate in the Array of documents that I am inserting, it stops the whole process and doesnt proceed to check any values after the break.
Figured it out. If your doing it from the shell, you need to use Bulk() and create insert jobs like this:
var bulk = db.collection.initializeUnorderedBulkOp();
bulk.insert( { name: "1234567890a"} );
bulk.insert( { name: "1234567890b"} );
bulk.insert( { name: "1234567890"} );
bulk.execute();
and in node, the continueOnError flag works on a straight collection.insert()
collection.insert( [{name:"1234567890a"},{name:"1234567890c"}],{continueOnError:true}, function(err, doc){}
Well, I think the solution here is quite simple if I understand correctly your issue.
Since the process is stopped when it finds a duplicated field you should basically check if the value doesn't already exists before to try to add it.
So, for each element in uniqueValues, make a find/findOne query, if it doesn't return any result then add the element, otherwise don't.
What is the best way to update a value within an array saved in a mongodb record? Currently, I'm trying it this way:
Record.find({ 'owner': owner}, {}, {sort: { date: -1 }}, function(err, record){
if(!err){
for (var i = 0; i < record[0].array.length; i++){
record[0].array[i].score = 0;
record[0].array[i].changed = true;
record[0].save();
}
}
});
And the schema looks like this:
var recordSchema = mongoose.Schema({
owner: {type: String},
date: {type: Date, default: Date.now},
array: mongoose.Schema.Types.Mixed
});
Right now, I can see that the array updates, I get no error in saving, but when I query the database again, the array hasn't been updated.
It would help if you explained your intent here as naming a property "array" conveys nothing about its purpose. I guess from your code you hope to go and set the score of each item there to zero. Note your save is currently being ignored because you can only save top-level mongoose documents, not nested documents.
Certain find-and-modify operations on arrays can be done with a single database command using the Array Update Operators like $push, $addToSet, etc. However I don't see any operators that can directly make your desired change in a single operation. Thus I think you need to find your record, alter the array date, and save it. (Note findOne is a convenience function you can use if you only care about the first match, which seems to be the case for you).
Record.findOne({ 'owner': owner}, {}, {sort: { date: -1 }}, function(err, record){
if (err) {
//don't just ignore this, log or bubble forward via callbacks
return;
}
if (!record) {
//Record not found, log or send 404 or whatever
return;
}
record.array.forEach(function (item) {
item.score = 0;
item.changed = true;
});
//Now, mongoose can't automatically detect that you've changed the contents of
//record.array, so tell it
//see http://mongoosejs.com/docs/api.html#document_Document-markModified
record.markModified('array');
record.save();
});
If you have a mongoose object of a document, you can of course update the array as in the question, with the following Caveat.
This is in fact a mongoose gotcha. Mongoose cannot track changes in the array of mixed, one has to use markModified:
doc.mixed.type = 'changed';
doc.markModified('mixed.type');
doc.save() // changes to mixed.type are now persisted