I have an index 'user' which has a mapping with field of "first", "last", and "email". The fields with names get indexed at one point, and then the field with the email gets indexed at a separate point. I want these indices to have the same id though, corresponding to one user_id parameter. So something like this:
function indexName(client, id, name) {
return client.update({
index: 'user',
type: 'text',
id: id,
body: {
first: name.first
last: name.last
}
})
}
function indexEmail(client, id, email) {
return client.update({
index: 'user',
type: 'text',
id: id,
body: {
email: email
}
})
}
When running:
indexName(client, "Jon", "Snow").then(indexEmail(client, "jonsnow#gmail.com"))
I get an error message saying that the document has not been created yet. How do I account for a document with a variable number of fields? And how do I create the index if it has not been created and then subsequently update it as I go?
The function you are using, client.update, updates part of a document. What you actually needs is to first create the document using the client.create function.
To create and index, you need the indices.create function.
About the variable number of fields in a document type, it is not a problem because Elastic Search support dynamic mapping. However, it would be advisable to provide a mapping when creating the index, and try to stick to it. Elastic Search default mapping can create you problems later on, e.g. analyzing uuids or email addresses which then become difficult (or impossible) to search and match.
Related
I am designing an item inventory system for a website that I am building.
The user's inventory is loaded from a Web API. This information is then processed so that it is more suited to my web app. I am trying to combine all the item records into one MongoDB collection - so other user inventories will be cached in the same place. What I have to deal with is deleting old item records if they are missing from the user's inventory (i.e. they sold it to someone) and also upserting the new items. Please note I have looked through several Stack Overflow questions about bulk upserts but I was unable to find anything about conditional updates.
Each item has two unique identifiers (classId and instanceId) that allow me to look them up (I have to use both IDs to match it) which remain constant. Some information about the item, such as its name, can change and therefore I want to be able to update those records when I fetch new inventory information. I also want new items that my site hasn't seen before to be added to my database.
Once the data returned from the Web API has been processed, it is left in a large array of objects. This means I am able to use bulk writing, however, I am unaware of how to upsert with conditions with multiple records.
Here is part of my item schema:
const ItemSchema = new mongoose.Schema({
ownerId: {
type: String,
required: true
},
classId: {
type: String,
required: true
},
instanceId: {
type: String,
required: true
},
name: {
type: String,
required: true
}
// rest of item attributes...
});
User inventories typically contain 600 or more items, with a max count of 2500.
What is the most efficient way of upserting this much data? Thank you
Update:
I have had trouble implementing the solution to the bulk insert problem. I made a few assumptions and I don't know if they were right. I interpreted _ as lodash, response.body as the JSON returned by the API and myListOfItems also as that same array of items.
import Item from "../models/item.model";
import _ from 'lodash';
async function storeInventory(items) {
let bulkUpdate = Item.collection.initializeUnorderedBulkOp();
_.forEach(items, (data) => {
if (data !== null) {
let newItem = new Item(data);
bulkUpdate.find({
classId: newItem.classId,
instanceId: newItem.instanceId
}).upsert().updateOne(newItem);
items.push(newItem);
}
});
await bulkUpdate.execute();
}
Whenever I run this code, it throws an error that complains about an _id field being changed, when the schema objects I created don't specify anything to do with schemas, and the few nested schema objects don't make a difference to the outcome when I change them to just objects.
I understand that if no _id is sent to MongoDB it auto generates one, but if it is updating a record it wouldn't do that anyway. I also tried setting _id to null on each item but to no avail.
Have I misunderstood anything about the accepted answer? Or is my problem elsewhere in my code?
This is how I do it :
let bulkUpdate = MyModel.collection.initializeUnorderedBulkOp();
//myItems is your array of items
_.forEach(myItems, (item) => {
if (item !== null) {
let newItem = new MyModel(item);
bulkUpdate.find({ yyy: newItem.yyy }).upsert().updateOne(newItem);
}
});
await bulkUpdate.execute();
I think the code is pretty readable and understandable. You can adjust it to make it work with your case :)
I have a table called followProduct in Dynamodb and it has following strucure
id - item id
email - user email
product - product id
Whenever a user follows a product I am making an entry in the table. I am trying to stop duplicate entry and using the following code
let params = {
TableName: "followProduct",
ConditionExpression: "email <> :email AND product <> :pid",
Item: {
email: "a#a.com",
product: req.body.productId,
id: shortid.generate()
},
ExpressionAttributeValues: {
':email': "a#a.com",
":pid": req.body.productId
}
};
createItemInDDB(params).then(() => {
res.status(200).send("Company Added");
}, err => {
console.log(err);
res.sendStatus(500);
});
CreateItemInDDB is just a function that takes params as input and run put function provided by document client. This params is still making a duplicate entry. I want that for every email each product id should be entered only once.
can you describe your table hash-range keys?
Dynamodb can force uniqueness only for hash-range table keys (not for global secondary index keys)
from http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
To prevent a new item from replacing an existing item, use a conditional expression that contains the attribute_not_exists function with the name of the attribute being used as the partition key for the table. Since every record must contain that attribute, the attribute_not_exists function will only succeed if no matching item exists.
and http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html:
The PutItem operation will overwrite an item with the same key (if it exists). If you want to avoid this, use a condition expression. This will allows the write to proceed only if the item in question does not already have the same key:
I am trying to stop duplicates in my Mongo DB Collection but they are still getting in. I am reading data from twitter and storing it like:
var data = {
user_name: response[i].user.screen_name,
profile_image: response[i].user.profile_image_url,
content: {
text: response[i].text
},
id: response[i].id_str,
};
and I have the following to stop any duplicates:
db[collection].ensureIndex( { id: 1, "content.text": 1 }, { unique: true, dropDups: true } );
The id field is working and no duplicates appear but "content.text" field does not work and duplicates are appearing. Any Ideas why?
When you enforce a unique constraint on a composite index, two documents are considered same only if the documents have the same value for both id and context.text fields and not for either key individually.
To enforce unique constraints on the fields, id and context.text individually, You could enforce it as below:
db.col.ensureIndex({"id":1},{unique:true}) and similarly for the other field.
Is it possible to limit available displayed options in a relationship type of KeystoneJS by specifying a value condition?
Basically, a model has two sets of array fields, instead of letting the admin user select any item from the field, I would like to restrict to only the items that are part of a specific collection _id.
Not sure if this is exactly the feature you're looking for, but you can specify a filter option on the Relationship field as an object and it will filter results so only those that match are displayed.
Each property in the filter object should either be a value to match in the related schema, or it can be a dynamic value matching the value of another path in the schema (you prefix the path with a :).
For example:
User Schema
User.add({
state: { type: Types.Select, options: 'enabled, disabled' }
});
Post Schema
// Only allow enabled users to be selected as the author
Post.add({
author: { type: Types.Relationship, ref: 'User', filter: { state: 'enabled' } }
});
Or for a dynamic example, imagine you have a role setting for both Posts and Users. You only want to match authors who have the same role as the post.
User Schema
User.add({
userRole: { type: Types.Select, options: 'frontEnd, backEnd' }
});
Post Schema
Post.add({
postRole: { type: Types.Select, options: 'frontEnd, backEnd' },
// only allow users with the same role value as the post to be selected
author: { type: Types.Relationship, ref: 'User', filter: { userRole: ':postRole' } }
});
Note that this isn't actually implemented as back-end validation, it is just implemented in the Admin UI. So it's more of a usability enhancement than a restriction.
To expand on Jed's answer, I think the correct property (at least in the latest version of KeystoneJS 0.2.22) is 'filters' instead of 'filter'. 'filter' doesn't work for me.
How can I add a new row with the update operation
I am using following code
statuscollection.update({
id: record.id
}, {
id: record.id,
ip: value
}, {
upsert: true
}, function (err, result) {
console.log(err);
if (!err) {
return context.sendJson([], 404);
}
});
While calling this first one i will add the row
id: record.id
Then id:value then i have to add id:ggh
How can i add every new row by calling this function for each document I need to insert
By the structure of your code you are probably missing a few concepts.
You are using update in a case where you probably do not need to.
You seem to be providing an id field when the primary key for MongoDB would be _id. If that is what you mean.
If you are intending to add a new document on every call then you probably should be using insert. Your use of update with upsert has an intended usage of matching a document with the query criteria, if the document exists update the fields as specified, if not then insert a new document with the fields specified.
Unless that actually is your goal then insert is most certainly what you need. In that case you are likely to rely on the value of _id being populated automatically or by supplying your own unique value yourself. Unless you specifically want another field as an identifier that is not unique then you will likely want to be using the _id field as described before.