I'm using mongify to migrate a mysql database into mongodb.
Doing that, 2 questions appeared:
1- How can i declare my translation file in order to have a embedded array of ids that references to the objects (that are stored in a different collection and can be retrieved through populate), instead of just embedding as json objects.
2- Embedded objects can have an unique id as objects in colections do?. On other projects i've used that approach to query for embedded objects, but if that id is not present i should use a different field.
Unfortunately the first request isn't possible with Mongify at the moment, it requires a custom script to do that.
I could give you more details if you want to send me your translation file (Make sure to remove any sensitive data).
As for number two, the embedded object will get a unique ID. You don't need to do anything special.
Hope that answers your questions.
from mongify isn't possible but in mongodb you can transform data as follows:
//find posts has array of objects
db.getCollection('posts').find({'_tags.0': {$exists: true}}).forEach( function (post) {
var items = [];
var property = '_tags';
post[property].forEach(function(element){
if(element._id !== undefined){
items.push(element._id);
}
});
if(items.length>0){
post[property] = items;
db.posts.update({_id:post._id},post);
}
});
Source Document:
{
"_id" : ObjectId("576aa0389863482f64051c81"),
"id_post" : 130155,
"_tags" : [
{
"_id" : ObjectId("576a9efd9863482f64000044")
},
{
"_id" : ObjectId("576a9efd9863482f6400004b")
},
{
"_id" : ObjectId("576a9efd9863482f64000052")
},
{
"_id" : ObjectId("576a9efd9863482f6400005a")
}
]
}
Final Document:
{
"_id" : ObjectId("576aa0389863482f64051c81"),
"id_post" : 130155,
"_tags" : [
ObjectId("576a9efd9863482f64000044"),
ObjectId("576a9efd9863482f6400004b"),
ObjectId("576a9efd9863482f64000052"),
ObjectId("576a9efd9863482f6400005a")
]
}
Related
I have a requirement to generate a unique number (ARN) in this format
DD/MM/YYYY/1, DD/MM/YYYY/2
and insert these in elastic search index.
The approach i am thinking of is to create an auto increment field in the doc and use it to generate a new entry and use the new auto generated number to create the ARN and update the doc.
doc structure that i am planning to use:
{ id: 1, arn: 17/03/2018/01 }
something like this.
How can i get auto increment field in elastic search?
It can't be done in a single step. First you have to insert the record into the database, and then update the ARN with it's id
There is no auto-increment equivalent, for example, to hibernate id generator. You could use the Bulk API (if you have to save multiple documents at a time) and increase the _id and the ending of your ARN value programmatically.
Note: if you want to treat your id as a number, you should implement it yourself (in this example, I added a new field "my_id", because the _id of the documents is treated as a string.
POST /bulk
{ "index" : { "_index" : "your_index", "_type" : "your_type", "_id" : "1" } }
{ "arn" : "2018/03/17/1", my_id: 1 }
{ "index" : { "_index" : "your_index", "_type" : "your_type", "_id" : "2" } }
{ "arn" : "2018/03/17/2", my_id: 2 }
Then, the next time that you want to save new documents, you query for the maximum id something like:
POST /my_index/my_type/_search?size=1
{
"query": {
"fields": ["my_id"],
"sort": [{
"my_id": { "order": "desc" } }
]
}
}
If your only requirement is that this ARN should be unique, you could also let elasticsearch calculate your _id by simply not setting it. Then you could relay at some unique token generator (UID.randomUUID().toString() if work with java). Pseudo code follows:
String uuid = generateUUID() // depends on the programming language
String payload = "{ \"arn\" : + uuid + "}" // concatenate the payload
String url = "http://localhost:9200/my_index" // your target index
executePost(url, payload) // implement the call with some http client library
I know the mongoose-encryption doc states:
update will work fine on unencrypted and unauthenticated fields, but will not work correctly if encrypted or authenticated fields are involved.
And I've observed that when I use the mongoose create method that my fields are encrypted into the _ct field. However if I then use findByIdAndUpdate to update my object I see the fields are created in plain text (as output from mongodb console via find command).
From save
> db.tenants.find().pretty()
{
"_id" : ObjectId("554b7f8e7806c204e0c7589e"),
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0
}
After findByIdAndUpdate
> db.tenants.find().pretty()
{
"_id" : ObjectId("554b7f8e7806c204e0c7589e"),
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0,
"userId" : ObjectId("55268f43cbfc87be221cd611"),
"social" : "123-45-6789",
"last" : "bar",
"first" : "foo"
}
Is there a recommended strategy for updating objects and maintaining the encryption with mongoose-encryption?
As you quoted, the documentation for mongoose-encryption clearly tells that it does not work for update.
https://github.com/joegoldbeck/mongoose-encryption
Mongoose update hook is little tricky as well.
What you can do potentially is model your collection in such a way that fields which needs to be encrypted are a separate collection altogether and in the paren collection just link them via ids.
Person = {
_id: <ObjectId>
name: Blah
..
..
documents: [
{ 'doc_id': <ObjectId1> },
{ 'doc_id': <ObjectId2> },
]
}
Documents = [
{
"_id" : <ObjectId1>,
"_ac" : BinData(0,"YdJjOUJhzDWuDE5oBU4SH33O4qM2hbotQTsF6NzDnx4hWyJfaWQiLCJfY3QiXQ=="),
"_ct" : BinData(0,"YaU4z/UY3djGCKBcgMaNIFHeNp8NJ9Woyh9ahff0hRas4WD80V80JE2B8tRLUs0Qd9B7IIzHsq6O4pYub5VKJ1PIQA+/dbStZpOH/KfvPoDC6DzR5JdoAu+feU7HyFnFCMY81RZeJF5BKJylhY1+mG4="),
"__v" : 0
}
...
...
]
This will increase code reuse as well.
I have implemented an strategy that i don´t think it is most efficient but it works.
I need to have all my data in database encrypted so i can´t use the above approach.
What i did is to create an update function that finds the document i want to modify, then i construct a new schema object and assing the _id of the found document to the new object.
Then i delete the original document and after that save the new object wich has the original _id. The only problem i found is that mongoose throw an error because duplicated _id that is printed in the console but it still works and _id aren´t duplicated.
I have tried replacing the_id and traking the document with another property but it still throw that error, anyway data is stored as expected.
exports.update= (req, res, next) => {
Solucion.findOne({_id: req.params.id})
.then(document => {
if (!document) {
res.status(404).json({
message: notFoundMessage,
data: null,
error: null
})
} else {
const solucion = new Solucion({
_id: document._id,
identificacion: document.identificacion,
informacion: document.informacion,
estado: req.body
})
Solucion.deleteOne({_id: document._id})
.then(() => {return solucion.save()})
.then(result=> {
return res.status(201).json({
message: editedSavedMessage,
data: result,
error: null
});
})
.catch(err => {
errorHandler.errorHandler(err, res);
})
}
})
};
UPDATE 29/07/2020
I have found that if you use the save method using the same _id, data is stored encrypted but Mongo creates your schema structure but with all values set to null.
Beyond that it seems to work as expected as data is not visible in DB.
consider the following records:
user record
{
"_id" : ObjectId("5234ccb7687ea597eabee677"),
"class" : [
{ "_id" : ObjectId("5234ccb7687ea597eabee671", "num" : 10, "color" : "blue" },
{ "_id" : ObjectId("5234ccb7687ea597eabee672", "num" : 100, "color" : "blue" }
]
}
this user has two class sub records, now I need a query that finds all users that have class property where "class._id" has a value of at least one users "class._id"
here is a more detail example:
suppose there is four user:
A:{_id:432645624232345,class:[{_id:123,name:'foo'}]}
B:{_id:432645624232555,class:[{_id:555,name:'foo'},{_id:123,name:'foo'}]}
C:{_id:432645344232345,class:[{_id:555,name:'foo'},{_id:111,name:'www'}]}
D:{_id:432644444232345,class:[{_id:222,name:'sss'},{_id:555,name:'www'},{_id:123,name:'foo'}]}
now if B login , I need to query all the user whose class subdocument contains at least one document which's _id==555 or _id==123 (555 and 123 come from B user), in this case the query result should be:
A:{_id:432645624232345,class:[{_id:123,name:'foo'}]} // match _id=123
B:{_id:432645624232555,class:[{_id:555,name:'foo'},{_id:123,name:'foo'}]} //match _id=123 and _id=555
C:{_id:432645344232345,class:[{_id:555,name:'foo'},{_id:111,name:'www'}]} //match _id=555
D:{_id:432644444232345,class:[{_id:222,name:'sss'},{_id:555,name:'www'},{_id:123,name:'foo'}]} ///match _id=123 and _id=555
which is all the user.
so far i get this:
{"class._id" : { $in : ["5234ccb7687ea597eabee671", "5234ccb7687ea597eabee672"] } }
but when different user login the class._id query condition is different. So is there any operator to do this
{"class._id" : { $in : req.user.class } }
hope I made myself clear.
In order to achieve what you want, first you must isolate the class _ids in an array, and then use it in the query argument.
var classIds = [];
var i = 0;
while (i < req.user.class.length) {
classIds.push(req.user.class[i]._id);
i++;
}
After that you can use classIds array in the query:
{"class._id" : { $in : classIds } }
The following query condition would give you all the users that have at least one class with id equal to any of the elements in the given array:
{"class._id" : { $in : ["5234ccb7687ea597eabee671", "5234ccb7687ea597eabee672"] } }
In the array for the $in clause you may provide any id's you needed , comma separated.
In addition, if you needed such, the below query condition should check for existence of nested document within "class" property that has a property "_id" :
{ "class._id" : { $exists : true } }
Both conditions should work no matter if "class._id" is a single-valued property or an array (mongo supports that).
i have collection of objects inside an invitation, having hard time to filter particular object and trigger it's boolean field.
Document:
"Invitation" : [
{
"__v" : 0,
"userID" : ObjectId("54afaabd88694dc019d3b628"),//ObjectId of personA
"__t" : "USER",
"_id" : ObjectId("54b5022b583973580c706784"),
"Accepted" : false
},
{
"__v" : 0,
"userID" : ObjectId("54af6ce091324fd00f97a15f"),//ObjectId of personB
"__t" : "USER",
"_id" : ObjectId("54bde39cdd55dd9016271f14"),
"Accepted" : false
}
]
here i have only two objects inside Invitation array,it can be more than two.
Let's say personA and personB send me Invitation, so two different invitation objects are inserted into database having different fields, with objectId of both persons(userID in above document), now if i accept only invitation of personA, it should trigger accepted field of personA object only, here is what i tried so far, but not working as per expectation.
Controller:
User.find({_id: req.user._id},'Invitation',function(err,docs) {
if (err) {
console.log(err);
}
var results = [];
async.each(docs,function(doc,callback) {
async.each(doc.Invitation,function(invite,callback) {
User.findOneAndUpdate(
{'_id': doc._id, 'Invitation._id': invite._id},
{'$set': {'Invitation.$.Accepted': !invite.Accepted}},
function(err,doc) {
results.push(doc);
callback(err);
}
);
},callback);
},function(err) {
if (err)
console.log(err);
console.log('end'+results);
});
});
finally i am looking for a query which can be used to filter a single element or object, like if i accept invitation of personA then Accepted field of personA object should be set to true.
i would be really helpful if some logic is provided.
Thank you
Not a very clear question. But it seems all you really need to do here is just match the only sub-document you want to update in the first place:
User.find(
{
"_id": "req.user._id",
"Invitation._id": personA.id
},
{ "Invitation.$": 1 },
function(err,docs) {
// and continue
}
);
This is the form of the positional $ operator in a "projection" context. Where only the "singular" matched element is returned.
Once you have a "singular" result, then all the other code works as designed.
I should know after all because I wrote it for you. Not that you are paying any decent respect to that.
Update on Aggregate in Mongodb
Toggle boolean value of subdocuments
Or personA.userID or whatever makes it work.
Just use the unique identifier for the "user" where you expect that to match the query conditions.
You can do this:
db.user.update({"invitation.userID": 1}, {"$set" : {"invitation.$.Accepted" : true}});
Replacing the value 1 with the user ID you want to update.
The code is in the syntax of MongoShell, simply convert to driver syntax you are using
The operator used was the $. According to the documentation: The positional $ operator identifies an element in an array to update without explicitly specifying the position of the element in the array. To project, or return, an array element from a read operation, see the $ projection operator.
For more details see: http://docs.mongodb.org/manual/reference/operator/update/positional/
I am looking for a CouchDB equivalent to "SQL joins".
In my example there are CouchDB documents that are list elements:
{ "type" : "el", "id" : "1", "content" : "first" }
{ "type" : "el", "id" : "2", "content" : "second" }
{ "type" : "el", "id" : "3", "content" : "third" }
There is one document that defines the list:
{ "type" : "list", "elements" : ["2","1"] , "id" : "abc123" }
As you can see the third element was deleted, it is no longer part of the list. So it must not be part of the result. Now I want a view that returns the content elements including the right order.
The result could be:
{ "content" : ["second", "first"] }
In this case the order of the elements is already as it should be. Another possible result:
{ "content" : [{"content" : "first", "order" : 2},{"content" : "second", "order" : 1}] }
I started writing the map function:
map = function (doc) {
if (doc.type === 'el') {
emit(doc.id, {"content" : doc.content}); //emit the id and the content
exit;
}
if (doc.type === 'list') {
for ( var i=0, l=doc.elements.length; i<l; ++i ){
emit(doc.elements[i], { "order" : i }); //emit the id and the order
}
}
}
This is as far as I can get. Can you correct my mistakes and write a reduce function? Remember that the third document must not be part of the result.
Of course you can write a different map function also. But the structure of the documents (one definig element document and an entry document for each entry) cannot be changed.
EDIT: Do not miss JasonSmith's comment to his answer, where he describes how to do this shorter.
Thank you! This is a great example to show off CouchDB 0.11's new
features!
You must use the fetch-related-data feature to reference documents
in the view. Optionally, for more convenient JSON, use a _list function to
clean up the results. See Couchio's writeup on "JOIN"s for details.
Here is the plan:
Firstly, you have a uniqueness contstraint on your el documents. If two of
them have id=2, that's a problem. It is necessary to use
the _id field instead if id. CouchDB will guarantee uniqueness, but also,
the rest of this plan requires _id in order to fetch documents by ID.
{ "type" : "el", "_id" : "1", "content" : "first" }
{ "type" : "el", "_id" : "2", "content" : "second" }
{ "type" : "el", "_id" : "3", "content" : "third" }
If changing the documents to use _id is absolutely impossible, you can
create a simple view to emit(doc.id, doc) and then re-insert that into a
temporary database. This converts id to _id but adds some complexity.
The view emits {"_id": content_id} data keyed on
[list_id, sort_number], to "clump" the lists with their content.
function(doc) {
if(doc.type == 'list') {
for (var i in doc.elements) {
// Link to the el document's id.
var id = doc.elements[i];
emit([doc.id, i], {'_id': id});
}
}
}
Now there is a simple list of el documents, in the correct order. You can
use startkey and endkey if you want to see only a particular list.
curl localhost:5984/x/_design/myapp/_view/els
{"total_rows":2,"offset":0,"rows":[
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","0"],"value":{"_id":"2"}},
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","1"],"value":{"_id":"1"}}
]}
To get the el content, query with include_docs=true. Through the magic of
_id, the el documents will load.
curl localhost:5984/x/_design/myapp/_view/els?include_docs=true
{"total_rows":2,"offset":0,"rows":[
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","0"],"value":{"_id":"2"},"doc":{"_id":"2","_rev":"1-4530dc6946d78f1e97f56568de5a85d9","type":"el","content":"second"}},
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","1"],"value":{"_id":"1"},"doc":{"_id":"1","_rev":"1-852badd683f22ad4705ed9fcdea5b814","type":"el","content":"first"}}
]}
Notice, this is already all the information you need. If your client is
flexible, you can parse the information out of this JSON. The next optional
step simply reformats it to match what you need.
Use a _list function, which simply reformats the view output. People use them to output XML or HTML however we will make
the JSON more convenient.
function(head, req) {
var headers = {'Content-Type': 'application/json'};
var result;
if(req.query.include_docs != 'true') {
start({'code': 400, headers: headers});
result = {'error': 'I require include_docs=true'};
} else {
start({'headers': headers});
result = {'content': []};
while(row = getRow()) {
result.content.push(row.doc.content);
}
}
send(JSON.stringify(result));
}
The results match. Of course in production you will need startkey and endkey to specify the list you want.
curl -g 'localhost:5984/x/_design/myapp/_list/pretty/els?include_docs=true&startkey=["abc123",""]&endkey=["abc123",{}]'
{"content":["second","first"]}