I am using Parse Dashboard for User Management of my iOS Application.Also, I am using external APIs which are using MongoDB database.
The issue currently I am facing is the User created from Parse Dashboard is having small id instead of MongoDB's ObjectID, and other resources which are not over parse are generated by normal ObjectID.
eg. User Object:
{
_id:"qVnyrGynJE",
user_name:"Aditya Raval"
}
Document Object:
{
_id:"507f191e810c19729de860ea",
doc_name:"Marksheet",
user:"qVnyrGynJE"
}
Task Object:
{
_id:"507f191e810c19729de860ea",
task_name:"Marksheet",
user:"qVnyrGynJE"
}
I am also using Keystone.js as a Backend Admin Dashboard.So basically due to this mix kind of IDs relationships inside KeyStone.js is broken and Keystone.js gets crashed.
So I want to migrate all my existing small IDs to normal MongoDB ObjectIDs without breaking into relationships or any other walkthrough by fixing Keystone.js
You can run something like this:
var users = db.Users.find({});
for(var i = 0; i < users.length(); i++)
{
var oldId = users[i]._id;
delete users[i]._id;
db.Users.insert(users[i], function(err, newUser) {
db.Documents.updateMany({"user": oldId},{ $set: { "user": newUser._id }});
//Do this for all collections that need to be update.
});
);
}
db.Users.deleteMany({_id: { $type: "string" }});
Related
Existing Cosmos DB documents need to be altered/updated with a new property & also existing documents of other collections need to be updated with the same new property along with its value.
Is there any recommended way or tool available to update existing documents on Cosmos DB, or is writing the custom c# application/PowerShell script using Cosmos DB SDK is the only option?
Example:
Existing user document
{
id:user1#mail.com,
name: "abc",
country: "xyz"
}
Updated user document
{
id:user1#mail.com,
name: "abc",
country: "xyz",
guid:"4334fdfsfewr" //new field
}
Existing order document of the user
{
id:user1#mail.com,
user: "user1#mail.com",
date: "09/28/2020",
amt: "$45"
}
Updated order document of the user
{
id:user1#mail.com,
user: "user1#mail.com",
userid: "4334fdfsfewr", // new field but with same value as in user model
date: "09/28/2020",
amt: "$45"
}
I'd probably go with:
Update user documents through a script
Have Azure Function with Cosmosdb trigger that would listen to changes on users documents and update orders appropriately
[UPDATE]
whatever type of script you feel best with: PS, C#, Azure Functions...
now, what do you mean they need to be altered with the new property "on the same time"? i'm not sure that's possible in any way. if you want such an effect then i guess your best bet is:
create new collection/container for users
have an Azure Function that listens to a change feed for your existing users container (so, with StartFromBeginning option)
update your documents to have new field and store them in a newly created container
once done, switch your application to use new container
its your choice how would you change other collections (orders): using changeFeed & Azure Functions from old or new users container.
PS.
Yes, whatever flow i'd go with, it would still be Azure Functions with Cosmos DB trigger.
I have added some solution for .Net Core API 3.0 or higher version.
// You can put any filter for result
var result = _containers.GetItemLinqQueryable<MessageNoteModel>().Where(d => d.id == Id
&& d.work_id.ToLower() == workId.ToLower()).ToFeedIterator();
if (result.HasMoreResults)
{
var existingDocuments = result.ReadNextAsync().Result?.ToList();
existingDocuments.ForEach(document =>
{
//Creating the partition key of the document
var partitionKey = new PartitionKey(document?.work_id);
document.IsConversation = true;
//Inserting/Updating the message in to cosmos db collection: Name
_containers.Twistle.ReplaceItemAsync(document, document.document_id, partitionKey);
});
}
We had the same issue of updating the Cosmos DB schema for existing documents. We were able to achieve this through a custom JsonSerializer.
We created CosmosJsonDotNetSerializer inspired from Cosmos DB SDK. CosmosJsonDotNetSerializer exposes the FromStream method that allows us to deal with raw JSON. You can update the FromStream method to update document schema to your latest version. Here is the pseudo-code:
public override T FromStream<T>(Stream stream)
{
using (stream)
{
if (typeof(Stream).IsAssignableFrom(typeof(T)))
{
return (T)(object)stream;
}
using (var sr = new StreamReader(stream))
{
using (var jsonTextReader = new JsonTextReader(sr))
{
var jsonSerializer = GetSerializer();
return UpdateSchemaVersion<T>(jsonSerializer.Deserialize<JObject>(jsonTextReader));
}
}
}
}
private T UpdateSchemaVersonToCurrent<T>(JObject jObject)
{
// Add logic to update JOjbect to the latest version. For e.g.
jObject["guid"] = Guid.NewGuid().ToString();
return jObject.ToObject<T>();
}
You can set Serializer to CosmosJsonDotNetSerializer in CosmosClientOptions while creating CosmosClient.
var cosmosClient = new CosmosClient("<cosmosDBConnectionString>",
new CosmosClientOptions
{
Serializer = new CosmosJsonDotNetSerializer()
};
This way, you always deal with the latest Cosmos document throughout the code, and when you save the entity back to Cosmos, it is persisted with the latest schema version.
You can take this further by running schema migration as a separate process, for example, inside an Azure function, where you load old documents, convert them to the latest version and then save it back to Cosmos.
I also wrote a post on Cosmos document schema update that explains this in detail.
Looking to update lookupSchema images nested fields
var lookupSchema = new Schema({
images:{
"img1":String,
"img2":String,
"img3":String,
"img4":String
}
}
As I don't know which fields are available in request body during findOneAndUpdate operation, I am unable to assign like this
$set: {
'images.img1': req.body.img1,
'images.img2': req.body.img2,
'images.img3': req.body.img3
}
If I code like above and specify all fields, already available fields become empty after update due to unavailable fields in request.
Need something like this to update available fields
$set: { "images.$[element]" : req.body }
I have searched a lot but unable to find a solution.
You can simply create a function like below to be able to achieve your result.
function update() {
let temp = {};
for(let field in req.body) {
temp["images."+field] = req.body[field];
}
db.collection.findOneAndUpdate({._id:...},{$set:temp},{upsert:true},function(err,result)
{...});
}
I tried to create a user & pet model in Many to Many relationship using mongoDB and sailsJS, db creation and loading the data's are just fine i checked in mongoDB for dbs and collection it exists. when i try to get the user list and pet list its showing me the contents but when i try to GET a single pet by its iD under user by his iD i'm not getting result
It shows "Response does not contain any data."
I tried to get PET like this
"http://localhost:1337/user/54ffd28e9d9ee93c166c7500/pets/54ffd28e9d9ee93c166c7503"
I also tried to display the pet by its name instead of iD its also not working
my sails version is 0.11.0..
my OS is Win7 64 bit
here's a maybe useful thing to know, when requesting by id while using mongoDB.
to get some document inside of a collection, passing the id through an URL, it is useful to use the const { ObjectId } = require('mongodb').
you can for instance do as following:
// VS code might or might not write this for you
// but if you VIM, you have to write yourself
const { ObjectId } = require('mongodb')
fastify.get('/something/:id', async function (req, reply) {
const db = await this.mongo.db;
await db.collection('YourCollection', callbackFunc);
function callbackFunc(err, col) {
if (err) reply.send(err)
col.findOne({ "_id": new ObjectId(req.params.id) }, (err, result) =>
{
console.log(result)
})
}
})
Let me know if it helps, best wishes
So I have a model that is for a recipe where it has a relation of 'ingredients' and that is just an array of ObjectIds. When I run the following query on mongo shell it works fine and returns all my data.
Example model :
{
"name": "...",
"_id": ObjectId("530ca903746515c0161e6b9f"),
"ingredients": [
ObjectId("53069363ff7447a81a3a7a1d"),
ObjectId("53069363ff7447a81a3a7a17")
]
}
Query:
db.drink.find({"ingredients":{"$in":[ObjectId("53069364ff7447a81a3a7a87"), ObjectId("530fb948c1a3ff480d58e43c")]}});
Using sails.js though their waterline orm, they don't really have a way to query this though or at least through any possible google search that I can find. So trying to use the native driver I have something like the following -
var ings = new Array();
for (var i in req.body) {
ings.push(new ObjectID(req.body[i].toString()));
}
Drink.native(function(err, collection){
if(err){
return res.send(err, 500);
} else {
collection.find({"ingredients":{"$in":ings}}).toArray(function(err, data){
console.log(data);
});
}
});
The thing is the data array returned in the callback is always empty. If I check the 'ings' array it is an array of objectids so I am not sure why it won't return any data. If I remove the json object in the 'find' function it does return everything. Anyone have any idea how to make this query return data when using sails.js?
The code above actually is working it was an internal data issue on the mongodb where IDs were not matching between relations. When the relations were rebuilt all is working as it should be.
As I understand it, when you enter something into GridFS it gets entered into 2 different collections under the hood. One for the raw chunks of data and one for the meta data files.
Also from what I understand from MongoDB's documentation is that you can only retrieve a document from GridFS with an id or name.
var gs = new mongodb.GridStore(db, "test.png", "w", {
"content_type": "image/png",
"metadata":{
"author": "Daniel"
},
"chunk_size": 1024*4
});
So what if I want to get a subset of documents from GridFS? For example what if I want all GridStores with:
metadata: {author: "Daniel"}
Why can't I use standard mongo queries { field: somevalue } and retrieve documents that way?
Does anybody know how this can be done? I'm using the javascript API on node.js.
You can query the db.files collection just like any other collection:
db.collection('fs.files')
.find({ 'metadata.author' : 'Daniel' })
.toArray(function(err, files) {
if (err) throw err;
files.forEach(function(file) {
var gs = new mongodb.GridStore(db, file._id, 'r');
...
});
});
Although instead of plain forEach you may want to use async.each or any of the other async.* methods.