I have versioned documents stored in a MongoDb collection as follows:
{ _id: 'doc1-1', id: doc1, version: 1 ...},
{ _id: 'doc1-2', id: doc1, version: 2 ...},
{ _id: 'doc2-1', id: doc2, version: 1 ...}
I want to return all of the latest documents.
Example:
{ _id: 'doc1-2', id: doc1, version: 2 ...},
{ _id: 'doc2-1', id: doc2, version: 1 ...}
I know MongoDb includes operators such as Max and First but these do not seem to apply to subsets of documents within a query so I cannot get each document with a Max version of a unique id.
My goal is to return the full document and not project each field if possible. I'm currently using the MongoDb NPM package so that format of solution is preferred but any MongoDb solution is greatly appreciated.
EDIT: I am using AWS DocumentDb which uses MongoDb 4.0 so some newer Mongo features are not supported.
After some research the only solution I could identify was to make two separate calls to MongoDB.
First I used an aggregate call and grouped on id and max version.
const results = await collection.aggregate([
{
$sort: { version : -1 }
},
{
$group: {
_id: {
id: "$id"
},
version: { $max: "$version"}
}
}
]);
This returned the following response with the id and latest version.
{
_id: { id: "doc1" },
version: 2
},
{
_id: { id: "doc2" },
version: 1
}
I then mapped this response to get an array of the id appended with version to use as my query for getting each document.
const _ids = results.map(({ _id: { id }, version }) => (`${id}-${version}`));
const documents = await collection.find({
_id: { $in: _ids }
});
This returned all of the latest versions with the full document and was much more performant than filtering on server. However, I was not able to determine a method to get the complete document in one call so that solution would still be appreciated if possible.
An aggregation like this should work:
collection.aggregate([
{$sort: {id: 1, version: -1}},
{$group: {_id: '$id', doc: {$first: '$$ROOT'}},
{$replaceRoot: {newRoot: '$doc'}}
])
First bring your documents into a defined order with the latest version first, then group by id, keeping the first document via the $$ROOT variable, then unpack that again to retain only the wanted document.
Related
I have two tables, one for messages to be logged and one that is for each member of the group. When a message is upvoted, I want to add an entry for the message and push each reaction on that message to an array.
Every time a reaction is added I want to update the member table to reflect the sum of all of the reaction.value fields.
This is the code I have written to do so. When this runs from a sandbox I made in VisualStudio using a MongoDB add on it runs fine, however when ran using my app, only the message document is added and without any error it appears to skip the aggregation to the member document.
Here is the code I am using to insert the documents to the database and to aggregate once the document is inserted:
await mongodb.collection("Messages").updateOne({ _id: reaction.message.id.toString()},
{
$set:{
authorid: reaction.message.author.id,
author: reaction.message.author.username
},
$push: {
reactions: {
reauth: reAuth,
reaction: reaction.emoji.name,
removed: false,
value: actualKarmaDB,
}
}
}, {safe: true, "upsert": true})
await mongodb.collection("Messages").aggregate([
{
$match: {
_id: reaction.message.id
}
},
{
$project: {
_id: "$authorid",
username: "$author",
messageKarma: {$sum: "$reactions.value"},
}
},
{ $merge: {
into: "Members",
on: "_id",
whenMatched: "replace",
whenNotMatched: "insert"
}
}])
Also here is a look at what the insertion into “Messages” looks like:
In this case the answer was due to mongoose not supporting merge for aggregation. Had to use $out.
Additionally this is a known issue with Mongoose Node.js, see here:
Mongodb node.js $out with aggregation only working if calling toArray()
I have mongoose schema that looks something like this:
{
_id: someId,
name: 'mike',
keys: {
apiKey: 'fsddsfdsfdsffds',
secretKey: 'sddfsfdsfdsfdsds'
}
}
I don't want to send back to the front the keys of course, but I want some indication, like:
{
_id: someId,
name: 'mike',
hasKeys: true
}
There is built in way to create 'field' on the way based on other fields, or do I need every time fetch the whole document, check if keys is not empty and set object property based on that?
For Mongo version 4.2+ What you're looking for is called pipelined updates, it let's you use a (restricted) aggregate pipeline as your update allowing the usage of existing field values.
Here is a toy example with your data:
db.collection.updateOne(
{ _id: someId },
[
{
"$set": {
"hasKeys": {
$cond: [
{
$ifNull: [
"$keys",
false
]
},
true,
false
]
}
}
},
])
Mongo Playground
For older Mongo versions you have to do it in code.
If you don't want to update the actual document but just populate this field when you fetch it you can use the same aggregation to fetch the document
you can use $project in mongoose aggregation like this.
$project: { hasKeys: { $cond: [{ $eq: ['$keys', null] }, false, true]}}
When trying to update the date inside the MongoDB document, nothing is updated.
To update, I send a request to NodeJS like this:
collection.updateOne(
{ _id: ObjectId(...) },
{ $set: { 'field1.field2.0.date': new Date('2020-02-01T00:00:00Z') } },
callback
)
The task uses the npm package mongodb#2.3.26 and MongoDB 3.6.20.
I tried to send the '2020-02-01T00:00:00Z' as a value - it is not updated. I tried to update the database version - it didn't help. I tried to update the driver version - it didn't help.
It is interesting that such a request goes through Robo3T and updates the value correctly.
Could you please tell me how to update the date inside the document for MongoDB?
UPD: Structure of document
const document = {
field1: {
exp: 1,
field2: [
{
name: "test",
date: 2019-10-01 00:00:00.000Z
}
]
},
settings: {
sharedTill: 2022-10-01 00:00:00.000Z
},
updatedBy: 'Vadim Prokopchuk'
}
UPD2: Added Response from MongoDB
MongoDB returned data and does not return an error.
the syntax you're using isn't correct.
Refer the query below, it would work
collection.updateOne(
{ _id: ObjectId(...) , 'field1.field2.name':"test"},
{ $set: { 'field1.field2.$.date': new Date('2020-02-01T00:00:00Z') } },
callback
)
Here, we are finding the element of the array which matches our condition and then updating the date of that particular element of the array.
Let's say I have the following document stored in a mongoDB collection 'people':
{
_id: 489324,
name: "Ryan Jones"
skills: [ "fishing", "programming" ]
}
I am trying to retrieve Ryan Jones's first skill in the array (skills[0]).
This seems like a dead simple operation but I can't seem to do it using the Node JS driver. I can retrieve just the skills array easily:
db.collection('people').findOne({ name:"Ryan Jones"},{ projection: { skills:1 }})
...but I don't want to transfer the whole array over the wire. I just want to get "fishing".
I have tried using slice and arrayElemAt within projection but I get a MongoError. How can I achieve this using the NodeJS driver? Does it require a more complex aggregation operation instead of findOne?
You can achieve that with aggregation , with $arrayElemAt something like this
db.collection('people').aggregate([
{
$match: {
name: "Ryan Jones"
}
},
{
$project: {
name: 1,
skills: {
$arrayElemAt: [
"$skills",
0
]
},
}
}
])
See demo here
Try this one:
db.collection('people').findOne({
name: "Ryan Jones"
},
{
skills: {
$slice: 1
}
})
MongoTemplate with .find
Playground with data sample: https://mongoplayground.net/p/OJKVFHLamig
Yet, when I run in mongoose within Node, the exact same collection and aggregation instead returns the total number of documents count and everything else is null:
[ { _id: null, myCount: 130111, site: null } ]
I've looked at all other variables, every comma in my production code and there's nothing else that explains this behaviour.
Is Mongoose unfit to use for the mongo aggregation framework or am I missing something about the syntax?
Schema:
import mongoose from 'mongoose';
import { SiteModel } from './site.schema';
const JobModel = new mongoose.Schema({
_id: String,
(other properties that are strings)
title: String,
site: { SiteModel },
});
export default mongoose.model('jobs', JobModel);
// SITE MODEL:
export const SiteModel = new mongoose.Schema({
_id: String;
title: String;
city: String;
UNID: String;
})
The models are incomplete as I'm only using it for reading purposes, the database is used by another app live and I'm merely building some reports/searching some data on it.
I am however pulling all the data that I need and It worked without a hiccup up until this.
EDIT 3: LOG ENTRY:
{ aggregate: "sites", pipeline:
[ { $group: { _id: "$site.UNID", myCount: { $sum: 1 },
score: { $first: "$score" } } },
{ $limit: 20 },
{ $project: { site: "$_id", myCount: 1 } } ]
I was using the wrong mongoose model to run the aggregation on, but the collections were similar enough that it was still returning results. I realized this by looking into the db logs and seeing exactly which collection my aggregation was running on.