I have data object that contains array. And I have providers table.
if array's Id should equals to provider's table id Id == id
if id is repeated take the repeated count as membersCounts else membersCounts = 0
Add the membersCounts with data object
data object
const data = {
Milk: [
{
Id: 1,
name: 'a'
},
{
Id: 2,
name: 'b'
},
{
Id: 3,
name: 'c'
},
{
Id: 4,
name: 'd'
},
{
Id: 52,
name: 'e'
}
],
Grocery: [
{
Id: 8,
name: '2a'
},
{
Id: 22,
name: '2b'
},
{
Id: 32,
name: '2c'
},
{
Id: 42,
name: '2d'
}
]
}
providers table
const providers = [
{
id: 1,
status: 'active'
},
{
id: 1,
status: 'active'
},
{
id: 1,
status: 'active'
},
{
id: 1,
status: 'active'
},
{
id: 4,
status: 'active'
},
{
id: 2,
status: 'active'
},
{
id: 3,
status: 'active'
},
{
id: 3,
status: 'active'
},
{
id: 52,
status: 'active'
},
{
id: 1,
status: 'active'
}
]
here javascript code
this code is working good but I want perform this with mongodb queries. So that performance is good .
Is possible to do with mongodb query. I need to covert the javascript code to mongodb query.
getMembersWithVendors(data, providers) {
for (var key in data) {
var arr = data[key]
arr.forEach((element) => {
element.memberCounts = 0
element.new = true
providers.map((el) => {
if (element._id == el.vendorId) {
(element.memberCounts = element.memberCounts + 1),
(element.new = false)
}
})
})
}
return data
}
output
{ Milk:
[ { Id: 1, name: 'a', memberCounts: 5 },
{ Id: 2, name: 'b', memberCounts: 1 },
{ Id: 3, name: 'c', memberCounts: 2 },
{ Id: 4, name: 'd', memberCounts: 1 },
{ Id: 52, name: 'e', memberCounts: 1 } ],
Grocery:
[ { Id: 8, name: '2a', memberCounts: 0 },
{ Id: 22, name: '2b', memberCounts: 0 },
{ Id: 32, name: '2c', memberCounts: 0 },
{ Id: 42, name: '2d', memberCounts: 0 } ] }
Thanks !!
this code is working good but I want perform this with mongodb queries. So that performance is good
That is not good idea to do all the operations in query, it may cause performance issues, because your input data is so big, but you can improve some things,
$group by query id and get count, this will return unique ids and its total counts
let providers = await db.providers.aggregate([
{
$group: {
_id: "$id",
count: { $sum: 1 }
}
}
]);
iterate loop of object's array
find from providers on the base of id
get count from filtered document
for (let key in data) {
data[key].forEach(e => {
let p = providers.find(p => p._id === e.Id);
e.memberCounts = p ? p.count : 0;
})
}
console.log(data);
Repl Playground
Related
I have a collection that have a field named "data" that can have any fields, and I have to get all existing fields in all collections in that "data" field or get the documents that have diferents fields in that "data" field.
for example, if I have:
[
{
_id: "45454",
name: "fulano",
city: "cali",
data: {
age: 12,
lastName: "panguano",
cars: 0
}
},
{
_id: "67899",
name: "juanito",
city: "cali",
data: {
age: 23,
lastName: "merlano",
cars: 2
}
},
{
_id: "67899",
name: "olito",
city: "nw",
data: {
lastName: "betito",
cars: 2
}
},
{
_id: "11223",
name: "cabrito",
city: "trujillo",
data: {
age: 28,
cars: 1,
moto: 3
}
},
]
what I would like to get:
["age", "lastName", "cars", "moto"]
or :
documents where the "data" fields vary, regardless of their values.
[
{
_id: "45454",
name: "fulano",
city: "cali",
data: {
age: 12,
lastName: "panguano",
cars: 0
}
},
{
_id: "67899",
name: "olito",
city: "nw",
data: {
lastName: "betito",
cars: 2
}
},
{
_id: "11223",
name: "cabrito",
city: "trujillo",
data: {
age: 28,
cars: 1,
moto: 3
}
}
]
THE COLLECTION HAVE SO MANY DOCUMENTS CAN BE A PROBLEM IF I USE
FINDALL AND THEN USE A LOOP LIKE FOR (FOR THE RESOURCES)
Regardless how you execute this (in memory or on the db) this is a very expensive query, with that said I agree doing this in memory is the wrong approach.
Here's how to do it using the aggregation pipeline and some standard operators like $map and $objectToArray:
db.collection.aggregate([
{
$project: {
keys: {
$map: {
input: {
"$objectToArray": "$data"
},
in: "$$this.k"
}
}
}
},
{
"$unwind": "$keys"
},
{
$group: {
_id: "$keys"
}
}
])
Mongo Playground
Here's a way using javascript once you have an array of all documents in the collection:
let arr = [
{
_id: "45454",
name: "fulano",
city: "cali",
data: {
age: 12,
lastName: "panguano",
cars: 0
}
},
{
_id: "67899",
name: "juanito",
city: "cali",
data: {
age: 23,
lastName: "merlano",
cars: 2
}
},
{
_id: "67899",
name: "olito",
city: "nw",
data: {
lastName: "betito",
cars: 2
}
},
{
_id: "11223",
name: "cabrito",
city: "trujillo",
data: {
age: 28,
cars: 1,
moto: 3
}
},
]
You can use the .map method to get an array of the data objects like so:
arr = arr.map(obj => obj.data)
This will return
[
{
"age": 12,
"lastName": "panguano",
"cars": 0
},
{
"age": 23,
"lastName": "merlano",
"cars": 2
},
{
"lastName": "betito",
"cars": 2
},
{
"age": 28,
"cars": 1,
"moto": 3
}
]
Then you can get an array of data object keys by looping through the array of data objects like so:
let dataKeys = [];
arr.forEach(obj => {
dataKeys = [...dataKeys, ...Object.keys(obj)]
})
This returns an array of non unique keys:
dataKeys = [
"age",
"lastName",
"cars",
"age",
"lastName",
"cars",
"lastName",
"cars",
"age",
"cars",
"moto"
]
Then filter out the unique keys using .filter and .findIndex methods:
let uniqueKeys = dataKeys.filter((elem, index) => dataKeys.findIndex(obj => obj === elem) === index)
And this will give you
[
"age",
"lastName",
"cars",
"moto"
]
I am trying to query a list of documents where a userid DOES NOT exist inside an array of objects.
The database (documents) looks like this:
[
{
title: 'object 1',
description: 'description 1',
members: [
{ profile: { id: '123', ...}, data: {} },
{ profile: { id: 'abc', ...}, data: {} },
{ profile: { id: 'def', ...}, data: {} },
]
},
{
title: 'object 2',
description: 'description 3',
members: [
{ profile: { id: 'aaa', ...}, data: {} },
{ profile: { id: 'bbb', ...}, data: {} },
{ profile: { id: 'ccc', ...}, data: {} },
]
},
]
Given that my userid is 'aaa' I am trying to query all documents where I am NOT a member.
I can successfully query all documents where my userid exists using this code:
await this._repository.findManyByQuery(
{
members: {
$elemMatch: {
"profile.id": "aaa",
},
},
},
)
However I am looking to query all objects where my ID DOES NOT exist. I have tried using $ne however it still returns the documents where the user id exists
members: {
$elemMatch: {
"profile.id": { $ne: "aaa" },
},
},
I guess I am looking for the opposite of $elemMatch but for querying inside an arry
You can use $not to negate the $elemMatch like this:
await this._repository.findManyByQuery({
members: {
"$not": {
$elemMatch: {
"profile.id": "aaa"
}
}
}
})
Example here
I have a collection of documents that look like this:
[
{ group_id: 1, value: 'foo' },
{ group_id: 1, value: 'bar' },
{ group_id: 1, value: 'bar' },
{ group_id: 1, value: 'bar' },
{ group_id: 2, value: 'bar' },
{ group_id: 2, value: 'foo' },
{ group_id: 2, value: 'foo' },
{ group_id: 2, value: 'foo' }
]
For each group_id I want to return the value that occurs the most. So my output should look something like this...
[
{ group_id: 1, maxValue: 'bar', maxValueCount: 3 },
{ group_id: 2, maxValue: 'foo', maxValueCount: 3 }
]
How would you do this using the Mongoose aggregate function?
Update:
This is as far as I've gotten, I just need to return the value of the maximum count now...
const records = await Record.aggregate([
{
$group: {
_id: {
id: '$group_id',
value: '$value'
},
count: { $sum: 1 }
}
}
])
You may achieve your goal by
grouping by group_id/value (and counting the occurrences of value)
then grouping by group_id and pushing all the found values within with their count
finally keeping from the array the value having the max count
data=[
{ group_id: 1, value: 'foo' },
{ group_id: 1, value: 'bar' },
{ group_id: 1, value: 'bar' },
{ group_id: 1, value: 'bar' },
{ group_id: 1, value: 'bar' },//one more added
{ group_id: 2, value: 'bar' },
{ group_id: 2, value: 'foo' },
{ group_id: 2, value: 'foo' },
{ group_id: 2, value: 'foo' }
]
db.products.remove({})
db.products.insert(data)
const stages = [
{
$group: {
_id: {
group_id: '$group_id',
value: '$value'
},
n: { $sum: 1 }
}
},
{
$group: {
_id: '$_id.group_id',
values: {
$push: {
value: '$_id.value',
n: '$n'
}
}
}
},
{
$project: {
group_id:1,
best: {
$reduce: {
input: '$values',
initialValue: { n: 0, value: ''},
in: {
$cond: [
{
$lt: ['$$value.n', '$$this.n']
},
'$$this',
'$$value'
]
}
}
}
},
},
{
$project: {
group_id: 1,
value: '$best.value',
maxValue: '$best.n'
}
}
]
printjson(db.products.aggregate(stages).toArray())
playground
I have this schema in Mongoose:
var CoinAmountSchema = new Schema(
{
user: [{ type: Schema.ObjectId, ref: 'User' }],
coinAmounts: [{
_id: false,
coinID: { type: Number, ref: 'Coin' },
amount: Number
}]
})
I am writing this query, that checks the userID and coinID and should update the amount of only that coinID's amount.
exports.coin_amount_update = [
(req, res, next) => {
CoinAmount.update({
"user": req.params.userId,
"coinAmounts.coinID": req.params.coinId
},
{
'$set': {
'coinAmounts.$.amount': req.body.amount
}
},
function (err, model) {
if (err) {
console.log(err)
return res.send(err)
}
return res.json(model)
})
}]
But like this, it only updates the first coin's in the array amount. BUT, if I delete the line "user": req.params.userId, it would find and update the right coin. I need to check for a user as well though, so how can I make it work?
Is there something wrong with the query or the way the data is structured?
EDIT: I send a request in React-native:
fetch(`${apiBaseURL}/users/${getState().user._id}/coins/${id}/update`, {
method: 'PUT',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
amount: getState().coins[id].amount
}),
})
If the request is /users/:userID/coins/0/update (with amount: 1)
then the result will be
{ _id: 5a579d0d44e7390ba3029327,
__v: 0,
coinAmounts:
[ { coinID: 0, amount: 1 },
{ coinID: 1, amount: 0 },
{ coinID: 2, amount: 0 },
{ coinID: 3, amount: 0 },
{ coinID: 4, amount: 0 } ],
user: [ 5a579d0d44e7390ba3029326 ] }
The same result if the request is /users/:userID/coins/1/update with the same amount.
But if as mentioned before, I remove the check for userID, the request /users/:userID/coins/1/update would produce this:
{ _id: 5a579d0d44e7390ba3029327,
__v: 0,
coinAmounts:
[ { coinID: 0, amount: 0 },
{ coinID: 1, amount: 1 },
{ coinID: 2, amount: 0 },
{ coinID: 3, amount: 0 },
{ coinID: 4, amount: 0 } ],
user: [ 5a579d0d44e7390ba3029326 ] }
Hope I was clear.
It looks like a bug when using two arrays in find with $ positional update, it gets matching index of user for $ positional update
tried below workarounds, both updates correct coinID
workaround-1, using arrayFilters
db.coins.update(
{ "user" : "5a579d0d44e7390ba3029326" }, //user
{
$set: { "coinAmounts.$[elem].amount" : 1 } //update
},
{
multi: false,
arrayFilters: [
{ "elem.coinID": 2 } //coinID
]
}
)
workaround-2, using elemMatch for user array
db.coins.update(
{
"coinAmounts.coinID" : 1, //coinID
"user" : { $elemMatch : { $eq : "5a579d0d44e7390ba3029326" } } //user
},
{ $set : { "coinAmounts.$.amount" : 1 } } //update
)
Let's say I have an array of Movie genres like so:
[
{ id: 28, name: 'Action' },
{ id: 12, name: 'Adventure' },
{ id: 16, name: 'Animation' },
{ id: 35, name: 'Comedy' },
{ id: 80, name: 'Crime' },
{ id: 99, name: 'Documentary' },
{ id: 18, name: 'Drama' },
{ id: 10751, name: 'Family' },
{ id: 14, name: 'Fantasy' },
{ id: 10769, name: 'Foreign' },
{ id: 36, name: 'History' },
{ id: 27, name: 'Horror' },
{ id: 10402, name: 'Music' },
{ id: 9648, name: 'Mystery' },
{ id: 10749, name: 'Romance' },
{ id: 878, name: 'Science Fiction' },
{ id: 10770, name: 'TV Movie' },
{ id: 53, name: 'Thriller' },
{ id: 10752, name: 'War' },
{ id: 37, name: 'Western' }
]
and I have a connection to a MongoDB (v3.2) instance: db, and I'm using the standard mongodb Node.js driver (const mongodb = require('mongodb').MongoClient).
What I want to be able to do is one bulk upsert operation onto a collection, say genres, where the _id field maps to the id field of our genre objects.
Now, I know I could loop through each item in the array, and do a simple upsert:
for (let i = 0; i < genres.length; i++) {
await db.collection('genres').update(
{ _id: genres[i].id },
genres[i],
{ upsert: true }
);
}
But this feels wasteful and wrong.
Is there an easier way to do what should be a relatively simple task?
Thanks
Use the bulkWrite API to carry out the updates:
var bulkUpdateOps = genres.map(function(doc) {
return {
"updateOne": {
"filter": { "_id": doc.id },
"update": { "$set": { "name": doc.name } },
"upsert": true
}
};
});
db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
// do something with result
})
If you're dealing with larger arrays i.e. > 1000 then consider sending the writes to the server in batches of 500 which gives you a better performance as you are not sending every request to the server, just once in every 500 requests:
var bulkUpdateOps = [],
counter = 0;
genres.forEach(function(doc) {
bulkUpdateOps.push({
"updateOne": {
"filter": { "_id": doc.id },
"update": { "$set": { "name": doc.name } },
"upsert": true
}
});
counter++;
if (counter % 500 == 0) {
db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
// do something with result
});
bulkUpdateOps = [];
}
})
if (counter % 500 != 0) {
db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
// do something with the result
});
}
I would try:
db.collection('genres').update(genres, {upsert: true, multi: true});
Note: untested code...
UPDATE: to remap id field to _id:
var _genres = genres.map(function(genre) {
return { _id: genre.id, name: genre.name };
});
db.collection('genres').update(_genres, {upsert: true, multi: true});