I have a database with 800+ different bars, clubs and restaurants across Australia.
I want to build a list of links for my website counting the number of different venues across different suburbs and primary categories.
Like this:
Restaurants, Bowen Hills (15)
Restaurants, Dawes Point (6)
Clubs, Sydney (138)
I could do it the hard way by first getting all venues. Then run a Venue.distinct('details.location.suburb') to get all the unique suburbs.
From here I could run subsequent queries to get the count for the number of venues in that particular suburb and category.
It will be a lot of calls though. There's got to be better way?
Can the Mongo aggregation framework help here?
It seems to be impossible to do this in a single query.
Here's the Venue model:
{
"name" : "Johnny's Bar & Grill",
"meta" : {
"category" : {
"all" : [
"restaurant",
"bar"
],
"primary" : "restaurant"
}
},
"details" : {
"location" : {
"streetNumber" : "180",
"streetName" : "abbotsford road",
"suburb" : "bowen hills",
"city" : "brisbane",
"postcode" : "4006",
"state" : "qld",
"country" : "australia"
},
"contact" : {
"phone" : [
"(07) 5555 5555"
]
}
}
}
}
Here's the prettified solution from BatScream that I ended up using:
Venue.aggregate([
{
$group: {
_id: {
primary: '$meta.category.primary',
suburb: '$details.location.suburb',
country: '$details.location.country',
state: '$details.location.state',
city: '$details.location.city'
},
count: {
$sum: 1
},
type: {
$first: '$meta.category.primary'
}
}
},
{
$sort: {
count: -1
}
},
{
$limit: 50
},
// Reshapes each document in the stream, such as by adding new fields or removing existing fields. For each input document, outputs one document.
{
$project: {
_id: 0,
type : '$type',
location : '$_id.suburb',
count: 1
}
}
],
function(err, res){
next(err, res);
});
}
You can get a very useful and easily transformable output using the following aggregation operation.
Group the records based on their country, category, state, city and
suburb.
Get the count of the records in each group.
Obtain the type of the group from the first record of the group.
Project the necessary fields.
Code:
db.collection.aggregate([
{$group:{"_id":{"primary":"$meta.category.primary",
"suburb":"$details.location.suburb",
"country":"$details.location.country",
"state":"$details.location.state",
"city":"$details.location.city"},
"count":{$sum:1},
"type":{$first:"$meta.category.primary"}}},
{$sort:{"count":-1}},
{$project:{"_id":0,
"type":"$type",
"location":"$_id.suburb",
"count":1}}
])
sample o/p:
{ "count" : 1, "type" : "restaurant", "location" : "bowen hills" }
Related
We were working on a project with a 300 documents with currentValue field in a main collection, in order to track the history of each document of first collection. we created another collection named history with approximately 6.5 millions of documents.
For each input of system we have to add around 30 history item and update currentValue field of main collection, so, We tried computational field design pattern for currentValue, which lead us to have writeConfilict in concurrent situations (at concurrency of around 1000 requests).
Then we tried to compute currentValue field with sum (amount field) and groupBy(mainId field) on history collection which takes too long (> 3s).
Main collection docs:
{
"_id" : ObjectId(...),
"stock" : [
{
"currentAmount" : -313430.0,
"lastPrice" : -10.0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
{
"currentAmount" : 30,
"lastPrice" : 0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
.
.
.
],
"name" : "name",
}
History collection docs:
{
"_id" : ObjectId("..."),
"mainId" : ObjectId("..."),
"amount" : 5,
}
If you have any other idea to handle this situation(application or db level), I would be thankful.
UPDATE 1
The update query if I use computed pattern would be:
mainCollection.findOneAndUpdate(
{
$and: [
{ _id: id },
{ "stock.storage": fromId },
{ "stock.deletedAt": null }
],
},
{
$inc: {
"stock.$.currentAmount": -1 * amount,
}
},
{
session
}
)
And Aggregation pipeline if I want to calculate currentAmount everytime:
mainCollection.aggregate([
{
$match: {
branch: new ObjectId("...")
}
},
{
$group: {
_id: "$ingredient",
currentAmount: {
$sum: "$amount"
}
}
}])
in order to have computed field, mongo design patterns, suggested computed field,
The Computed Pattern is utilized when we have data that needs to be computed repeatedly in our application. link
like below:
// your main collection will look like this
{
"_id" : ObjectId(...),
"stock" : [
{
"currentAmount" : -313430.0,
"lastPrice" : -10.0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
{
"currentAmount" : 30,
"lastPrice" : 0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
"totalAmount": 20000 // for example
}
but for having concurrent there is a better way to solve this problem with cumulative summation, in this algorithm, we sum last documents inputs, with current input:
{
"_id" : ObjectId("..."),
"mainId" : ObjectId("..."),
"amount" : 5,
"cumulative": 15 // sum of last documents input
}
I have tried to get sorted in backend & tested via postman and I am getting sorted order.
const locationInfo = await locationDetails.find(query).sort({sectionName:1});
res.json(locationInfo);
[
{ //some other keys &values
"sectionName": "Closet",
},
{
"sectionName": "Dining",
},
{
"sectionName": "Kitchen",
},
{
"sectionName": "Other",
},
{
"sectionName": "Refrigerator",
}
]
After REST call storing result to,
this.result=data;
but when I try to display the same resultant data on UI, Its not getting displayed in sorted order as well as checked in console also resultant data order got changed.
Console Data
[{
sectionName: "Refrigerator",
},
{
sectionName: "Kitchen",
},
{
sectionName: "Dining",
},
{
sectionName: "Closet",
},
{
sectionName: "Other",
}]
Note: Tried to sort from .ts file also but it is not working.
this.result.sort(function(a,b){a.sectionName-b.sectionName});
If any help would be appreciated. Thanks!
SectioName is not a valid criterion for MongoDB to sort the return result. In this case, MongoDB does not know how to sort it.
Here is an example directly from the MongoDB documentation about cursor.sort():
db.restaurants.insertMany( [
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan"},
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens"},
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn"},
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan"},
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn"},
] );
# The following command uses the sort() method to sort on the borough field:
db.restaurants.find().sort( { "borough": 1 } )
Documents are returned in alphabetical order by borough, but the order of those documents with duplicate values for borough might not be the same across multiple executions.
.sort works best with numerical values. If you are in control of the backend and are able to change how data is stored in the database. I suggest you create a field for the creation date or just an index to indicate the order of the items.
Let's say your document looks something like this:
# Doc 1
{
sectionName: "Refrigerator",
order:1
}
# Doc 2
{
sectionName: "Refrigerator",
order:2
}
Then you can do
const locationInfo = await locationDetails.find(query).sort({order:1});
which will return you the documents sorted using the order field, and the order will be consistent.
I am trying to compare and find different documents from two collections
below are the samples, Mongodb version:4.0, ORM:mongoose
**col1: Has one new document**
{ "id" : 200001, "mobileNo" : #######001 }
{ "id" : 200002, "mobileNo" : #######002 } //mobileNo may not be unique.
{ "id" : 200003, "mobileNo" : #######002 }
{ "id" : 200004, "mobileNo" : #######004 }
**col2:**
{ "id" : 200001, "mobileNo" : #######001 }
{ "id" : 200002, "mobileNo" : #######002 }
{ "id" : 200003, "mobileNo" : #######003 }
Now I want to insert the document { "id" : 200004, "mobileNo" : #######004 } from col1 to col2
i.e; the documents which doesn't match.
This is what I've tried so far :
const col1= await Col1.find({}, { mobileNo: 1,id: 1, _id: 0 })
col1.forEach(async function (col1docs) {
let col2doc = await Col2.find({ mobileNo: { $ne: col1docs.mobileNo},
id:{$ne:col1docs.id} }, { mobileNo: 1, _id: 0, id: 1 })
if (!(col2doc)) {
Col2.insertMany(col1docs);
}
});
I have also tried with $eq instead of $ne but neither i am getting the unmatched documents nor they are getting inserted. Any suggestions??? Combination of id+phoneNo is unique
I would say instead of doing two .find() calls plus iteration & then third call to write data, try this query :
db.col1.aggregate([
{
$lookup: {
from: "col2",
let: { id: "$id", mobileNo: "$mobileNo" },
pipeline: [
{
$match: { $expr: { $and: [ { $eq: [ "$id", "$$id" ] }, { $gte: [ "$mobileNo", "$$mobileNo" ] } ] } }
},
{ $project: { _id: 1 } } // limiting to `_id` as we don't need entire doc of `col2` - just need to see whether a ref exists or not
],
as: "data"
}
},
{ $match: { data: [] } // Filtering leaves docs in `col1` which has no match in `col2`
},
{ $project: { data: 0, _id: 0 } }
])
Test : mongoplayground
Details : From the above query you're taking advantage of specifying conditions in $lookup to get docs from col1 which have reference in col2. Let's say $lookup will run on each document of col1 - So with the unique combination of id & mobileNo from current document in col1 has a matching in col2 then col2 doc's _id will be pushed in data array, at the end what we get out of col1 is data: [] to say no matching docs were found for these col1 doc's. Now you can just write all the returned docs to col2 using .insertMany(). Actually you can do this entire thing using $merge on MongoDB version > 4.2 without any need of 2nd write call(.insertMany()).
For your scenario on MongoDB version > 4.2 something like this will merge docs to second collection :
{ $merge: 'col2' } // Has to be final stage in aggregation
Note : If this has to be done periodically - no matter how you do this, try to minimize data that you're operating on, maybe maintain a time field & you can use that field to filter docs first & do this job, or you can also take advantage of _id to say we've done for all these docs in last run & we need to start from this docs - which helps you a lot to reduce data to be worked on. Additionally don't forget to maintain indexes.
I'm building a ranking system for my book collection ordered by award count and below is the result returns from my database
{
'awardCount' : 12000,
'year' : 1967,
'by' : 'IEEE Computer Society'
},
{
'awardCount' : 11230,
'year' : 1993,
'by' : 'National Academy of Engineering'
},
{
'awardCount' : 10600,
'year' : 1993,
'by' : 'National Academy of Engineering'
}
........about 10000+ more and sorted by awardCount
I use this query to get the result above
.find()
.sort({awardCount: -1 })
my question is if it is possible to have add 'rank' field in each item to show ranking order? For example
{
'awardCount' : 12000,
'year' : 1967,
'by' : 'IEEE Computer Society',
'rank': 1
},
{
'awardCount' : 11230,
'year' : 1993,
'by' : 'National Academy of Engineering',
'rank': 2
}
if not, what would be the best solution to get ranking in this situation? Thank you very much!
you could use mongodb aggregate to achieve this.You basically sort the documents by wins and kills and then push each user in a new array while preserving the index, this index is the users' rank after that you can simply use 'unwind' on this newly created array and find the user in question, this will give you the user's rank with other details.
Here is a sample aggregate pipeline:
[{
"$sort": {
"awardCount": -1
}
},
{
"$group": {
"_id": false,
"books": {
"$push": {
"_id": "$_id",
"by": "$by",
"year": "$year",
"awardCount": "$awardCount"
}
}
}
},
{
"$unwind": {
"path": "$books",
"includeArrayIndex": "rank"
}
}]
I have a variable var correctAnswers;
In my MongoDB I have the following document (below). I am trying to write a query that takes all of the "correct" fields from the "quiz" field and put them into their own array, so I can set that array equal to var correctAnswers;.
"title" : "Economics questions"
"quiz": "[{
"question": "Which of these involves the analysis of of a business's financial statements, often used in stock valuation?",
"choices": ["Fundamental analysis", "Technical analysis"],
"correct": 0
}, {
"question": "What was the name of the bond purchasing program started by the U.S. Federal Reserve in response to the 2008 financial crisis?",
"choices": ["Stimulus Package", "Mercantilism", "Quantitative Easing"],
"correct": 2
}, {
"question": "Which term describes a debt security issued by a government, company, or other entity?",
"choices": ["Bond", "Stock", "Mutual fund"],
"correct": 0
}, {
"question": "Which of these companies has the largest market capitalization (as of October 2015)?",
"choices": ["Microsoft", "General Electric", "Apple", "Bank of America"],
"correct": 2
}, {
"question": "Which of these is a measure of the size of an economy?",
"choices": ["Unemployment rate", "Purchasing power index", "Gross Domestic Product"],
"correct": 2
}]"
How should I go about that, or can someone point me in the right direction? I have tried projections, but should I do an aggregation? Thank you for any help.
Edit for clarity: the output I am looking for in this example is an array, [0,2,0,2,2]
you can get this result
[{correct:0},{correct:2},{correct:0},{correct:2}] but [0,2,0,2,2] type of result is not possible unless we use distinct
db.quiz.aggregate(
// Initial document match (uses index, if a suitable one is available)
{ $match: {
"title" : "Economics questions"
}},
// Convert embedded array into stream of documents
{ $unwind: '$quiz' },
},
// Note: Could add a `$group` by _id here if multiple matches are expected
// Final projection: exclude fields with 0, include fields with 1
{ $project: {
_id: 0,
score: "$quiz.correct"
}} )
db.users.find( { }, { "quiz.correct": 1,"_id":0 } )
// above query will return following output :
{
"quiz" : [
{
"correct" : 0
},
{
"correct" : 2
},
{
"correct" : 0
},
{
"correct" : 2
},
{
"correct" : 2
}
]
}
Process this output as per requirement in the node js.
Try this:
db.getCollection('quize').aggregate([
{$match:{_id: id }},
{$unwind:'$quiz'},
{$group:{
_id:null,
score: {$push:"$quiz.correct"}
}}
])
It will give you the expected output.
One way to achieve this through aggregation
db.collectionName.aggregate([
// use index key in match pipeline,just for e.g using title here
{ $match: { "title" : "Economics questions" }},
{ $unwind: "$quiz" },
{ $group: {
_id:null,
quiz: { $push: "$quiz.correct" }
}
},
//this is not required, use projection only if you want to exclude/include fields
{
$project: {_id: 0, quiz: 1}
}
])
Above query will give you the following output
{
"quiz" : [ 0, 2, 0, 2, 2 ]
}
Then simply process this output as per your need.