In a nodejs app with mongodb storage, I have the following query from user:
const rawQuery = [
'{"field":"ingredient","type":"AND","value":"green and blue"}',
'{"field":"ingredient","type":"AND","value":"black"}',
'{"field":"ingredient","type":"OR","value":"pink"}',
'{"field":"ingredient","type":"OR","value":"orange"}',
'{"field":"place","type":"AND","value":"london"}',
'{"field":"school","type":"NOT","value":"fifth"}',
'{"field":"food","type":"OR","value":"burger"}',
'{"field":"food","type":"OR","value":"pizza"}',
'{"field":"ownerFirstName","type":"AND","value":"Jimmy"}'
];
I have a collection called restaurant, and a collection called owners.
Would this query aim to handle such a search scenario?
const query = {
$and: : [
{ ingredient: 'green and blue' },
{ ingredient: 'black' },
{ $or : [
{ ingredient: 'pink' },
{ ingredient: 'orange' },
]
},
{ place: 'london' },,
{ school: { $ne: 'fifth' } },
{ $or : [
{ food: 'burger' },
{ food: 'pizza' },
]
}
]
};
How can I transform the rawQuery into this mongo query? (Given that it has to be dynamic, because I have many fields, and in this example I just included a couple of them.)
This example query aims to get the restaurants that match the description/place/school/food queries in the restaurant and also to match the owner's first name from another collection. Each restaurant document will have a ownerUuid field that points to the owner in the other collection.
What is the best solution to do a search in the mongodb for such a query in production env?
How can this be achieved with Elasticsearch?
Related
$project: {
_id: 1,
edited: 1,
game: {
gta: {
totalUserNumber: {
$reduce: {
input: "$gta.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$gta.users", <----- paginating this
},
DOTA2: {
totalUserNumber: {
$reduce: {
input: "$dota2.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$dota2.users", <------ paginating this
},
},
.... More Games
},
I have this $project. I have paginated the list of games by using $facet,$sort, $skip and $limit after $project.
I am trying also trying to paginate each game's userList. I have done to get the total value in order to calculate the page number and more.
But, I am struggling to apply $sort and $limit inside the $project. So far, I have just returned the document and then paginated with the return value. However, I don't think this is very efficient and wondering if there is any way that I can paginate the field inside the $project.
Is there any way that I can apply $sort and $limit inside the $project, in order to apply pagination to the fields and return?
------ Edit ------
this is for paginating the field. Because, I am already paginating the document (game list), I could not find any way that I can paginate the field, because I could not find any way that I can apply $facet to the field.
e.g. document
[
gta: {
userID: ['aa', 'bb', 'cc' ......],
},
dota: {
userID: ['aa', 'bb', 'cc' ......],
}
....
]
I am using $facet to paginate the list of games (dota, gta, lol and more). However, I did not want to return all the userID. I had to return the entire document and then paginate the userID to replace the json doc.
Now, I can paginate the field inside the aggregate pipeline by using $function.
thanks to Mongodb sort inner array !
const _function = function (e) {
e // <---- will return userID array. You can do what you want to do.
return {
};
};
game
.collection("game")
.aggregate([
{},
{
$set: {
"game": {
$function: {
body: _function,
args: ["$userID"],
lang: "js",
},
},
},
},
])
.toArray();
By using $function multiple time, you will be able to paginate the field. I don' really know if this is faster or not tho. Plus, make sure you can use $function. I read that you can't use this if you are on the free tier at Atlas.
What you are looking for is the $slice Operator.
It requires three parameters.
"$slice": [<Array>, <start-N>, <No-Of.elements to fetch>]
userList: {"$slice": ["$dota2.users", 20, 10]} // <-- Will ignore first 20 elements in array and gets the next 10
I am working on moving my database from sqlite3 to mongo. I went
through mongo university, yet I'm not sure I have found a really good
example of upsertting in bulk.
Use case : user uploads a data file with a list of players and their stats. The app needs to either update a player or add a new player if they do not already exist.
Current Implementation : Function takes a list of Players and creates SQL statement.
let template = '(Player.Rank, Player.Player_ID, Player.Player, Player.Score, TTP, 1),(Player.Rank, Player_ID, ...) ... (... TTP, 1)';
const stmt = `INSERT INTO playerStats (Rank, Player_ID, Player, Score, TPP, Games_Played)
VALUES ${template}
ON CONFLICT(Player_ID) DO UPDATE
SET Score = Score+excluded.Score, TPP=TPP+excluded.TPP, Games_Played=Games_Played+1`;
db.run(stmt, callback);
Im hoping to have each document be a league which contains players, games, and managers.
Mongo DB document template
{
"LEAGUE_ID": "GEO_SAM",
"Players": [
{
"id": "PlayerID",
"name": "Player",
"score": "Score",
"rank": "Rank",
"xPlayed": "Games_Played",
"ttp": "TTP"
}
],
"Managers": [
{...}
],
"Games": [
{...}
]
}
I am totally lost and not sure how to get this done. Do I need to create a loop and ask mongo to upsert on each iteration? I have searched through countless examples but all of them use static hard coded values.
Here is my testing example.
const query = { League_id : "GEO_SAM", Players.$.id: $in{ $Players }};
const update = { $inc: { score: $Player.Score}};
const options = { upsert: true };
collection.updateMany(query, update, options);
I also don't understand how to pass the entire player object to push to the array if the player_id isn't found.
My solution was to create a metaData field containing the league ID with a single player. If anyone else has a better solution I would love to hear from you.
{
MetaData: { "LEAGUE_ID": "GEO_SAM"},
Player: {
"id": "PlayerID",
"name": "Player",
"score": "Score",
"rank": "Rank",
"xPlayed": "Games_Played",
"ttp": "TTP"
}
}
Then I mapped over the values and inserted each one.
client.connect().then((client) => {
const db = client.db(dbName);
const results = Players.map((player) => {
db.collection('Players').updateOne(
{ Player_Name: player.Player_ID },
{
$setOnInsert: {
Player_ID: player.Player_ID,
Player: player.Player,
Rank: player.Rank,
},
$inc: { Score: player.Score, Games_Played: 1, TPP: player.TPP },
},
{ upsert: true, multi: true },
);
});
So basically i have model with a bunch of string fields like so:
const Schema: Schema = new Schema(
{
title: {
type: String,
trim: true
},
description: {
type: String,
trim: true
},
...
}
);
Schema.index({ '$**': 'text' });
export default mongoose.model('Watch', Schema);
where I index all of them.
Now when I search being that this schema is used as a ref for another model I do a search like this where user is an instance of the other model
const { search, limit = 5 } = req.query;
const query = search && { match: { $text: { $search: new RegExp(search, 'i') } } };
const { schemaRes } = await user
.populate({
path: 'schema',
...query,
options: {
limit
}
})
.execPopulate();
and the searching itself seems to work ok, the problem is when search fields starts to be more specific it seems to me the it does not regard it well.
Example
db
{ title: 'Rolex', name: 'Submariner', description: 'Nice' }
{ title: 'Rolex', name: 'Air-King', description: 'Nice' }
When the search param is Rolex I get both items which is ok but when the search param becomes Rolex Air-King i keep on getting both items which to me is not ok because I would rather get only one.
Is there something I could do to achieve this?
Returning both items is correct, since both items match your search params, but with different similarity score.
You can output the similarity score to help sorting the result.
user.aggregate([
{ $match: { $text: { $search: "Rolex Air-King" } } },
{ $set: { score: { $meta: "textScore" } } }
])
// new RegExp("Rolex Air-King", 'i') is not necessary and even invalid,
// as $search accepts string and is already case-insensitive by default
The query will return
[{
"_id": "...",
"title": "Rolex",
"name": "Air-King",
"description": "Nice",
"score": 2.6
},
{
"_id": "....",
"title": "Rolex",
"name": "Submariner",
"description": "Nice",
"score": 1.1
}]
Since the second result item matches your search query (even partially), MongoDB returns it.
You could use the score to help sort the items. But determining the right threshold to filter the result is complex, as the score depends on the word count as well.
On a side note: You can assign different weights to the fields if they are not equally important
https://docs.mongodb.com/manual/tutorial/control-results-of-text-search/
I'm a little newer to using MongoDB and NoSQL for my stack. ULTIMATELY, I'm switching to NoSQL for the fact of JSON parsed data already. I can accomplish what I want in MySQL - but I'm also attempting to teach myself something new.
Currently, I have no problem getting connected and setting up my Schemas for NodeJS. I have a document within MongoDB that returns what my customers pay on fuel according to National Averages - our customers give us ranges for us to be able to find what specific dollar amount they are paying.
My MongoDB Document looks as the following :
main_database
|- customerFSC (name of document)
|--
{
"_id":{"$oid":"5e5ecc04e8da861114079ab2"},
"custID":"01",
"custName":"Customer ABC",
"avgLow":["1.19","1.24","1.29","1.34","1.39","1.44","1.49","1.54","1.59","1.64","1.69","1.74","1.79","1.84","1.89","1.94","1.99"],
"avgHigh":["1.239","1.289","1.339","1.389","1.439","1.489","1.539","1.589","1.639","1.689","1.739","1.789","1.839","1.889","1.939","1.989","2.039"],
"custFscPM":["0.01","0.02","0.03","0.04","0.05","0.06","0.07","0.08","0.09","0.10","0.11","0.12","0.13","0.14","0.15","0.16","0.17"]
}
If fuel average for the week is at 1.215 - it would be ranged between the avgLow of 1.19 and the avgHigh of 1.239 but return the actual pay of 0.01 in custFscPM
My MongoDB Mongoose Node code is as follows
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const fscCustSchema = new Schema({
custID: String,
custName: String,
avgLow: Array,
avgHigh: Array,
custFscPM: Array
},
{ collection: 'customerFSC' }
);
module.exports = mongoose.model('customerFSC', fscCustSchema);
fscCustModel.find( { $and: [
{ $gte: [ { avgLow: USAFscAVG } ] },
{ $lte: [ { avgHigh: USAFscAVG } ] }
]},function(err2,resp2) {
if (err2) console.log("Error Thrown Looking up Fuel:" + err2);
console.log(resp);
});
You can use aggregation which is way to powerful way of querying MongoDB than using .find(), Try below query :
db.collection.aggregate([
{
$project: {
custFscPM: {
$arrayElemAt: [
"$custFscPM",
{
$indexOfArray: [
"$avgLow",
{
$arrayElemAt: [
{
$filter: {
input: "$avgLow",
cond: {
$gte: [
{
$toDouble: "$$this"
},
1.94 // Your input
]
}
}
},
0
]
}
]
}
]
}
}
}
])
Looking at your data, in this query we're finding the element in avgLow array which is greater than or equal to passed in value & getting it's index & find the element from custFscPM at the same index would resolve your requirement.
Note : As your values in avgLow are strings we need to convert those to double in order to do comparison.
Test : MongoDB-Playground
I am trying to get a list of users with total transaction count and each user should have latest transaction detail as embedded object using MongoDB's aggregate pipelines to fetch results in GET API.
I have the following database schema:
User: _id, name, phone, address
Product: _id, name, unit_price, description
Transaction: _id, date, product_id(ref to Product), user_id(ref to User), quantity, total_price
Expected Response JSON
[
{
name:"",
phone:"",
address:"",
total_transaction:
latest_transaction_detail: {
product_id:
quantity:
total_price:
}
},
{
name:"",
phone:"",
address:"",
total_transaction:
latest_transaction_detail: {
product_id:
quantity:
total_price:
}
}
]
How do I generate an aggregate query to return the above?
You can achieve this by running an aggregation query.
A lookup stage will join your User collection with your Transaction collection (no need to join Product in your expected result). Its pipeline is splitted with $facet, for both get the count result and the latest transaction for that user
A project stage will reshape your data and extract array elements to documents.
Here's such a query :
db.User.aggregate(
[
{
$lookup:
{
from: "Transaction",
let: { userId: "$_id" },
pipeline: [
{$facet:
{count:[{$match:{$expr:{$eq:["$$userId","$user_id"]}}}, {$count:"total_transaction"}],
latest:[
{$match:{$expr:{$eq:["$$userId","$user_id"]}}},
{$sort:{date:-1}},
{$limit:1}]
} }],
as: "transactions"
}
},
{
$project: {
last_name:1,
phone:1,
address:1,
total_transaction : {
$let:{
vars:{
count:{
$arrayElemAt:["$transactions.count",0]
}
},
in:{
$arrayElemAt:["$$count.total_transaction",0]
}
}
},
latest_transaction : {
$let:{
vars:{
latest:{
$arrayElemAt:["$transactions.latest",0]
}
},
in:{
$arrayElemAt:["$$latest",0]
}
}
},
}
},
]
);