GET request with $function in Mongodb - node.js

Summary:
I am trying to Combine the result of Mongodb aggregation with a third party API and I can't find anything relevant to it.
Explanation:
The below Express route finds all Games that comes after the provided Date and have not been cancelled. The next step is to get some data of that single game from the Third party API and attach it to the object and continue further in the pipeline
Issue:
It seems that you can't have a XHR request inside the $function (I didn't find anything in the official documentation so I'm not sure on that)
const today = moment();
today.year(2021);
today.month(5);
let response = await Game.aggregate([
{
$match: {
$expr: {
$and: [
{ $gte: ["$date", moment(today).startOf('day').toDate()] },
{ $eq: ["$canceled", false] },
]
}
}
},
{ $sort: { date: 1 } },
{
$addFields: {
boxScore: {
$function:
{
body: async function (season, week, home_team) {
const result = await axios.get(
`SINGLEGAMEURL/${season}/${week}/${home_team}`,
{
headers: {
'Subscription-Key': 'SOMEKEY',
},
}
);
return result.data;
},
args: ["$season", '$week', 'home_team'],
lang: "js"
}
}
}
}
]);
I would really appreciate any help/direction on this, Cheers!

I doubt that you can use asynchronous functions in $function, because they return a promise that resolves to result.data, rather than the data themselves. Instead, consider performing the asynchronous operation in your express middleware, after the MongoDB operation. Something like this:
app.use("/path", async function(req, res) {
const today = moment();
today.year(2021);
today.month(5);
let response = await Game.aggregate([
{$match: ...},
{$sort: {date: 1}}
]).toArray();
await Promise.all(response.map(row => axios.get(
`SINGLEGAMEURL/${row.season}/${row.week}/${row.home_team}`,
{headers: {'Subscription-Key': 'SOMEKEY'}}
).then(function(result) {
row.boxScore = result.data;
})));
res.json(response);
});
(Probably the Promise.all can be avoided, but I'm not experienced enough with async/await to know how.)

Related

How to fix 'Updated field become undefined' issue in MongoDb query

I'm currently writing a mongoDB query to update set of queries in the db, The requirement is to fetch the db entries that has mediaOrigin = 'Kaltura' and the mediaType= 'YOUTUBE', then update the entries as mediaOrigin= 'AWS' and the mediaUrl.dataUrl and mediaUrl.downloadUrl with the value of mediaUrl.originalUrl. So I have completed a script to update the relevent queries but the value that the mediaUrl.dataUrl and mediaUrl.downloadUrl taking is undefined. So how can I solve that, I need to fill that two fields with the value of mediaUrl.originalUrl.
Here is the query I have written,
try {
db.getCollection('media').find({mediaOrigin: { $eq: 'KALTURA' }, mediaType: {$eq: 'YOUTUBE' }, delete: false
})
.forEach(function(media) {
var youtubeUrl = media.mediaUrl.originalUrl;
var Url = youtubeUrl;
db.getCollection('media').update(
{
_id: media._id
},
{
$set: {
'mediaUrl.downloadUrl': Url,
'mediaUrl.dataUrl': Url,
mediaOrigin: 'AWS'
}
}
);
});} catch (e) {
print(e);}
So how can I solve that.
Here I have attached the example entry in the db that I need to update.
You are attaching .forEach end of the .find() method get results from your collection.
You have to wait to get results before sending the result into foreach.
So use it like this:
const medias = await db.getCollection('media').find({
mediaOrigin: { $eq: 'KALTURA' },
mediaType: {$eq: 'YOUTUBE' },
delete: false
}).toArray();
medias.forEach(async(media) => {
var youtubeUrl = media.mediaUrl.originalUrl;
var Url = youtubeUrl;
await db.getCollection('media').update(
{
_id: media._id
},
{
$set: {
'mediaUrl.downloadUrl': Url,
'mediaUrl.dataUrl': Url,
mediaOrigin: 'AWS'
}
}
);
});

I have 2 issues with mongoose aggregation and index method

I have 2 issues
FIRST ONE:
I am trying to make review schema that a user should add 1 review per bootcamp
Code:
ReviewSchema.index({ bootcamp: 1, user: 1 }, { unique: true });
It doesnt work .. and the user still can add more than one review
SECOND ISSUE:
I am trying to calculate the averagerating of reviews but it doesn`t get added to the db when am fetching the bootcamps
Code:
// Static Method to get the avg rating of reviews and save
ReviewSchema.statics.getAverageRating = async function (bootcampId) {
const obj = await this.aggregate([
{
$match: { bootcamp: bootcampId },
},
{
$group: {
_id: '$bootcamp',
averageRating: { $avg: '$rating' },
},
},
]);
try {
await this.model('Bootcamp').findByIdAndUpdate(bootcampId, {
averageRating: obj[0].averageRating,
});
} catch (err) {
console.log(err);
}
//Call averageRating after save
ReviewSchema.post('save', async function () {
await this.constructor.getAverageRating(this.bootcamp);
});
//Call averageRating before remove
ReviewSchema.pre('remove', async function () {
await this.constructor.getAverageRating(this.bootcamp);
});
** It doesnt work and the averagerating never gets added to the database (as a bootcamp`s field)**
I Did the same as the tutorial and it didn`t work at the first but then i figured out that missing a semi-colon.

Can mongodb send query pipelining with no loop?

I'm new to NodeJS and MongoDB.
I wanna get user's profile with one user's following list. If I use RDB, it was so simple with EQ join but I didn't have much experience of MongoDB, I don't know how.
Sample data below.
// list of users
[
{
_id: "oid_1",
nickname: "user_01",
link: "url/user_01"
},
{
_id: "oid_2",
nickname: "user_02",
link: "url/user_02"
},
{
_id: "oid_3",
nickname: "user_03",
link: "url/user_03"
}
...
]
user_01's followList
[
{
followOid: "foid_1",
userOid: "user_01"
},
{
followOid: "foid_2",
userOid: "user_02"
},
]
My solution is, get follow list, then use loop with follows.findOne() like below
const dataSet = [];
Follow.getFollowerList(userId) // for pipeline, use promise
.exec()
.then( async (result) => { // no async-await, no data output...
for (let data of result) {
let temp = await Users.getUserInfo( // send query for each data, I think it's not effective
data.userId,
{ nickname: 1, link: 1 }
);
dataSet.push(temp);
}
return dataSet;
})
.then((data) => {
res.status(200).json(data);
})
.catch( ... )
I think it's not best solution. If you are good at mongodb, plz save my life :)
thanks
One option would be to use aggregation.
const userId = 'Fill with UserId';
const pipe = [
{
'$match': {
'_id': userId
}
}, {
'$lookup': {
'from': 'followListCollectionName',
'localField': '_id',
'foreignField': 'userOid',
'as': 'followList'
}
}
];
const result = await UserModel.aggregate(pipeline);
and then you can find an array in result which contains one user with given Id ( and more if there are with same Id) and result[0].followList you can find follow objects as array
Second Option is to use virtuals
https://mongoosejs.com/docs/tutorials/virtuals.html
but for this schema of your collection needs some changes.
Good luck

How to implement map function of Mongodb cursor in node.js (node-mondodb-native)

I am trying to implement following MongoDB query in NodeJS
db.tvseries.find({}).map(function(doc){
var userHasSubscribed = false;
doc.followers && doc.followers.forEach(function(follower) {
if(follower.$id == "abc") {
userHasSubscribed = true;
}
});
var followers = doc.followers && doc.followers.map(function(follower) {
var followerObj;
db[follower.$ref].find({
"_id" : follower.$id
}).map(function(userObj) {
followerObj = userObj;
});
return followerObj;
});
return {
"id": doc.name,
"userHasSubscribed": userHasSubscribed,
"followers": followers || []
};
})
Following is the db
users collection
{
"id": ObjectId("abc"),
"name": "abc_name"
},
{
"id": ObjectId("def"),
"name": "def_name"
},
{
"id": ObjectId("ijk"),
"name": "ijk_name"
}
tvseries collection
{
"id": ObjectId("123"),
"name": "123_name",
"followers": [
{
"$ref": "users",
"$id": ObjectId("abc"),
},
{
"$ref": "users",
"$id": ObjectId("def"),
}
]
},
{
"id": ObjectId("456"),
"name": "456_name",
"followers": [
{
"$ref": "users",
"$id": ObjectId("ijk"),
},
]
},
{
"id": ObjectId("789"),
"name": "789_name"
}
I am not able to figure out how to execute the above MongoDB query in NodeJS with the help of node-mongodb-native plugin.
I tried the below code but then I get TypeError: undefined is not a function at .map
var collection = db.collection('users');
collection.find({}).map(function(doc) {
console.log(doc);
});
How to execute .map function in NodeJS?
Thanks in advance
I struggled with this for some time. I found that by adding .toArray() after the map function works.
You could even skip map and only add .toArray() to get all the documents fields.
const accounts = await _db
.collection('accounts')
.find()
.map(v => v._id) // leaving this out gets you all the fields
.toArray();
console.log(accounts); // [{_id: xxx}, {_id: xxx} ...]
Please take note that in order for map to work the function used must return something - your example only console.logs without returning a value.
The forEach solution works but I really wanted map to work.
I know that I'm pretty late but I've arrived here by searching on Google about the same problem. Finally, I wasn't able to use map function to do it, but using forEach did the trick.
An example using ES6 and StandardJS.
let ids = []
let PublicationId = ObjectID(id)
feeds_collection
.find({PublicationId})
.project({ _id: 1 })
.forEach((feed) => {
ids.push(feed._id)
}, () => done(ids))
To echo #bamse's anwer, I got it working with .toArray(). Here is an async example:
async function getWordArray (query) {
const client = await MongoClient.connect(url)
const collection = client.db('personal').collection('wordBank')
const data = await collection.find(query).map(doc => doc.word).toArray()
return data
}
Then I use it in my Express route like this:
app.get('/search/:fragment', asyncMiddleware(async (req, res, next) => {
const result = await getWordArray({word: 'boat'})
res.json(result)
}))
Finally, if you need a guide to async/await middleware in NodeJS, here is a guide: https://medium.com/#Abazhenov/using-async-await-in-express-with-node-8-b8af872c0016
map returns a cursor, toArray returns a Promise that will execute a cursor and return it's results. That may be an array of the original query find, limit etc. or a promise of an array of those result piped through a function.
This is typically useful when you want to take the documents of the cursor and process that (maybe fetch something else) while the cursor is still fetching documents, as opposed to waiting until they have all been fetched to node memory
Consider the example
let foos = await db.collection("foos")
.find()
.project({
barId: 1
})
.toArray() // returns a Promise<{barId: ObjectId}[]>
// we now have all foos into memory, time to get bars
let bars = await Promise.all(foos.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
})))
this is roughly equivalent to
bars = await db.collection("foos")
.find()
.project({
barId: 1
})
.toArray() // returns a Promise<{barId: ObjectId}[]>
.then(docs => docs
.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
})))
using map you can perform the operation asynchrounsly and (hopefully) more efficiently
bars = await db.collection("foos")
.find()
.project({
barId: 1
})
.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
}))
.toArray()
.then(barPromises => Promise.all(barPromises)) // Promise<Bar[]>
The main point is that map is simply a function to be applied to the results fetched by the cursor. That function won't get executed until you turn it into a Promise, using either forEach or more sensibly, map

MongoDB and NodeJS get related data from 3 collections

i have a mongoDB query to get data with $group and $count.
This data contains the _id from other documents collection.
How can i get the other documents by its _id in NodeJS and MongoDB asyncrohnous?
db.orders.aggregate([
{$match: { 'works.TechnicianId': {$in:['53465f9d519c94680327965d','5383577a994be8b9a9e3f01e']},
'works.Date': {$gte: ISODate("2013-05-21T06:40:20.299Z"), $lt: ISODate("2016-05-21T06:40:20.299Z")}}},
{$unwind: "$works" },
{$group: {_id: "$works.TechnicianId",total:{$sum:'$works.price'},ordersId: { $push: "$_id" }}},
])
This is the result:
{
"result" : [
{
"_id" : "53465f9d519c94680327965d",
"total" : 198,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
},
{
"_id" : "5383577a994be8b9a9e3f01e",
"total" : 22,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
}
],
"ok" : 1
}
Now i need to get from orders collection the documents with id from ordersId, and from other collection the documents with _id from the result _id field.
I try with this:
var collection = db.collection('orders');
var result = [];
collection.aggregate([
{
$match: {
'works.TechnicianId': {
$in: ids
},
'works.Date': {
$gte: new Date(startDate),
$lt: new Date(endDate)
}
}
},
{
$unwind: "$works"
},
{
$group: {
_id: "$works.TechnicianId",
total: {
$sum: '$works.price'
},
orderId: {
$push: "$_id"
}
}
}
],
function (e, docs) {
if (e) {
error(e);
}
var usersCollection = db.collection('users');
_.each(docs, function (doc) {
usersCollection.findOne({_id: new ObjectID(doc._id)}, function (e, doc) {
doc.tech = doc;
});
doc.orders = [];
_.each(doc.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
doc.orders.push(order);
});
});
success(docs);
});
});
But the success its called before all the _.eachs are finished..Any help, or idea?
Edit:
I try with Q promises, this is my code:
var usersCollection = db.collection('users');
var promises = [];
_.each(reports, function (report) {
var promise = usersCollection.findOne({_id: new ObjectID(report._id)}).then(
function (e, orderUserReported) {
if (e) {
error(e);
}
report.tech = orderUserReported;
_.each(orderUserReported.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
report.orders.push(order);
});
});
});
promises.push(promise);
});
Q.allSettled(promises).then(success(reports));
and the error:
/Users/colymore/virteu/aa/server/node_modules/mongodb/lib/mongodb/connection/base.js:245
throw message;
^
TypeError: Cannot call method 'then' of undefined
Because of asynchronous execution you have to wait until results are returned. There are several options available:
async library https://github.com/caolan/async
promises https://github.com/kriskowal/q
Async is closer to your current code, you could use async.parallel https://github.com/caolan/async#parallel to wait untill you get data back
Update
Mongoose functions don't return Q promises, so you need to convert mongoose calls to promises by using something like Q.denodeify(User.findOne.bind(models.User))({ _id: userId}).then(...
For your case Q.denodeify(userCollection.findOne.bind(userCollection))({_id: new ObjectID(report._id)}).then(...
Short answer: Use promises. Look at Q.allSettled ( https://github.com/kriskowal/q )
Just run success asynchronously when all subtask are done.
Also using https://github.com/iolo/mongoose-q package may be helpful to not combine mongoose promises with Q ones if you want use mongoose in your mongo.

Resources