How to implement map function of Mongodb cursor in node.js (node-mondodb-native) - node.js

I am trying to implement following MongoDB query in NodeJS
db.tvseries.find({}).map(function(doc){
var userHasSubscribed = false;
doc.followers && doc.followers.forEach(function(follower) {
if(follower.$id == "abc") {
userHasSubscribed = true;
}
});
var followers = doc.followers && doc.followers.map(function(follower) {
var followerObj;
db[follower.$ref].find({
"_id" : follower.$id
}).map(function(userObj) {
followerObj = userObj;
});
return followerObj;
});
return {
"id": doc.name,
"userHasSubscribed": userHasSubscribed,
"followers": followers || []
};
})
Following is the db
users collection
{
"id": ObjectId("abc"),
"name": "abc_name"
},
{
"id": ObjectId("def"),
"name": "def_name"
},
{
"id": ObjectId("ijk"),
"name": "ijk_name"
}
tvseries collection
{
"id": ObjectId("123"),
"name": "123_name",
"followers": [
{
"$ref": "users",
"$id": ObjectId("abc"),
},
{
"$ref": "users",
"$id": ObjectId("def"),
}
]
},
{
"id": ObjectId("456"),
"name": "456_name",
"followers": [
{
"$ref": "users",
"$id": ObjectId("ijk"),
},
]
},
{
"id": ObjectId("789"),
"name": "789_name"
}
I am not able to figure out how to execute the above MongoDB query in NodeJS with the help of node-mongodb-native plugin.
I tried the below code but then I get TypeError: undefined is not a function at .map
var collection = db.collection('users');
collection.find({}).map(function(doc) {
console.log(doc);
});
How to execute .map function in NodeJS?
Thanks in advance

I struggled with this for some time. I found that by adding .toArray() after the map function works.
You could even skip map and only add .toArray() to get all the documents fields.
const accounts = await _db
.collection('accounts')
.find()
.map(v => v._id) // leaving this out gets you all the fields
.toArray();
console.log(accounts); // [{_id: xxx}, {_id: xxx} ...]
Please take note that in order for map to work the function used must return something - your example only console.logs without returning a value.
The forEach solution works but I really wanted map to work.

I know that I'm pretty late but I've arrived here by searching on Google about the same problem. Finally, I wasn't able to use map function to do it, but using forEach did the trick.
An example using ES6 and StandardJS.
let ids = []
let PublicationId = ObjectID(id)
feeds_collection
.find({PublicationId})
.project({ _id: 1 })
.forEach((feed) => {
ids.push(feed._id)
}, () => done(ids))

To echo #bamse's anwer, I got it working with .toArray(). Here is an async example:
async function getWordArray (query) {
const client = await MongoClient.connect(url)
const collection = client.db('personal').collection('wordBank')
const data = await collection.find(query).map(doc => doc.word).toArray()
return data
}
Then I use it in my Express route like this:
app.get('/search/:fragment', asyncMiddleware(async (req, res, next) => {
const result = await getWordArray({word: 'boat'})
res.json(result)
}))
Finally, if you need a guide to async/await middleware in NodeJS, here is a guide: https://medium.com/#Abazhenov/using-async-await-in-express-with-node-8-b8af872c0016

map returns a cursor, toArray returns a Promise that will execute a cursor and return it's results. That may be an array of the original query find, limit etc. or a promise of an array of those result piped through a function.
This is typically useful when you want to take the documents of the cursor and process that (maybe fetch something else) while the cursor is still fetching documents, as opposed to waiting until they have all been fetched to node memory
Consider the example
let foos = await db.collection("foos")
.find()
.project({
barId: 1
})
.toArray() // returns a Promise<{barId: ObjectId}[]>
// we now have all foos into memory, time to get bars
let bars = await Promise.all(foos.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
})))
this is roughly equivalent to
bars = await db.collection("foos")
.find()
.project({
barId: 1
})
.toArray() // returns a Promise<{barId: ObjectId}[]>
.then(docs => docs
.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
})))
using map you can perform the operation asynchrounsly and (hopefully) more efficiently
bars = await db.collection("foos")
.find()
.project({
barId: 1
})
.map(doc => db
.collection("bars")
.findOne({
_id: doc.barId
}))
.toArray()
.then(barPromises => Promise.all(barPromises)) // Promise<Bar[]>
The main point is that map is simply a function to be applied to the results fetched by the cursor. That function won't get executed until you turn it into a Promise, using either forEach or more sensibly, map

Related

Why Is Firebase-admin Not returning the Documents but a different object in express

I Am trying to create a node wrapper around my firebase function on a different server
technically using firebase as my main Database
And anytime I try to read using the default query function in the doc
const db = firebase-admin.firestore()
const data = await db.collection("cities").get()
Now instead of getting an array of objects like this
"_query": {
"_firestore": {
"projectId": "fir-demo-jd1"
},
"_queryOptions": {
"parentPath": {
"segments": []
},
"collectionId": "bets",
"converter": {},
"allDescendants": false,
"fieldFilters": [
],
"fieldOrders": [
],
"limit": null,
"limitType": 0,
"kindless": false,
"requireConsistency": true
},
"_serializer": {
"allowUndefined": false
},
"_allowUndefined": false
},
"_readTime": {
"_seconds": 1676797103,
"_nanoseconds": 433061000
},
"_size": 2,
"_materializedDocs": null,
"_materializedChanges": null
},
What I want is to be able to see the list of country alone
As I woud later to use paganation also on the data thats like
.limit() with ease
So Apartly I have to map and convert it to JSON using this function
const db = firebase-admin.firestore()
const data = await db.collection("cities").get()
console.log(docs.map((doc)=>doc.data()))
I thought firebase handled all this under the hood
db.collection("cities").get() returns a QuerySnapshot and from there you can loop over the DocumentSnapshots, as explained in the doc:
const querySnapshot = await db.collection("cities").get();
querySnapshot.forEach((doc) => {
// doc.data() is never undefined for query doc snapshots
console.log(doc.id, " => ", doc.data());
});

GET request with $function in Mongodb

Summary:
I am trying to Combine the result of Mongodb aggregation with a third party API and I can't find anything relevant to it.
Explanation:
The below Express route finds all Games that comes after the provided Date and have not been cancelled. The next step is to get some data of that single game from the Third party API and attach it to the object and continue further in the pipeline
Issue:
It seems that you can't have a XHR request inside the $function (I didn't find anything in the official documentation so I'm not sure on that)
const today = moment();
today.year(2021);
today.month(5);
let response = await Game.aggregate([
{
$match: {
$expr: {
$and: [
{ $gte: ["$date", moment(today).startOf('day').toDate()] },
{ $eq: ["$canceled", false] },
]
}
}
},
{ $sort: { date: 1 } },
{
$addFields: {
boxScore: {
$function:
{
body: async function (season, week, home_team) {
const result = await axios.get(
`SINGLEGAMEURL/${season}/${week}/${home_team}`,
{
headers: {
'Subscription-Key': 'SOMEKEY',
},
}
);
return result.data;
},
args: ["$season", '$week', 'home_team'],
lang: "js"
}
}
}
}
]);
I would really appreciate any help/direction on this, Cheers!
I doubt that you can use asynchronous functions in $function, because they return a promise that resolves to result.data, rather than the data themselves. Instead, consider performing the asynchronous operation in your express middleware, after the MongoDB operation. Something like this:
app.use("/path", async function(req, res) {
const today = moment();
today.year(2021);
today.month(5);
let response = await Game.aggregate([
{$match: ...},
{$sort: {date: 1}}
]).toArray();
await Promise.all(response.map(row => axios.get(
`SINGLEGAMEURL/${row.season}/${row.week}/${row.home_team}`,
{headers: {'Subscription-Key': 'SOMEKEY'}}
).then(function(result) {
row.boxScore = result.data;
})));
res.json(response);
});
(Probably the Promise.all can be avoided, but I'm not experienced enough with async/await to know how.)

Query with Mongoose multiple times without nesting

I'm trying to generate a document with node.js that needs to run multiple unrelated database queries from a mongo database.
Here is my current code:
Data.find({}, function(err, results) {
if (err) return next(err);
//finished getting data
res.render('page');
}
}
The problem is if I try to run another query, I seem to have to nest it within the first one so that it waits for the first one to finish before starting, and then I have to put res.render() within the innermost nested query (if I don't, res.render() will be called before the database is finished grabbing data, and it wont be rendered with the page).
What I have to do:
Data.find({}, function(err, results) {
if (err) return next(err);
//finished getting data
Data2.find({}, function(err, results2) {
if (err) return next(err);
//finished getting data 2
res.render('page');
}
}
}
}
I am going to have more than 2 queries, so if I keep nesting them it's going to get really messy really fast. Is there a cleaner way to do this, such as a way to make the code wait until all the data is returned and the function is run before continuing with the script?
For mongoose you can probably just do a Promise.all() and use .concat() on the resulting arrays of each query.
As a full demo:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
var d1Schema = new Schema({ "name": String });
var Data1 = mongoose.model("Data1", d1Schema);
var d2Schema = new Schema({ "title": String });
var Data2 = mongoose.model("Data2", d2Schema);
mongoose.set('debug',true);
mongoose.connect('mongodb://localhost/test');
async.series(
[
// Clean
function(callback) {
async.each([Data1,Data2],function(model,callback) {
model.remove({},callback)
},callback);
},
// Setup some data
function(callback) {
async.each([
{ "name": "Bill", "model": "Data1" },
{ "title": "Something", "model": "Data2" }
],function(data,callback) {
var model = data.model;
delete data.model;
mongoose.model(model).create(data,callback);
},callback);
},
// Actual Promise.all demo
function(callback) {
Promise.all([
Data1.find().exec(),
Data2.find().exec()
]).then(function(result) {
console.log([].concat.apply([],result));
callback()
}).catch(callback);
}
],
function(err) {
if (err) throw err;
mongoose.disconnect();
}
)
I'm just mixing in async there for brevity of example, but the meat of it is in:
Promise.all([
Data1.find().exec(),
Data2.find().exec()
]).then(function(result) {
console.log([].concat.apply([],result));
})
Where the Promise.all() basically waits for and combines the two results, which would be an "array of arrays" here but the .concat() takes care of that. The result will be:
[
{ _id: 59420fd33d48fa0a490247c8, name: 'Bill', __v: 0 },
{ _id: 59420fd43d48fa0a490247c9, title: 'Something', __v: 0 }
]
Showing the objects from each collection, joined together in one array.
You could also use the async.concat method as an alternate, but unless you are using the library already then it's probably just best to stick to promises.

MongoDB and NodeJS get related data from 3 collections

i have a mongoDB query to get data with $group and $count.
This data contains the _id from other documents collection.
How can i get the other documents by its _id in NodeJS and MongoDB asyncrohnous?
db.orders.aggregate([
{$match: { 'works.TechnicianId': {$in:['53465f9d519c94680327965d','5383577a994be8b9a9e3f01e']},
'works.Date': {$gte: ISODate("2013-05-21T06:40:20.299Z"), $lt: ISODate("2016-05-21T06:40:20.299Z")}}},
{$unwind: "$works" },
{$group: {_id: "$works.TechnicianId",total:{$sum:'$works.price'},ordersId: { $push: "$_id" }}},
])
This is the result:
{
"result" : [
{
"_id" : "53465f9d519c94680327965d",
"total" : 198,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
},
{
"_id" : "5383577a994be8b9a9e3f01e",
"total" : 22,
"ordersId" : [
ObjectId("537b5ea4c61b1d1743f4341f"),
ObjectId("537b4633021d75bd36863f29")
]
}
],
"ok" : 1
}
Now i need to get from orders collection the documents with id from ordersId, and from other collection the documents with _id from the result _id field.
I try with this:
var collection = db.collection('orders');
var result = [];
collection.aggregate([
{
$match: {
'works.TechnicianId': {
$in: ids
},
'works.Date': {
$gte: new Date(startDate),
$lt: new Date(endDate)
}
}
},
{
$unwind: "$works"
},
{
$group: {
_id: "$works.TechnicianId",
total: {
$sum: '$works.price'
},
orderId: {
$push: "$_id"
}
}
}
],
function (e, docs) {
if (e) {
error(e);
}
var usersCollection = db.collection('users');
_.each(docs, function (doc) {
usersCollection.findOne({_id: new ObjectID(doc._id)}, function (e, doc) {
doc.tech = doc;
});
doc.orders = [];
_.each(doc.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
doc.orders.push(order);
});
});
success(docs);
});
});
But the success its called before all the _.eachs are finished..Any help, or idea?
Edit:
I try with Q promises, this is my code:
var usersCollection = db.collection('users');
var promises = [];
_.each(reports, function (report) {
var promise = usersCollection.findOne({_id: new ObjectID(report._id)}).then(
function (e, orderUserReported) {
if (e) {
error(e);
}
report.tech = orderUserReported;
_.each(orderUserReported.orderId, function (queryOrder) {
collection.findOne({_id: new ObjectID(queryOrder._id)}, function (e, order) {
report.orders.push(order);
});
});
});
promises.push(promise);
});
Q.allSettled(promises).then(success(reports));
and the error:
/Users/colymore/virteu/aa/server/node_modules/mongodb/lib/mongodb/connection/base.js:245
throw message;
^
TypeError: Cannot call method 'then' of undefined
Because of asynchronous execution you have to wait until results are returned. There are several options available:
async library https://github.com/caolan/async
promises https://github.com/kriskowal/q
Async is closer to your current code, you could use async.parallel https://github.com/caolan/async#parallel to wait untill you get data back
Update
Mongoose functions don't return Q promises, so you need to convert mongoose calls to promises by using something like Q.denodeify(User.findOne.bind(models.User))({ _id: userId}).then(...
For your case Q.denodeify(userCollection.findOne.bind(userCollection))({_id: new ObjectID(report._id)}).then(...
Short answer: Use promises. Look at Q.allSettled ( https://github.com/kriskowal/q )
Just run success asynchronously when all subtask are done.
Also using https://github.com/iolo/mongoose-q package may be helpful to not combine mongoose promises with Q ones if you want use mongoose in your mongo.

mongodb/mongoose findMany - find all documents with IDs listed in array

I have an array of _ids and I want to get all docs accordingly, what's the best way to do it ?
Something like ...
// doesn't work ... of course ...
model.find({
'_id' : [
'4ed3ede8844f0f351100000c',
'4ed3f117a844e0471100000d',
'4ed3f18132f50c491100000e'
]
}, function(err, docs){
console.log(docs);
});
The array might contain hundreds of _ids.
The find function in mongoose is a full query to mongoDB. This means you can use the handy mongoDB $in clause, which works just like the SQL version of the same.
model.find({
'_id': { $in: [
mongoose.Types.ObjectId('4ed3ede8844f0f351100000c'),
mongoose.Types.ObjectId('4ed3f117a844e0471100000d'),
mongoose.Types.ObjectId('4ed3f18132f50c491100000e')
]}
}, function(err, docs){
console.log(docs);
});
This method will work well even for arrays containing tens of thousands of ids. (See Efficiently determine the owner of a record)
I would recommend that anybody working with mongoDB read through the Advanced Queries section of the excellent Official mongoDB Docs
Ids is the array of object ids:
const ids = [
'4ed3ede8844f0f351100000c',
'4ed3f117a844e0471100000d',
'4ed3f18132f50c491100000e',
];
Using Mongoose with callback:
Model.find().where('_id').in(ids).exec((err, records) => {});
Using Mongoose with async function:
const records = await Model.find().where('_id').in(ids).exec();
Or more concise:
const records = await Model.find({ '_id': { $in: ids } });
Don't forget to change Model with your actual model.
Combining Daniel's and snnsnn's answers:
let ids = ['id1', 'id2', 'id3'];
let data = await MyModel.find({
'_id': {
$in: ids
}
});
Simple and clean code. It works and tested against:
"mongodb": "^3.6.0",
"mongoose": "^5.10.0",
Use this format of querying
let arr = _categories.map(ele => new mongoose.Types.ObjectId(ele.id));
Item.find({ vendorId: mongoose.Types.ObjectId(_vendorId) , status:'Active'})
.where('category')
.in(arr)
.exec();
This code works for me just fine as of mongoDB v4.2 and mongoose 5.9.9:
const Ids = ['id1','id2','id3']
const results = await Model.find({ _id: Ids})
and the Ids can be of type ObjectId or String
Both node.js and MongoChef force me to convert to ObjectId. This is what I use to grab a list of users from the DB and fetch a few properties. Mind the type conversion on line 8.
// this will complement the list with userName and userPhotoUrl
// based on userId field in each item
augmentUserInfo = function(list, callback) {
var userIds = [];
var users = []; // shortcut to find them faster afterwards
for (l in list) { // first build the search array
var o = list[l];
if (o.userId) {
userIds.push(new mongoose.Types.ObjectId(o.userId)); // for Mongo query
users[o.userId] = o; // to find the user quickly afterwards
}
}
db.collection("users").find({
_id: {
$in: userIds
}
}).each(function(err, user) {
if (err) {
callback(err, list);
} else {
if (user && user._id) {
users[user._id].userName = user.fName;
users[user._id].userPhotoUrl = user.userPhotoUrl;
} else { // end of list
callback(null, list);
}
}
});
}
if you are using the async-await syntax you can use
const allPerformanceIds = ["id1", "id2", "id3"];
const findPerformances = await Performance.find({
_id: {
$in: allPerformanceIds
}
});
I tried like below and it works for me.
var array_ids = [1, 2, 6, 9]; // your array of ids
model.find({
'_id': {
$in: array_ids
}
}).toArray(function(err, data) {
if (err) {
logger.winston.error(err);
} else {
console.log("data", data);
}
});
I am using this query to find the files in mongo GridFs. I wanted to get the by its Ids.
For me this solution is working: Ids type of ObjectId.
gfs.files
.find({ _id: mongoose.Types.ObjectId('618d1c8176b8df2f99f23ccb') })
.toArray((err, files) => {
if (!files || files.length === 0) {
return res.json('no file exist');
}
return res.json(files);
next();
});
This is not working: Id type of string
gfs.files
.find({ _id: '618d1c8176b8df2f99f23ccb' })
.toArray((err, files) => {
if (!files || files.length === 0) {
return res.json('no file exist');
}
return res.json(files);
next();
});

Resources