Node.js and MongoDB Use results from one query in another - node.js

I have created a node.js module that can already query MongoDB for a set of documents using a find and output those results to JSON. My question is, knowing that node.js is asynchronous, how can I use the results from this query (items) to create a query that goes back to MongoDB to find another set of documents. This query basically returns a list of employee ids that can be used to query documents containing information on those employees(i.e. firstName, lastName etc.). Then output those results instead as JSON. The first query is basically saying, give me all of the employees that can be viewed by a particular user. I then need to take the employee ids and do a query on another set of documents that contains those individuals information, like you see below.
Here are the two documents schema:
Employee
{
"_id" : ObjectId("5208db78ecc00915e0900699"),
"clientId" : 1,
"employeeId" : "12345",
"lastName" : "DOE",
"firstName" : "JOHN",
"middleName" : "A",
"badge" : "8675309",
"birthDate" : "10/12/1978"
}
Users an employee can access (User Cache)
{
"_id" : ObjectId("520920a99bc417b7c5e36abf"),
"clientSystem" : "SystemX",
"customerNumber" : "1",
"clientUserId" : "jdoe3",
"securityCode" : "authorize",
"employeeId" : "12345",
"creationDate" : "2013-Aug-12 13:51:37"
}
Here is my code:
exports.employeeList = function(req, res) {
console.log(req.params);
var clientSystem = req.query["clientSystem"];
var clientUserId = req.query["clientUserId"];
var customerNumber = req.query["customerNumber"];
var securityCode = req.query["securityCode"];
if (clientSystem != null && clientUserId != null && customerNumber != null && securityCode != null){
db.collection('ExtEmployeeList', function(err, collection){
collection.find({'clientSystem': clientSystem, 'clientUserId':clientUserId, 'customerNumber':customerNumber, 'securityCode': securityCode}).toArray(function (err, items){
console.log(items);
res.jsonp(items);
});//close find
});//close collection
}//close if
else {
res.send(400);
}//close else
};//close function

What you're wanting to do is possible, but probably not the most effective use of Mongo. I tend to design Mongo documents around how the data will actually be used. So if I needed the user's names to show up in a list of users I can view, I would embed that data so I don't have to do multiple round trips to mongo to get all the information I need. I would do something like the following:
{
"_id" : ObjectId("520920a99bc417b7c5e36abf"),
"clientSystem" : "SystemX",
"customerNumber" : "1",
"clientUserId" : "jdoe3",
"securityCode" : "authorize",
"employeeId" : "12345",
"creationDate" : "2013-Aug-12 13:51:37"
"employee": {
"_id" : ObjectId("5208db78ecc00915e0900699"),
"clientId" : 1,
"employeeId" : "12345",
"lastName" : "DOE",
"firstName" : "JOHN",
"middleName" : "A",
"badge" : "8675309",
"birthDate" : "10/12/1978"
}
}
Yes, you are duplicating data but you're dramatically reducing the number of round trips to the database. This is typically the tradeoff you make when using document based databases since you can't join tables.

Related

Firebase Realtime Database Query: Filtering data based on the UserId signed in and a further unique identifier within the database

I have a query regarding an app I am trying to develop with node js react and Firebase Realtime Database.
The app is for a school and I am trying to write the correct code for filtering the data by course based on the course that the student has signed up for.
On the Firebase realtime database, I have two structure as per below:
- Courses
{
"courseData" : [ {
"course" : {
"day" : "Tuesday",
"duration" : "10 weeks",
"language" : "German",
"location" : "Online",
"startdate" : "12th January",
"term" : "January",
"time" : "17.30-18.30",
"timeofday" : "Evening",
},
"courseID" : "JRNGETNXXOLTUV",
"dates" : {
"class1" : "12/01/2021",
}
}],
"users" : {
"kwvjUSgZKXXfxxxxxxxxxxxxxxxxxx" : {
"courseID" : "JRNGETNXXOLTUV",
"email" : "test#test.com",
"username" : "Test"
},
"vXf4WcRGQcxxxxxxxxxxxxxxxxxxx" : {
"courseID" : "JRNGETNXXOLTUV",
"email" : "test2#test.com",
"username" : "Test Test"
I have a courseID in both courseData and the users section of the Firebase Realtime Database.
At the moment I can generate course data for a specific course when I manually insert the courseID as you will see below in the excerpt below.
Excerpt 1
filtercourse(courseID) {
return function (coursedata) {
return coursedata.courseID === courseID;
};
}
....
Excerpt 2
<tbody>
{this.state.courseData.filter(this.filtercourse('JANSPADBGOLWEE')).map((data, index) => (
<tr key={index}>
...
Instead of manually inserting the courseID (in this case it's JANSPADBGOLWEE), I understand that I need to create a function where the courseData data is filtered by course/ courseID based on the courseData.courseID being equal to the users.uid.courseID, however, I this is beyond me it seems. Any help or advice here would be greatly appreciated.
I think you're looking for a Firebase query, which allow you to sort and filter data.
On Courses, you could get only its child nodes where courseID has a specific value with:
let courses = firebase.database().ref().child("courseData");
let courseSuery = courses.orderByChild("courseID").equalTo("JANSPADBGOLWEE");
courseSuery.once("value").then((snapshot) => {
snapshot.forEach((courseSnapshot) => {
console.log(courseSnapshot.key, courseSnapshot.child("course/day").val());
});
});
If you first need to look up the course ID for the current user, that'd be:
let users = firebase.database().ref().child("users");
if (!firebase.auth().currentUser) throw "No current user";
let uid = firebase.auth().currentUser.uid;
users.child(uid).once("value").then((userSnapshot) => {
console.log(userSnapshot.val().courseID);
});
Note that this is a fairly standard way of loading data from Firebase, so I recommend reading some more of the documentation, and taking a few tutorials to get self-sufficient with it.

Complex search and extendable query on two and more collections

I am working on Node.js + mongodb application and my question related to design of my database and queries. Firstly, describe my db structure. I have in my database two collections users and values. Document in users collection looks like this:
{
"_id" : 31450861,
"first_name" : "Jon",
"last_name" : "Doe",
"sex" : 2,
"bdate" : ISODate("1981-08-01T21:00:00Z"),
"city" : {
"id" : 282,
"title" : "Minsk"
},
"country" : {
"id" : 3,
"title" : "Belarussia"
},
"photo_max" : "https://foto.ru/RnhOKp2YJE4.jpg",
"relation" : 4,
"interests" : "science",
"music" : "pop",
"lang" : "RU",
}
This collection filled with users data (language, birthday, etc).
Documents from collection values looks like this:
{
"_id" : ObjectId("548023e5c16f310c23b75075"),
"userId" : "31450861", //reference to _id in users collection
"group1" : {
"value1" : 13,
" value2" : 7,
" value3" : 3,
" value4" : 14,
" value5" : 17
},
"group2" : {
" value1" : 9,
" value2" : 6,
" value3" : 17,
" value4" : 12,
" value5" : 13
}
So I need to make search for users on this database with lots of parameters from this two collection (their values (from collection values), sex, city, language etc.). I didn’t embed document values into users because I do a lot queries separately on them (but may be it’s anyway wrong design I need help on this). In future there will be more collections with similar structure like values (or at least there will be reference to userId), which I’ll have to include in search query, and I’ll need agility to extend my query on more collections.
So I need to run complex query on this collections (I know there is no JOINs in mongodb, so I know I have to query twice or use mapreduce).
So far my thoughts on this issue (aren’t tested in code just thoughts).
I need to write search function which performs 2 queries (and more in future):
Find users with same values and getting their ids.
var values = db.collection('values');
var ids = values.find({ value1: 1, value2: 2, value3: 3 }, {userId: 1 } ) //then transform ids so it became array with userId
Then in this found set find users on some more parameters (sex, birthday, language, etc)
var users = db.collection('users');
users.find({ $and: [{ _id: { $in: ids } }, {sex: 2 }, {lang: “RU” } ] });
My questions are:
1. Is it normal approach or I’ll end up with very slow performance and mess in code when adding new collections and queries?
2. If is it normal, how to easily add one more query to one more collection?
Any help, any thoughts are welcome! Thanks in advance!

Learning Map-Reduce in MongoDB

While searching through the internet, I found out that joins can be emulated in mongodb through the map-reduce function. Going through the docs was confusing.
I have two collections: one with a list of friends of one user. And the other collection is of all the users. I want to fetch the profile pictures of all the friends. how do I create a mongodb query to get the desired results?
The USERS collection:
{
"_id" : ObjectId("524c194a6e3715ce0a000001"),
"email" : "qwerty#abc.com",
"password" : "",
"phone" : "",
"salt" : "",
"upic" : "someuser2fd2751259bb7519d7b760ffee9b7fce203ad1f34.jpg",
"username" : "someuser2"
}
{
"_id" : ObjectId("524be475fafb35480a000001"),
"email" : "",
"password" : "",
"phone" : "",
"salt" : "",
"upic" : "amitverma2522b7a52e054c350f78fd7f3558919f2e2dab58.jpg",
"username" : "amitverma"
}
The friends of each user collection:
{
"_id" : ObjectId("526547ed2389630000000001"),
"friends" : [
{
"_id" : ObjectId("524be475fafb35480a000001"),
"username" : "amitverma"
},
{
"_id" : ObjectId("524be475fafb35480a000001"),
"username" : "someuser2"
}
],
"upic" : "macbookfd2751259bb7519d7b760ffee9b7fce203ad1f34.jpg",
"username" : "someuser"
}
Help would be appreciated.
There are no official docs for this as it's not a recommended best practice. It's complex that you need to do multiple passes carefully outputting the same results into the same collection.
You'd be better served by gathering the list of friends and using the $in operator (reference) to fetch the users and projecting the results to only include the fields you require (like the image).
Ideally, you'd cache those results locally to avoid needlessly requesting image paths. Following is untested code that should work in the shell:
db.friends.find({ username: 'someuser'}).forEach(friend_list) {
// this would gather the list of friend's _ids
// the _id will be passed as an array for the $in operator
var friends = friend_list.friends.map(function(friend) { return this._id; });
// gather up the images for each of the friends
var upics = db.users.find({_id : { $in : friends }},
{ _id: 1, upic: 1 }).toArray();
// now, do something with upics -- outside of the MongoDB shell, this will
// return asynchronously ....
});

selecting an nested attribute to update in mongo with node driver

In the following code using the Node Js Driver for MongoDB, the console log in the callback would log the number of drivers for a particular vehicle. My problem is with trying to increment the number of drivers by one with this code
$inc:{vehicles[vehicle_number].drivers:1}
It's giving me unexpected token errors with the [ and also I'm not even sure if by starting the selector with vehicles it would be acting on the family that's been queried. Can you explain how I might change the code to make the increment not break the function.
families.findAndModify({_id:family_id}, [],
{$inc:{vehicles[vehicle_number].drivers:1}} , function(err, doc) {
console.log(doc.vehicles[vehicle_number].drivers)
})
Update
the vehicles key contains an array of vehicles. In the code above, vehicle_number represents the index in the array. I'm trying to increment the number of drivers in the findAndModify above.
{
"_id" : ObjectId("5143ddf5bcf1bf4ab37da054"),
"name" : "Jones",
"vehicles" : [
{
"make" : "Ford",
"year" : "2001",
"color" : "blue",
"registration" : "xdklde",
"drivers" : 3
},
{
"make" : "Dodge",
"year" : "1992",
"color" : "green",
"registration" : "klrv7z",
"drivers" : 2
},
var selector = {};
selector['vehicles.' + vehicle_number + '.drivers'] = 1;
then you use it in your query:
families.update({'_id':family_id}, {'$inc':selector} , function(err, doc) {
console.log(doc);
});
This should work.

Group by date in MongoDB

I'm running a blog-style web application on AppFog (ex Nodester).
It's written in NodeJS + Express and uses Mongoose framework to persist to MongoDB.
MongoDB is version 1.8 and I don't know whether AppFog is going to upgrade it to 2.2 or not.
Why this intro? Well, now my "posts" are shown in a basic "paginated" visualization, I mean they're just picked up from mongo, sorted by date descending, a page at a time. Here's a snippet:
Post
.find({pubblicato:true})
.populate("commenti")
.sort("-dataInserimento")
.skip(offset)
.limit(archivePageSize)
.exec(function(err,docs) {
var result = {};
result.postsArray = (!err) ? docs : [];
result.currentPage = currentPage;
result.pages = howManyPages;
cb(null, result);
});
Now, my goal is to GROUP BY 'dataInserimento' and show posts like a "diary", I mean:
1st page => 2012/10/08: I show 3 posts
2nd page => 2012/10/10: I show 2 posts (2012/10/09 has no posts, so I don't allow a white page)
3rd page => 2012/10/11: 35 posts and so on...
My idea is to get first the list of all dates with grouping (and maybe counting posts for each day) then build the pages link and, when a page (date) is visited, query like above, adding date as parameter.
SOLUTIONS:
Aggregation framework would be perfect for that, but I can't get my hands on that version of Mongo, now
Using .group() in some way, but the idea it doesn't work in sharded environments does NOT excite me! :-(
writing a MAP-REDUCE! I think this is the right way to go but I can't imagine how map() and reduce() should be written.
Can you help me with a little example, please?
Thanks
EDIT :
The answer of peshkira is correct, however, I don't know if I need exactly that.
I mean, I will have URLs like /archive/2012/10/01, /archive/2012/09/20, and so on.
In each page, it's enough to have the date for querying for posts. But then I have to show "NEXT" or "PREV" links, so I need to know what's the next or previous day containing posts, if any. Maybe can I just query for posts with dates bigger or smaller than the current, and get the first one's date?
Assuming you have something similar as:
{
"author" : "john doe",
"title" : "Post 1",
"article" : "test",
"created" : ISODate("2012-02-17T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 2",
"article" : "foo",
"created" : ISODate("2012-02-17T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 3",
"article" : "bar",
"created" : ISODate("2012-02-18T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 4",
"article" : "foo bar",
"created" : ISODate("2012-02-20T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 5",
"article" : "lol cat",
"created" : ISODate("2012-02-20T00:00:00Z")
}
then you can use map reduce as follows:
Map
It just emits the date as key and the post title. You can change the title to the _id, which will probably be more useful to you. If you store the time of the date you will want to use only the date (without time) as the key, otherwise mongo will group by date time and not only date. In my test case all posts have the same time 00:00:00 so it does not matter.
function map() {
emit(this.created, this.title);
}
Reduce
It does nothing more, then just push all values for a key to an array and then the array is wrapped in a result object, because mongo does not allow arrays to be the result of a reduce function.
function reduce(key, values) {
var array = [];
var res = {posts:array};
values.forEach(function (v) {res.posts.push(v);});
return res;
}
Execute
Using db.runCommand({mapreduce: "posts", map: map, reduce: reduce, out: {inline: 1}}) will output the following result:
{
"results" : [
{
"_id" : ISODate("2012-02-17T00:00:00Z"),
"value" : {
"posts" : [
"Post 2",
"Post 1"
]
}
},
{
"_id" : ISODate("2012-02-18T00:00:00Z"),
"value" : "Post 3"
},
{
"_id" : ISODate("2012-02-20T00:00:00Z"),
"value" : {
"posts" : [
"Post 5",
"Post 4"
]
}
}
],
...
}
I hope this helps

Resources