Group by date in MongoDB - node.js

I'm running a blog-style web application on AppFog (ex Nodester).
It's written in NodeJS + Express and uses Mongoose framework to persist to MongoDB.
MongoDB is version 1.8 and I don't know whether AppFog is going to upgrade it to 2.2 or not.
Why this intro? Well, now my "posts" are shown in a basic "paginated" visualization, I mean they're just picked up from mongo, sorted by date descending, a page at a time. Here's a snippet:
Post
.find({pubblicato:true})
.populate("commenti")
.sort("-dataInserimento")
.skip(offset)
.limit(archivePageSize)
.exec(function(err,docs) {
var result = {};
result.postsArray = (!err) ? docs : [];
result.currentPage = currentPage;
result.pages = howManyPages;
cb(null, result);
});
Now, my goal is to GROUP BY 'dataInserimento' and show posts like a "diary", I mean:
1st page => 2012/10/08: I show 3 posts
2nd page => 2012/10/10: I show 2 posts (2012/10/09 has no posts, so I don't allow a white page)
3rd page => 2012/10/11: 35 posts and so on...
My idea is to get first the list of all dates with grouping (and maybe counting posts for each day) then build the pages link and, when a page (date) is visited, query like above, adding date as parameter.
SOLUTIONS:
Aggregation framework would be perfect for that, but I can't get my hands on that version of Mongo, now
Using .group() in some way, but the idea it doesn't work in sharded environments does NOT excite me! :-(
writing a MAP-REDUCE! I think this is the right way to go but I can't imagine how map() and reduce() should be written.
Can you help me with a little example, please?
Thanks
EDIT :
The answer of peshkira is correct, however, I don't know if I need exactly that.
I mean, I will have URLs like /archive/2012/10/01, /archive/2012/09/20, and so on.
In each page, it's enough to have the date for querying for posts. But then I have to show "NEXT" or "PREV" links, so I need to know what's the next or previous day containing posts, if any. Maybe can I just query for posts with dates bigger or smaller than the current, and get the first one's date?

Assuming you have something similar as:
{
"author" : "john doe",
"title" : "Post 1",
"article" : "test",
"created" : ISODate("2012-02-17T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 2",
"article" : "foo",
"created" : ISODate("2012-02-17T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 3",
"article" : "bar",
"created" : ISODate("2012-02-18T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 4",
"article" : "foo bar",
"created" : ISODate("2012-02-20T00:00:00Z")
}
{
"author" : "john doe",
"title" : "Post 5",
"article" : "lol cat",
"created" : ISODate("2012-02-20T00:00:00Z")
}
then you can use map reduce as follows:
Map
It just emits the date as key and the post title. You can change the title to the _id, which will probably be more useful to you. If you store the time of the date you will want to use only the date (without time) as the key, otherwise mongo will group by date time and not only date. In my test case all posts have the same time 00:00:00 so it does not matter.
function map() {
emit(this.created, this.title);
}
Reduce
It does nothing more, then just push all values for a key to an array and then the array is wrapped in a result object, because mongo does not allow arrays to be the result of a reduce function.
function reduce(key, values) {
var array = [];
var res = {posts:array};
values.forEach(function (v) {res.posts.push(v);});
return res;
}
Execute
Using db.runCommand({mapreduce: "posts", map: map, reduce: reduce, out: {inline: 1}}) will output the following result:
{
"results" : [
{
"_id" : ISODate("2012-02-17T00:00:00Z"),
"value" : {
"posts" : [
"Post 2",
"Post 1"
]
}
},
{
"_id" : ISODate("2012-02-18T00:00:00Z"),
"value" : "Post 3"
},
{
"_id" : ISODate("2012-02-20T00:00:00Z"),
"value" : {
"posts" : [
"Post 5",
"Post 4"
]
}
}
],
...
}
I hope this helps

Related

Firebase Realtime Database Query: Filtering data based on the UserId signed in and a further unique identifier within the database

I have a query regarding an app I am trying to develop with node js react and Firebase Realtime Database.
The app is for a school and I am trying to write the correct code for filtering the data by course based on the course that the student has signed up for.
On the Firebase realtime database, I have two structure as per below:
- Courses
{
"courseData" : [ {
"course" : {
"day" : "Tuesday",
"duration" : "10 weeks",
"language" : "German",
"location" : "Online",
"startdate" : "12th January",
"term" : "January",
"time" : "17.30-18.30",
"timeofday" : "Evening",
},
"courseID" : "JRNGETNXXOLTUV",
"dates" : {
"class1" : "12/01/2021",
}
}],
"users" : {
"kwvjUSgZKXXfxxxxxxxxxxxxxxxxxx" : {
"courseID" : "JRNGETNXXOLTUV",
"email" : "test#test.com",
"username" : "Test"
},
"vXf4WcRGQcxxxxxxxxxxxxxxxxxxx" : {
"courseID" : "JRNGETNXXOLTUV",
"email" : "test2#test.com",
"username" : "Test Test"
I have a courseID in both courseData and the users section of the Firebase Realtime Database.
At the moment I can generate course data for a specific course when I manually insert the courseID as you will see below in the excerpt below.
Excerpt 1
filtercourse(courseID) {
return function (coursedata) {
return coursedata.courseID === courseID;
};
}
....
Excerpt 2
<tbody>
{this.state.courseData.filter(this.filtercourse('JANSPADBGOLWEE')).map((data, index) => (
<tr key={index}>
...
Instead of manually inserting the courseID (in this case it's JANSPADBGOLWEE), I understand that I need to create a function where the courseData data is filtered by course/ courseID based on the courseData.courseID being equal to the users.uid.courseID, however, I this is beyond me it seems. Any help or advice here would be greatly appreciated.
I think you're looking for a Firebase query, which allow you to sort and filter data.
On Courses, you could get only its child nodes where courseID has a specific value with:
let courses = firebase.database().ref().child("courseData");
let courseSuery = courses.orderByChild("courseID").equalTo("JANSPADBGOLWEE");
courseSuery.once("value").then((snapshot) => {
snapshot.forEach((courseSnapshot) => {
console.log(courseSnapshot.key, courseSnapshot.child("course/day").val());
});
});
If you first need to look up the course ID for the current user, that'd be:
let users = firebase.database().ref().child("users");
if (!firebase.auth().currentUser) throw "No current user";
let uid = firebase.auth().currentUser.uid;
users.child(uid).once("value").then((userSnapshot) => {
console.log(userSnapshot.val().courseID);
});
Note that this is a fairly standard way of loading data from Firebase, so I recommend reading some more of the documentation, and taking a few tutorials to get self-sufficient with it.

paginating using angularfire2

suppose I have a data structure in firebase real time database like
{ "donors" :
"uid1" : { "name" : "x", "bloodGroup" : "A+", "location" : "some Place"},
"uid2" : { "name" : "y", "bloodGroup" : "A-", "location" : "some place"},
...
...
}
now if I have millions of donor records like this. how could I filter them based on bloodGroup location and fetching say 100 records from server at a time using angularfire2.
I have found this page which was really helpful to me when using queries to query my firebase data:
https://howtofirebase.com/collection-queries-with-firebase-b95a0193745d
A very simple example would be along the lines of:
this.donorsData = af.database.list('/donors', {
query: {
orderByChild: 'bloodGroup',
equalTo: 'A+',
}
});
Not entirely sure how to fetch 100 records, then another 100, I am using datatables in my app, which fetches all my data and using the datatables for pagination.

Node.js and MongoDB Use results from one query in another

I have created a node.js module that can already query MongoDB for a set of documents using a find and output those results to JSON. My question is, knowing that node.js is asynchronous, how can I use the results from this query (items) to create a query that goes back to MongoDB to find another set of documents. This query basically returns a list of employee ids that can be used to query documents containing information on those employees(i.e. firstName, lastName etc.). Then output those results instead as JSON. The first query is basically saying, give me all of the employees that can be viewed by a particular user. I then need to take the employee ids and do a query on another set of documents that contains those individuals information, like you see below.
Here are the two documents schema:
Employee
{
"_id" : ObjectId("5208db78ecc00915e0900699"),
"clientId" : 1,
"employeeId" : "12345",
"lastName" : "DOE",
"firstName" : "JOHN",
"middleName" : "A",
"badge" : "8675309",
"birthDate" : "10/12/1978"
}
Users an employee can access (User Cache)
{
"_id" : ObjectId("520920a99bc417b7c5e36abf"),
"clientSystem" : "SystemX",
"customerNumber" : "1",
"clientUserId" : "jdoe3",
"securityCode" : "authorize",
"employeeId" : "12345",
"creationDate" : "2013-Aug-12 13:51:37"
}
Here is my code:
exports.employeeList = function(req, res) {
console.log(req.params);
var clientSystem = req.query["clientSystem"];
var clientUserId = req.query["clientUserId"];
var customerNumber = req.query["customerNumber"];
var securityCode = req.query["securityCode"];
if (clientSystem != null && clientUserId != null && customerNumber != null && securityCode != null){
db.collection('ExtEmployeeList', function(err, collection){
collection.find({'clientSystem': clientSystem, 'clientUserId':clientUserId, 'customerNumber':customerNumber, 'securityCode': securityCode}).toArray(function (err, items){
console.log(items);
res.jsonp(items);
});//close find
});//close collection
}//close if
else {
res.send(400);
}//close else
};//close function
What you're wanting to do is possible, but probably not the most effective use of Mongo. I tend to design Mongo documents around how the data will actually be used. So if I needed the user's names to show up in a list of users I can view, I would embed that data so I don't have to do multiple round trips to mongo to get all the information I need. I would do something like the following:
{
"_id" : ObjectId("520920a99bc417b7c5e36abf"),
"clientSystem" : "SystemX",
"customerNumber" : "1",
"clientUserId" : "jdoe3",
"securityCode" : "authorize",
"employeeId" : "12345",
"creationDate" : "2013-Aug-12 13:51:37"
"employee": {
"_id" : ObjectId("5208db78ecc00915e0900699"),
"clientId" : 1,
"employeeId" : "12345",
"lastName" : "DOE",
"firstName" : "JOHN",
"middleName" : "A",
"badge" : "8675309",
"birthDate" : "10/12/1978"
}
}
Yes, you are duplicating data but you're dramatically reducing the number of round trips to the database. This is typically the tradeoff you make when using document based databases since you can't join tables.

Best way to do one-to-many "JOIN" in CouchDB

I am looking for a CouchDB equivalent to "SQL joins".
In my example there are CouchDB documents that are list elements:
{ "type" : "el", "id" : "1", "content" : "first" }
{ "type" : "el", "id" : "2", "content" : "second" }
{ "type" : "el", "id" : "3", "content" : "third" }
There is one document that defines the list:
{ "type" : "list", "elements" : ["2","1"] , "id" : "abc123" }
As you can see the third element was deleted, it is no longer part of the list. So it must not be part of the result. Now I want a view that returns the content elements including the right order.
The result could be:
{ "content" : ["second", "first"] }
In this case the order of the elements is already as it should be. Another possible result:
{ "content" : [{"content" : "first", "order" : 2},{"content" : "second", "order" : 1}] }
I started writing the map function:
map = function (doc) {
if (doc.type === 'el') {
emit(doc.id, {"content" : doc.content}); //emit the id and the content
exit;
}
if (doc.type === 'list') {
for ( var i=0, l=doc.elements.length; i<l; ++i ){
emit(doc.elements[i], { "order" : i }); //emit the id and the order
}
}
}
This is as far as I can get. Can you correct my mistakes and write a reduce function? Remember that the third document must not be part of the result.
Of course you can write a different map function also. But the structure of the documents (one definig element document and an entry document for each entry) cannot be changed.
EDIT: Do not miss JasonSmith's comment to his answer, where he describes how to do this shorter.
Thank you! This is a great example to show off CouchDB 0.11's new
features!
You must use the fetch-related-data feature to reference documents
in the view. Optionally, for more convenient JSON, use a _list function to
clean up the results. See Couchio's writeup on "JOIN"s for details.
Here is the plan:
Firstly, you have a uniqueness contstraint on your el documents. If two of
them have id=2, that's a problem. It is necessary to use
the _id field instead if id. CouchDB will guarantee uniqueness, but also,
the rest of this plan requires _id in order to fetch documents by ID.
{ "type" : "el", "_id" : "1", "content" : "first" }
{ "type" : "el", "_id" : "2", "content" : "second" }
{ "type" : "el", "_id" : "3", "content" : "third" }
If changing the documents to use _id is absolutely impossible, you can
create a simple view to emit(doc.id, doc) and then re-insert that into a
temporary database. This converts id to _id but adds some complexity.
The view emits {"_id": content_id} data keyed on
[list_id, sort_number], to "clump" the lists with their content.
function(doc) {
if(doc.type == 'list') {
for (var i in doc.elements) {
// Link to the el document's id.
var id = doc.elements[i];
emit([doc.id, i], {'_id': id});
}
}
}
Now there is a simple list of el documents, in the correct order. You can
use startkey and endkey if you want to see only a particular list.
curl localhost:5984/x/_design/myapp/_view/els
{"total_rows":2,"offset":0,"rows":[
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","0"],"value":{"_id":"2"}},
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","1"],"value":{"_id":"1"}}
]}
To get the el content, query with include_docs=true. Through the magic of
_id, the el documents will load.
curl localhost:5984/x/_design/myapp/_view/els?include_docs=true
{"total_rows":2,"offset":0,"rows":[
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","0"],"value":{"_id":"2"},"doc":{"_id":"2","_rev":"1-4530dc6946d78f1e97f56568de5a85d9","type":"el","content":"second"}},
{"id":"036f3614aeee05344cdfb66fa1002db6","key":["abc123","1"],"value":{"_id":"1"},"doc":{"_id":"1","_rev":"1-852badd683f22ad4705ed9fcdea5b814","type":"el","content":"first"}}
]}
Notice, this is already all the information you need. If your client is
flexible, you can parse the information out of this JSON. The next optional
step simply reformats it to match what you need.
Use a _list function, which simply reformats the view output. People use them to output XML or HTML however we will make
the JSON more convenient.
function(head, req) {
var headers = {'Content-Type': 'application/json'};
var result;
if(req.query.include_docs != 'true') {
start({'code': 400, headers: headers});
result = {'error': 'I require include_docs=true'};
} else {
start({'headers': headers});
result = {'content': []};
while(row = getRow()) {
result.content.push(row.doc.content);
}
}
send(JSON.stringify(result));
}
The results match. Of course in production you will need startkey and endkey to specify the list you want.
curl -g 'localhost:5984/x/_design/myapp/_list/pretty/els?include_docs=true&startkey=["abc123",""]&endkey=["abc123",{}]'
{"content":["second","first"]}

How can I get a view of favorite user documents by user in Couchdb map/reduce?

My Couchdb database as a main document type that looks something like:
{
"_id" : "doc1",
"type" : "main_doc",
"title" : "the first doc"
...
}
There is another type of document that stores user information. I want users to be able to tag documents as favorites. Different users can save the same or different documents as favorites. My idea was to introduce a favorite document to track this something like:
{
"_id" : "fav1",
"type" : "favorite",
"user_id" : "user1",
"doc_id" : "doc1"
}
It's easy enough to create a view with user_id as the key to get a list of their favorite doc IDs. E.g:
function(doc) {
if (doc.type == "favorite") {
emit(doc.user_id, doc.doc_id);
}
}
However I want to list of favorites to display the user_id, doc_id and title from the document. So output something like:
{ "key" : "user1", "value" : ["doc1", "the first doc"] }
In CouchDB 0.11 (just recently released), the include_docs=true feature allows you to look up any document in your view row. For example:
function(doc) {
if(doc.type == "favorite") {
emit(doc.user_id, {_id: doc.doc_id});
}
}
When you query your view with include_docs=true, you should see JSON like this:
// ... normal stuff
rows: [
{
"key":"user1",
"value":{"_id":"doc1"},
"doc": {
"_id" : "doc1",
"type" : "main_doc",
"title" : "the first doc"
// ...
}
},
{
// another doc, etc...
}
]
If you can't use the include_docs=true feature with v0.11, then you must have all information on-hand when you emit data for your view/map.
Instead of a traditional "join" style, consider storing a list of "favoriting" users in the main_doc documents.
{
"_id" : "doc1",
"type" : "main_doc",
"title" : "the first doc",
"favorited_by": ["user1", "user2"]
// ...
}
That way when your view runs, you can emit everything based on the information in that one document.
function(doc) {
if(doc.type == "main_doc") {
for (var a in doc.favorited_by) {
emit(doc.favorited_by[a], [doc._id, doc.title]);
}
}
}

Resources