I've been working on a news site where users can post articles and leave comments. Part of the comments is to designate an emotion to the comment (one of 5 options: Amazed, Happy, Neutral, Sad, Angry).
What i want to do with this is make a 'top 5' of articles with the highest emotion count for a specific emotion. And sure, i know on a forum the emotion is always 'angry' so i should just work with that ;D , but this is part of a learning experience and i'd like to know how to do this.
So for other top5's ( 'hot discussion' and 'hot news' ) i have introduced counters in my Article Schema, that get incremented either when a comment is made (for 'hotDiscussion'), or when a user designates something as truly 'hot news' ( 'like button functionality' ). Then i simply search for the Articles and sort by these counters...
BUT, for this specific case, where there are 5 options, how do i sort ?!? ... at first i thought i would just implement the counters when a certain emotion is picked (pull down menu). But then all the counters are 'on the Article' as a single entity and i don't see how you would sort by all of them to get a comprehensive list. So then i though to put them in an object or array... something like:
Article {
title: String,
author: user...
comments: []
emotions: [ or {
Amazed: amazedCount,
Happy: happyCount,
Neutral: neutralCount,
Sad: sadCount,
Angry: angryCount
} or ]
Then, maybe i can find all the articles and sort by emotions (somehow?) and then skim off the top 5 from the list. But here again i run into the 5 options thing. And i have no clue how to sort the emotions? to then perhaps within the results, sort by the first (highest) count. To then skim of the top5.
So looking around here somewhat hints at me going for something like this
Article {
...
...
emotions: [
{name: Amazed, value: amazedCount},
{name: Happy, value: happyCount},
{name: Neutral, value: neutralCount},
{name: Sad, value: sadCount},
{name: Angry, value: angryCount}
]
And i like this direction somewhat, due to the clear keys 'name' & 'value'.
But here i run into the same issue, aka. lack of knowledge ... Because:
how to sort on a second order value, as i would be sorting on value?
will the same article just show up multiple times in the result? (i'dd like to prevent that if at all possible, a very busy mixed emotion thread should not show up twice over a less busy but clearly 'happy' thread in the top5 ).
So hopefully somebody can help me, as i can see this type of searching valuable in various types of situations, not just for some learning environment type 'forum' top5.
For those needing a more useful 'use case'
warehouse{
name: String,
... : ...
parts: [
{name: part1, value: Count}
{name: part2, value: Count}
{etc }
]
}
Generating a list of warehouses based upon the count of parts (where double mention of a certain warehouse isn't a problem); Or say a list of Car Dealers with the highest count for available brands or models (to get input for redistribution f/e). Or a list of online shops with a multi value rating system (f/e: delivery speed, customer service, price of products, etc.)
Well, it turns out this is actually fairly simple ... as i was most the way there. Just needed to put the sort criteria in between '...' /doh !
! THIS CODE IS NOT COMPLETE, as in Copy/Paste you are done, but the Code below does show you most the places where you need to 'do' things to make this work (/show routes not included, aka the pages the user sees and can input stuff on)
// In the Schemas:
Article {
...
...
emotions: [ {emotion: String, eCount: Number} ]
}
Comment{
emotion: String; //to save the emotion of the user on the comment
}
// Upon initializing an Article:
router.post("/route", function(req, res){
let etcetera: etc,
let theEmotions = [ // in your usecase you may populate these elsewhere
{ emotion: "Amazed", eCount: 0 },
{ emotion: "Happy", eCount: 0 },
{ emotion: "Neutral", eCount: 0 },
{ emotion: "Sad", eCount: 0 },
{ emotion: "Angry", eCount: 0 }
];
let newArticle = {
etc: etc,
emotions: theEmotions
};
Article.create(newArticle, function(err, newArticle){
*stuff here*
});
});
// when a comment gets made, it updates the scores based upon the emotion selected
//REST :: CREATE --> actually creates a new comment
router.post("/", function(req, res){
Article.findById(req.params.id, function(err, foundArticle){
if(err){
console.log("ERROR # finding article to POST the COMMENT");
res.redirect('/articles');
} else {
Comment.create(req.body.comment, function(err, createdComment){
if(err){
console.log("ERROR # CREATing new COMMENT");
} else {
switch(createdComment.emotion){
case "Amazed":
foundArticle.emotions[0].eCount += 1;
break;
case "Happy":
foundArticle.emotions[1].eCount += 1;
break;
case "Neutral":
foundArticle.emotions[2].eCount += 1;
break;
case "Sad":
foundArticle.emotions[3].eCount += 1;
break;
case "Angry":
foundArticle.emotions[4].eCount += 1;
break;
default:
console.log("ERROR # Emotion Switch in comments route");
}
//add username and id to comment
createdComment.user.id = req.user._id;
createdComment.user.username = req.user.username;
createdComment.save();
foundArticle.comments.push(createdComment);
foundArticle.save();
console.log("SUCCESS # CREATing new COMMENT");
res.redirect('/articles/' + foundArticle._id);
}
});
}
});
});
//Where one wants to sort: ... note the '...' around 'emotions.eCount'
router.get("/specificRoute, function(req, res){
Article.find({}).sort({'emotions.eCount': -1}).exec(function(err, hotEmotions){
*stuff happens here*
});
});
Hopefully this will be of some help to someone somewhere in the future :)
Related
I have two collections in MongoDB; users & challenges.
The structure of the users collection looks like this:
name: "John Doe"
email: "john#doe.com"
progress: [
{
_id : ObjectId("610be25ae20ce4872b814b24")
challenge: ObjectId("60f9629edd16a8943d2cab9b")
completed: true
date_completed: 2021-08-06T12:15:32.129+00:00
}
{
_id : ObjectId("611be24ae32ce4772b814b32")
challenge: ObjectId("60g6723efd44a6941l2cab81")
completed: true
date_completed: 2021-08-07T12:15:32.129+00:00
}
]
date: 2021-08-05T13:06:34.129+00:00
The structure of the challenges collection looks like this:
_id: ObjectId("610be25ae20ce4872b814b24")
section_no: 1
section_name: "Print Statements"
challenge_no: 1
challenge_name: "Hello World!"
default_code: "public class Main {public static void main(String[] args) {}}"
solution: "Hello World!"
What I want to be able to do is find the most recent entry in a particular user's 'progress' array within the users collection and based on that result I want to query the challenges collection to find the next challenge for that user.
So say the most recent challenge entry in that user's 'progress' array is...
{
_id : ObjectId("611be24ae32ce4772b814b32")
challenge: ObjectId("60g6723efd44a6941l2cab81")
completed: true
date_completed: 2021-08-07T12:15:32.129+00:00
}
...which is Section 1 Challenge 2. I want to be able to query the challenges collection to return Section 1 Challenge 3, and if that doesn't exist then return Section 2 Challenge 1.
Apologies if this is worded poorly, I am fairly new to MongoDb and unsure of how to create complex queries in it.
Thanks in advance!
One approach:
[
{ // Unwind all arrays
"$unwind":"$progress"
},
{ // Sort in descending order all documents
"$sort":{
"progress.date_completed":-1
}
},
{ // Group them together again but pick only the most recent array element
"$group":{
"_id":"$_id",
"latestProgress":{
"$first":"$progress"
}
}
},
{ // Join with other collection
"$lookup":{
"from":"challenges",
"localField":"latestProgress.challenge",
"foreignField":"challenge",
"as":"Progress"
}
},
{ // Only pick the first array element (since there will be just one)
"$set":{
"Progress":{
"$first":"$Progress"
}
}
}
]
I have provided a comment for each stage so that it would be easier to understand the idea. I'm not confident it's the best approach but it does work since I have tested.
Just a note that there could be a case where Progress field is missing. In that case there is no such challenge document.
i have Elastic Search Index with around 1 million records.
I want to do multi prefix search against 2 fields in the Elastic Search Index, Name and ID (there are around 10 total).
Does creating EdgeNGram autocomplete filter make sense at all?
Or i am missing the point of the EdgeNGram.
Here is the code i have for creation of the index:
client.indices.create({
index: 'testing',
// type: 'text',
body: {
settings: {
analysis: {
filter: {
autocomplete_filter: {
type: 'edge_ngram',
min_gram: 3,
max_gram: 20
}
},
analyzer: {
autocomplete: {
type: 'custom',
tokenizer: 'standard',
filter: [
'lowercase',
'autocomplete_filter'
]
}
}
}
}
}
},function(err,resp,status) {
if(err) {
console.log(err);
}
else {
console.log("create",resp);
}
});
Code for searching
client.search({
index: 'testing',
type: 'article',
body: {
query: {
multi_match : {
query: "87041",
fields: [ "name", "id" ],
type: "phrase_prefix"
}
}
}
},function (error, response,status) {
if (error){
console.log("search error: "+error)
}
else {
console.log("--- Response ---");
console.log(response);
console.log("--- Hits ---");
response.hits.hits.forEach(function(hit){
console.log(hit);
})
}
});
The search returns the correct results, so my question being does creating the edgengram filter and analyzer make sense in this case?
Or this prefix functionality would be given out of the box?
Thanks a lot for your info
It is depending on your use case. Let me explain.
You can use ngram for this feature. Let's say your data is london bridge, then if your min gram is 1 and max gram is 20, it will be tokenized as l, lo, lon, etc..
Here the advantage is that even if you search for bridge or any tokens which is part of the generated ngrams, it will be matched.
There is one out of box feature completion suggester. It uses FST model to store them. Even the documentation says it is faster to search but costlier to build. But the think is it is prefix suggester. Meaning searching bridge will not bring london bridge by default. But there are ways to make this work. Workaround to achieve is that, to have array of tokens. Here london bridge and bridge are the tokens.
There is one more called context suggester. If you know that you are going to search on name or id, it is best over completion suggester. As completion suggester works over on all the index, context suggester works on a particular index based on the context.
As you say, it is prefix search you can go for completion. And you mentioned that there 10 such fields. And if you know the field to be suggested at fore front, then you can go for context suggester.
one nice answer about edge ngram and completion
completion suggester for middle of the words - I used this solution, it works like charm.
You can refer documentation for other default options available within suggesters.
I initialize my DB in the usual way:
mongoose.connect(`mongodb://uname:pword#127.0.0.1:port/dbname?authSource=admin`, {useNewUrlParser: true, autoIndex: false});
And I have a Schema, something like:
var materialSchema = new Schema({
bookID: {type: String, required: true},
active: Boolean,
name: {type: String, required: true},
stockLength: {type: Number, required: true}
});
module.exports = mongoose.model('material', materialSchema);
When I create a new material and add it to the database, it is automatically assigned the usual _id - which is a behaviour I want to maintain. BUT, I'd also like for bookID to be a unique, auto-incrementing index. This is for physical shelf storage, and not for queries or anything like that.
I'd like for bookID to increment in the following way:
A-001
A-002
A-003
...
A-098
A-099
A-100
B-001
...
B-100
...
Z-001
...
Z-100
In case the pattern above isn't clear, the pattern starts at A-001 and ultimately ends at Z-100. Each letter goes from 001 through 100 before moving to the next letter. Each new collection entry is just the next ID in the pattern. It is unlikely that the end will ever be reached, but we'll cross that bridge when we get there.
I've only ever used the default _id for indexing, and can't figure out how to make this pattern.
Thanks for any insight!
Edit #1
The best solution I've come up with so far is to have a separate .txt file with all of the IDs listed in order. As each new object is created, pop (... shift) the next ID off the top of the file. This might also have the added benefit of easily adding additional IDs at a later date. This will probably be the approach I take, but I'm still interested in the mongoose solution requested above.
Edit #2
So I think the solution I'm going to use is a little different. Basically, findOne sorted by bookID descending. Then use the value returned to set the next.
Material.findOne()
.sort({bookID : -1})
.exec((err, mat) => {
if(err) {
// Send error
}else if(!mat) {
// First bookID
}else {
// Indexes exist...
let nextId = getNextID(mat.bookID);
// ...
}
});
Still easy to modify getNextID() to add new/different IDs in the future (if/when "Z100" is reached)
Thanks again!
Ok, so to expand a little bit on Edit #2, I've come up with the following solution.
Within the model (schema) file, we add a schema pre() middleware, that executes when .save() is called, before the save occurs:
// An arrow function will not work on this guy, if you want to use the "this" keyword
materialSchema.pre('save', function(next) {
this.model('material').findOne() // Don't forget the .model(...) bit!
.sort({bookID : -1}) // All I need is the highest (i.e. most recent) bookID
.select('bookID') // Ditto above (not really necessary)
.exec((err, result) => {
if(err) {
return next(err); // Oopsies, an error!
}else if(!result) {
this.bookID = 'A-001'; // The case when collection is empty
}else {
this.bookID = getNextID(result.bookID); // Otherwise, increment ID
}
next(); // Don't forget this sucker! This is how you save
});
});
And that's about it! It isn't an in-built solution direct from Mongoose, but it works a treat.
Just for completeness, the getNextID function looks like:
function getNextID(curID) {
let letter = curID.split('-')[0];
let number = parseInt(curID.split('-')[1]);
if(number >= 100) { // Increase the letter and reset the number
letter = String.fromCharCode(letter.charCodeAt(0) + 1)
number = '001';
}else { // Only increase the number
number = ('' + (number + 1)).padStart(3, '0'); // Makes sure the numbers are always 3 digits long
}
return `${letter}-${number}`;
}
This'll do just dandy for now. Until we get to Z100. But I'll cross that bridge if/when it comes. No big deal at all.
And you don't need to do anything special to use it. Just save a new doc as normal, and it automatically fires:
new Material({
// New material properties
}).save((err, mat) => {
// Handle errors and returns ...
});
I'm developing a small NodeJS web app using Mongoose to access my MongoDB database. A simplified schema of my collection is given below:
var MySchema = mongoose.Schema({
content: { type: String },
location: {
lat: { type: Number },
lng: { type: Number },
},
modifierValue: { type: Number }
});
Unfortunately, I'm not able to sort the retrieved data from the server the way it is more convenient for me. I wish to sort my results according to their distance from a given position (location) but taking into account a modifier function with a modifierValue that is also considered as an input.
What I intend to do is written below. However, this sort of sort functionality seems to not exist.
MySchema.find({})
.sort( modifierFunction(location,this.location,this.modifierValue) )
.limit(20) // I only want the 20 "closest" documents
.exec(callback)
The mondifierFunction returns a Double.
So far, I've studied the possibility of using mongoose's $near function, but this doesn't seem to sort, not allow for a modifier function.
Since I'm fairly new to node.js and mongoose, I may be taking a completely wrong approach to my problem, so I'm open to complete redesigns of my programming logic.
Thank you in advance,
You might have found an answer to this already given the question date, but I'll answer anyway.
For more advanced sorting algorithms you can do the sorting in the exec callback. For example
MySchema.find({})
.limit(20)
.exec(function(err, instances) {
let sorted = mySort(instances); // Sorting here
// Boilerplate output that has nothing to do with the sorting.
let response = { };
if (err) {
response = handleError(err);
} else {
response.status = HttpStatus.OK;
response.message = sorted;
}
res.status(response.status).json(response.message);
})
mySort() has the found array from the query execution as input and the sorted array as output. It could for instance be something like this
function mySort (array) {
array.sort(function (a, b) {
let distanceA = Math.sqrt(a.location.lat**2 + a.location.lng**2);
let distanceB = Math.sqrt(b.location.lat**2 + b.location.lng**2);
if (distanceA < distanceB) {
return -1;
} else if (distanceA > distanceB) {
return 1;
} else {
return 0;
}
})
return array;
}
This sorting algorithm is just an illustration of how sorting could be done. You would of course have to write the proper algorithm yourself. Remember that the result of the query is an array that you can manipulate as you want. array.sort() is your friend. You can information about it here.
I have a hasMany to belongsTo relation between boxes and box_states.
I want to grab all my boxes with their latest state, so this is my query:
new Box()
.orderBy('id', 'ASC')
.fetchAll({
withRelated: [{
'states' : function(qb) {
qb.limit(1)
.orderBy('id', 'DESC');
}
}]
})
.then(function (boxes) {
res.json(boxes);
});
This returns all the boxes, but only the last box has one relation; everything has no relation.
Help would be appreciated.
Well, this works pretty much as expected. If you debug your solution, you'll see something like this (I'm giving you my example, which has the same relation as yours - one town, many addresses):
select "addresses".* from "addresses" where "addresses"."town_id" in (1, 2, 3, 4) group by "id" limit 1
What this does is basically select just ONE(!) related model (address) for ALL instances (the addresses.town_id in (1,2,3,4) segment).
So basically what you need to do is foreach the main results and go withRelated upon each.
new Town().fetchAll().then(function(towns) {
var my_towns_with_only_last_address = [];
Promise.map(towns.toJSON(), function(town) {
return new Town({
id: town.id
}).fetch({
withRelated: [{
'address': function(qb) {
qb.limit(1);
qb.orderBy('id', 'desc');
}
}]
}).then(function(result) {
return my_towns_with_only_last_address.push(result.toJSON());
});
}).then(function() {
res.json(my_towns_with_only_last_address);
});
});
I'm not sure if there are any better options... this works.
The best way to do this now (newest release of Bookshelf) is to use a Collection. Although withRelated is not available in .fetchAll(), it is available when using .fetch() on a Collection. See http://bookshelfjs.org/#Collection-instance-fetch