I have a model named Rest with lots of columns and i only want to fetch few among them while applying near operator on that Rest model. this is my code
Rest
.select('rest_status rest_address rest_name rest_contact rest_photo rest_menu rest_avg_rating')
.aggregate().near({
near:[parseFloat(req.body.lng),parseFloat(req.body.lat)],
maxDistance:100000,
spherical:true,
distanceField:"dist.calculated"
})
.then(rests =>{
// const response=[];
// for(const rest of rests){
// console.log(rest);
// response.push(rest);
// }
res.send({rests,response_status});
}).catch(err => res.send(err));
when i try like this . i get an error that select is not a function. i tried changing select position like below aggregate and near but it didnt work. I'm new to this mongoose,please tell me if there is any function or way around to fetch limited columns from my model.
i forgot to mention both near and select and working fine when other one is not used and also please help me with changing the data obtain from model
You could do something like this below:
Rest.aggregate([
{
$geoNear: {
near: {
type: "Point",
coordinates: [parseFloat(req.body.lng), parseFloat(req.body.lat)]
},
maxDistance: 100000,
spherical: true,
distanceField: "dist.calculated"
}
},
{
$project: {
rest_status: 1,
rest_address: 1,
rest_name: 1,
rest_contact: 1,
rest_photo: 1,
rest_menu: 1,
rest_avg_rating: 1
}
}
//even you can add limit skip below , if u need
,{$limit:<Number>},
{ $skip: <Number>}
]
//depends on your mongo version you may need to set cursor as well.
,
{ cursor: { batchSize: <Number or keep 0> } }
)
.then()
.catch();
Note: before execute the query make sure you have added 2dsphere index to your dist.calculated field
in case of aggegate $project is used for what you want to do with .select
instead of using .near() i have used $geoNear.
if you want to use limit skip you can follow the example, other wise can remove that.
you also can add distanceMultiplier and includeLocs field depends on your requirement.
in the above, depends on the mongoDB version you may need to use cursor in aggregate.
if not you can go aheadwithout using the cursor.
Hope this helps.
if you still get any error, please comment.
Related
On a daily basis, I'm pushing data (time_series) to Elasticsearch. I created an index pattern, and my index have the name: myindex_* , where * is today date (an index pattern has been setup). Thus after a week, I have: myindex_2022-06-20, myindex_2022-06-21... myindex_2022-06-27.
Let's assume my index is indexing products' prices. Thus inside each myindex_*, I have got:
myindex_2022-06-26 is including many products prices like this:
{
"reference_code": "123456789",
"price": 10.00
},
...
myindex_2022-06-27:
{
"reference_code": "123456789",
"price": 12.00
},
I'm using this query to get the reference code and the corresponding prices. And it works great.
const data = await elasticClient.search({
index: myindex_2022-06-27,
body: {
query: {
match: {
"reference_code": "123456789"
}
}
}
});
But, I would like to have a query that if in the index of the date 2022-06-27, there is no data, then it checks, in the previous index 2022-06-26, and so on (until e.g. 10x).
Not sure, but it seems it's doing this when I replace myindex_2022-06-27 by myindex_* (not sure it's the default behaviour).
The issue is that when I'm using this way, I got prices from other index but it seems to use the oldest one. I would like to get the newest one instead, thus the opposite way.
How should I proceed?
If you query with index wildcard, it should return a list of documents, where every document will include some meta fields as _index and _id.
You can sort by _index, to make elastic search return the latest document at position [0] in your list.
const data = await elasticClient.search({
index: myindex_2022-*,
body: {
query: {
match: {
"reference_code": "123456789"
}
}
sort : { "_index" : "desc" },
}
});
I am trying to build a "number of visitors" collection in mongoDb using Node.JS backend of my website. The frontend sends the following info to Node.JS backend as JSON.
isUniqueVisitor - 1 if yes, 0 if no
country - standard country code - "JP", "IN", "UK", etc
My database looks like following
{
"today": 2019-06-07,
"uniqueVisitors": {
"count": 230,
"countries": {
"JP": 102,
"IN": 88,
"UK": 30
}
}
}
It works well if I use $inc with fixed values
Eg. $inc: {count: 1} // for string/integers keys
Eg. $inc: {"uniqueVisitors.count": 1} // inside quotes to access key of a JSON
Main issue:
I am not able to access a document name using variable.
Eg. $inc: {`uniqueVisitors.countries[${req.body.country}]`}
This creates an error as backticks can't be used for Mongo.
I tried with
Eg. $inc: {uniqueVisitors["countries"][req.body.country]}
But even this creates error.
I followed the web and found that mongo $set using variables can be realized by passing the required JSON directly to $set. Hence I resorted to code it the following way.
mongoClient.connect(mongoURL, async function (err, db) {
if (err) throw err;
console.log("Database connected");
// Identifying my document with today's date
var myQuery = {
date: getTodayDate()
};
// Defining the JSON to be passed to uniqueVisitors $inc
var uniqueVisitorsInc = {
"uniqueVisitors": {
"count": 0,
"countries": {}
}
};
// Populating the JSON to be passed to uniqueVisitors $inc => essentially asking to increase count by 1 and increase that country's count by 1
uniqueVisitorsInc["uniqueVisitors"]["count"] = 1;
uniqueVisitorsInc["uniqueVisitors"]["countries"][myData.country] = 1;
var newValues = {
$inc: uniqueVisitorsInc
};
await db.collection("visitorStats").update(myQuery, newValues, {upsert: true});
db.close();
});
The above method worked well on editor but threw the following runtime error:
$inc requires numerical values
Basically asking me to pass values to $inc in {var1: 1, var2: 5} pattern.
Please help me bypass this weird situation.
I know I can do a two step process where I read the values first, increment in variable and $set it in Mongo.
But does anyone know how to overcome this situation using $inc?
If this update were hardcoded to update "JP" only, it'd need to look like:
$inc: { "uniqueVisitors.country.JP": 1 }
So you were almost there with the backtick method but change the syntax a bit and keep the : 1 part like so:
$inc: { [`uniqueVisitors.country.${req.body.country}`]: 1 }
const A = mongoose.Schema({
...
bs: [B.schema],
...
});
So basically i have two schemas, and one is subdocument of another.
From my datatable i get params. like filter, page limit, page, sort...
What i need to do is, to create query that will with _id from A schema get all his B schemas and always sort, limit, skip, filter with params. that i sent
I tried something like this
b = await A.find({'_id' : idA},
{ 'bs' :
{ $slice: [ offset * limit, limit ]
}
});
And it's working but i can't still figure out how to filter and sort.
So if somebody have some idea welcome to share.
P.S Sorry for bad english
Regards,
Salesh
What you're trying to do is not find A documents that fulfill your array criteria, but to modify the results to accommodate to your needs. You can do this with two approaches, depending on where you want the processing to be done:
1. Use MongoDB Aggregation. The processing is done in the DB.
The aggregation pipeline is a series of steps you determine that documents go through being queried and transformed.
A rough untested (and probably syntactically wrong) example would be:
A.aggregate([
{ $match: { _id: "id" }},
{ $project: {
bs: {
$filter: { input: "$bs" , as: "filteredBs" , cond: { /* conditions object */} }},
}
},
{ $slice: ["$filteredBs", offset * limit, limit ] }
/* ... */
]);
2. Get the document by Id and process the array on your server.
Here you're just limited by javascript and its array capabilites.
const found = A.findById('id');
const bs = A.bs.filter( /* filter function */ ).slice() // ... whatever you want.
A.bs = bs;
return A;
The query receives a pair of coordinates, a maximum Distance radius, a "skip" integer and a "limit" integer. The function should return the closest and newest locations according to the position given. There is no visible error in my code, however, when I call the query again, it returns repeated results. "skip" variable is updated according to the results returned.
Example:
1) I make query with skip = 0, limit = 10. I receive 10 non-repeated locations.
2) Query is called again now, skip = 10, limit = 10. I receive another 10 locations with repeated results from the first query.
QUERY
Locations.find({ coordinates :
{ $near : [ x , y ],
$maxDistance: maxDistance }
})
.sort('date_created')
.skip(skip)
.limit(limit)
.exec(function(err, locations) {
console.log("[+]Found Locations");
callback(locations);
});
SCHEMA
var locationSchema = new Schema({
date_created: { type: Date },
coordinates: [],
text: { type: String }
});
I have tried looking everywhere for a solution. My only option would be versions of Mongo? I use mongoose 4.x.x and mongodb is like 2.5.6. I believe. Any ideas?
There are a couple of things to consider here in the sort of results that you want, with the first consideration being that you have a "secondary" sort criteria in the "date_created" to deal with.
The basic problem there is that the $near operator and like operators in MongoDB do not at present "project" any field to indicate the "distance" from the queried location, and simply just "default sort" the data. So in order to do that "secondary" sort, a field with the "distance" needs to be present. There are therefore other options for this.
The second case is that "skip" and "limit" style paging is horrible form performance on large sets of data and should be avoided where you can. So it's better to select data based on a "range" where it occurs rather than "skip" through all the results you have previously displayed.
The first thing to do here is use a command that can "project" the distance into the document along with the other information. The aggregation command of $geoNear is good for this, and especially since we want to do other sorting:
var seenIds = [],
lastDistance = null,
lastDate = null;
Locations.aggregate(
[
{ "$geoNear": {
"near": [x,y],
"maxDistance": maxDistance
"distanceField": "dist",
"limit": 10
}},
{ "$sort": { "dist": 1, "date_created": -1 }
],
function(err,results) {
results.forEach(function(result) {
if ( ( result.dist != lastDistance ) || ( result.date_created != lastDate ) ) {
seenIds = [];
lastDistance = result.dist;
lastDate = result.date_created;
}
seenIds.push(result._id);
});
// save those variables to session or other persistence
// do something with results
}
)
That is the first iteration of your results where you fetch the first 10. Noting the logic inside the loop, where each document in the results is inspected for either a change in the "date_created" or the projected "dist" field now present in the document and where this occurs the "seenIds" array is wiped of all current entries. The general action is that all the variables are tested and possibly updated on each iteration and where there is no change then items are added to the list of "seenIds".
All those three variables being worked on need to be stored somewhere awaiting the next request. For web applications the session store is ideal, but different approaches vary. You just want those values to be recalled when we start the next request, as on the next and subsequent iterations we alter the query a bit:
Locations.aggregate(
[
{ "$geoNear": {
"near": [x,y],
"maxDistance": maxDistance,
"minDistance": lastDistance,
"distanceField": "dist",
"limit": 10,
"query": {
"_id": { "$nin": seenIds },
"date_created": { "$lt": lastDate }
}
}},
{ "$sort": { "dist": 1, "date_created": -1 }
],
function(err,results) {
results.forEach(function(result) {
if ( ( result.dist != lastDistance ) || ( result.date_created != lastDate ) ) {
seenIds = [];
lastDistance = result.dist;
lastDate = result.date_created;
}
seenIds.push(result._id);
});
// save those variables to session or other persistence
// do something with results
}
)
So there the "minDistance" parameter is entered as you want to exclude any of the "nearer" results that have already been seen, and the additional checks are placed in the query with the "date_created" needing to be "less than" the "lastDistance" recorded as well since we are in descending order of sort, with the final "sure" filter in excluding any "_id" values that were recorded within the list because the values had not changed.
Now with geospatial data that "seenIds" list is not likely to grow as generally you are not going to find things all at the same distance, but it is a general process of paging a sorted list of data like this, so it is worth understanding the concept.
So if you want to be able to use a secondary field to sort on with geospatial data and also considering the "near" distance then this is the general approach, by projecting a distance value into the document results as well as storing the last seen values before any changes that would not make them unique.
The general concept is "advancing the minimum distance" to enable each page of results to get gradually "further away" from the source point of origin used in the query.
I use Node.js and MongoDB with monk.js and i want to do the logging in a minimal way with one document per hour like:
final doc:
{ time: YYYY-MM-DD-HH, log: [ {action: action1, count: 1 }, {action: action2, count: 27 }, {action: action3, count: 5 } ] }
the complete document should be created by incrementing one value.
e.g someone visits a webpage first this hour and the incrementation of action1 should create the following document with a query:
{ time: YYYY-MM-DD-HH, log: [ {action: action1, count: 1} ] }
an other user in this hour visits an other webpage and document should be exteded to:
{ time: YYYY-MM-DD-HH, log: [ {action: action1, count: 1}, {action: action2, count: 1} ] }
and the values in count should be incremented on visiting the different webpages.
At the moment i create vor each action a doc:
tracking.update({
time: moment().format('YYYY-MM-DD_HH'),
action: action,
info: info
}, { $inc: {count: 1} }, { upsert: true }, function (err){}
Is this possible with monk.js / mongodb?
EDIT:
Thank you. Your solution looks clean and elegant, but it looks like my server can't handle it, or i am to nooby to make it work.
i wrote a extremly dirty solution with the action-name as key:
tracking.update({ time: time, ts: ts}, JSON.parse('{ "$inc":
{"'+action+'": 1}}') , { upsert: true }, function (err) {});
Yes it is very possible and a well considered question. The only variation I would make on the approach is to rather calculate the "time" value as a real Date object ( Quite useful in MongoDB, and manipulative as well ) but simply "round" the values with basic date math. You could use "moment.js" for the same result, but I find the math simple.
The other main consideration here is that mixing array "push" actions with possible "updsert" document actions can be a real problem, so it is best to handle this with "multiple" update statements, where only the condition you want is going to change anything.
The best way to do that, is with MongoDB Bulk Operations.
Consider that your data comes in something like this:
{ "timestamp": 1439381722531, "action": "action1" }
Where the "timestamp" is an epoch timestamp value acurate to the millisecond. So the handling of this looks like:
// Just adding for the listing, assuming already defined otherwise
var payload = { "timestamp": 1439381722531, "action": "action1" };
// Round to hour
var hour = new Date(
payload.timestamp - ( payload.timestamp % ( 1000 * 60 * 60 ) )
);
// Init transaction
var bulk = db.collection.initializeOrderedBulkOp();
// Try to increment where array element exists in document
bulk.find({
"time": hour,
"log.action": payload.action
}).updateOne({
"$inc": { "log.$.count": 1 }
});
// Try to upsert where document does not exist
bulk.find({ "time": hour }).upsert().updateOne({
"$setOnInsert": {
"log": [{ "action": payload.action, "count": 1 }]
}
});
// Try to "push" where array element does not exist in matched document
bulk.find({
"time": hour,
"log.action": { "$ne": payload.action }
}).updateOne({
"$push": { "log": { "action": payload.action, "count": 1 } }
});
bulk.execute();
So if you look through the logic there, then you will see that it is only ever possible for "one" of those statements to be true for any given state of the document either existing or not. Technically speaking, the statment with the "upsert" can actually match a document when it exists, however the $setOnInsert operation used makes sure that no changes are made, unless the action actually "inserts" a new document.
Since all operations are fired in "Bulk", then the only time the server is contacted is on the .execute() call. So there is only "one" request to the server and only "one" response, despite the multiple operations. It is actually "one" request.
In this way the conditions are all met:
Create a new document for the current period where one does not exist and insert initial data to the array.
Add a new item to the array where the current "action" classification does not exist and add an initial count.
Increment the count property of the specified action within the array upon execution of the statement.
All in all, yes posssible, and also a great idea for storage as long as the action classifications do not grow too large within a period ( 500 array elements should be used as a maximum guide ) and the updating is very efficient and self contained within a single document for each time sample.
The structure is also nice and well suited to other query and possible addtional aggregation purposes as well.