MongoDB Location NearBy unique - node.js

I have created a collection in DB and updated with proper indexing to retrieve it back. Im getting with the folloging
Example:
db.car.save({ "name":"Toyota car", "affiliation":"Toyota", "loc":{"lon":55.93939251390387,"lat":-113.999}})
db.car.find({"loc" : {"$within" : {"$center" : [[50.93939251390,-114],5]}}})
I have 100K + records in Database when I search the above query returns 20K records.
Actually this has some duplicate values in the "name" column. How can I get the distinct value of "name".

You can't get the results you need using a query in MongoDB 2.0.x, so will need to manipulate the results in your application code.
If you are only after names you can limit the fields in the output:
db.car.find(
{"loc" : {"$within" : {"$center" : [[50.93939251390,-114],5]}}},
{"_id": 0, "name": 1}
)
If you want to find distinct names you could do something like:
// Save cars we've seen
> var cars_seen = {}
// The unique list we actually want
> var cars = []
// Find closest cars to given geo point
> db.car.find(
{"loc" : {"$within" : {"$center" : [[50.93939251390,-114],5]}}},
{"_id": 0, "name": 1}
).forEach(
function(doc) {
// Add names we haven't seen yet
if (!cars_seen[doc.name]) {
cars_seen[doc.name] = 1;
cars.push(doc.name);
}
}
)
> cars
[ "Skoda", "Benz", "Skoda SUV" ]

Related

Azure Search. How to get result counts when having pagination

Lets say I have following schema in Azure Search Collection, with 100s of records.
{
"id": '1',
"Status" : "Available",
"name" : "demo 1"
},
{
"id": '2',
"Status" : "Available",
"name" : "demo 1"
},
{
"id": '3',
"Status" : "Removed",
"name" : "demo 1"
},
{
"id": '4',
"Status" : "Booked",
"name" : "demo 4"
}
In My Status field I can have three different values,
"Booked", "Available", "Removed".
Now I am getting the data by pagination using Skip and Top from the Azure Search.
However like in ElasticSearch Aggregation function, is there a way in Azure search to get total no of sites having status Booked , Available or not removed etc..
Because I cant do a count in Client Side because, I will have limited no of records, not all records from Azure Search.
If you are using a search index client object (Microsoft.Azure.Search.SearchIndexClient) to search for your documents you can provide an object as a parameter (Microsoft.Azure.Search.Models.SearchParameters) that contains a property IncludeTotalResultCount. Once you set this property to true and call the method Documents.Search(...) passing the parameter object, your response object (Microsoft.Azure.Search.DocumentSearchResult) will contain a value for the property Count considering the total count for the filter, regardless how many items selected by the pagination.
SearchParameters searchParameter = new SearchParameters
{
Filter = "organizationId eq '1'",
Skip = 0,
Top = 20,
IncludeTotalResultCount = true
};
using (var client = new SearchIndexClient(...)
{
DocumentSearchResult response = client.Documents.Search("*", searchParameter);
//pagination items
var collection = response.Results.ToArray();
//total items
var counter = response.Count;
}
When you create the index, make the status field facetable and filterable. Then issue a facets query on status, with $filter=status ne 'Booked'. This will give you the total counts of documents in each status category other than Booked, irrespective of pagination.

paginating using angularfire2

suppose I have a data structure in firebase real time database like
{ "donors" :
"uid1" : { "name" : "x", "bloodGroup" : "A+", "location" : "some Place"},
"uid2" : { "name" : "y", "bloodGroup" : "A-", "location" : "some place"},
...
...
}
now if I have millions of donor records like this. how could I filter them based on bloodGroup location and fetching say 100 records from server at a time using angularfire2.
I have found this page which was really helpful to me when using queries to query my firebase data:
https://howtofirebase.com/collection-queries-with-firebase-b95a0193745d
A very simple example would be along the lines of:
this.donorsData = af.database.list('/donors', {
query: {
orderByChild: 'bloodGroup',
equalTo: 'A+',
}
});
Not entirely sure how to fetch 100 records, then another 100, I am using datatables in my app, which fetches all my data and using the datatables for pagination.

How to query parent based on subdocument's _id?

consider the following records:
user record
{
"_id" : ObjectId("5234ccb7687ea597eabee677"),
"class" : [
{ "_id" : ObjectId("5234ccb7687ea597eabee671", "num" : 10, "color" : "blue" },
{ "_id" : ObjectId("5234ccb7687ea597eabee672", "num" : 100, "color" : "blue" }
]
}
this user has two class sub records, now I need a query that finds all users that have class property where "class._id" has a value of at least one users "class._id"
here is a more detail example:
suppose there is four user:
A:{_id:432645624232345,class:[{_id:123,name:'foo'}]}
B:{_id:432645624232555,class:[{_id:555,name:'foo'},{_id:123,name:'foo'}]}
C:{_id:432645344232345,class:[{_id:555,name:'foo'},{_id:111,name:'www'}]}
D:{_id:432644444232345,class:[{_id:222,name:'sss'},{_id:555,name:'www'},{_id:123,name:'foo'}]}
now if B login , I need to query all the user whose class subdocument contains at least one document which's _id==555 or _id==123 (555 and 123 come from B user), in this case the query result should be:
A:{_id:432645624232345,class:[{_id:123,name:'foo'}]} // match _id=123
B:{_id:432645624232555,class:[{_id:555,name:'foo'},{_id:123,name:'foo'}]} //match _id=123 and _id=555
C:{_id:432645344232345,class:[{_id:555,name:'foo'},{_id:111,name:'www'}]} //match _id=555
D:{_id:432644444232345,class:[{_id:222,name:'sss'},{_id:555,name:'www'},{_id:123,name:'foo'}]} ///match _id=123 and _id=555
which is all the user.
so far i get this:
{"class._id" : { $in : ["5234ccb7687ea597eabee671", "5234ccb7687ea597eabee672"] } }
but when different user login the class._id query condition is different. So is there any operator to do this
{"class._id" : { $in : req.user.class } }
hope I made myself clear.
In order to achieve what you want, first you must isolate the class _ids in an array, and then use it in the query argument.
var classIds = [];
var i = 0;
while (i < req.user.class.length) {
classIds.push(req.user.class[i]._id);
i++;
}
After that you can use classIds array in the query:
{"class._id" : { $in : classIds } }
The following query condition would give you all the users that have at least one class with id equal to any of the elements in the given array:
{"class._id" : { $in : ["5234ccb7687ea597eabee671", "5234ccb7687ea597eabee672"] } }
In the array for the $in clause you may provide any id's you needed , comma separated.
In addition, if you needed such, the below query condition should check for existence of nested document within "class" property that has a property "_id" :
{ "class._id" : { $exists : true } }
Both conditions should work no matter if "class._id" is a single-valued property or an array (mongo supports that).

Insertion order of array elements in MongoDB

I am having trouble preserving insertion order of a bulk insert into mongodb.
My applications requires posting data continuously (via HTTP POST # once a second) to a server. On the server side, the HTTP POST is handled and this data is stored in a capped collection in mongodb v2.4. The size of this capped collection is large (50MB). The format of this data is JSON and it has arrays in it like this:
{"Data":[{"Timestamp":"2014-08-02 13:38:18:852","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0021321268286556005,"b":-0.0010663296561688185}],"Monkec":[{"a":17.511783599853516,"c":-0.42092469334602356,"b":-0.42092469334602356}]},{"Timestamp":"2014-08-02 13:38:18:858","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.892329216003418,"c":-0.2339634746313095,"b":-0.2342628538608551}]},{"Timestamp":"2014-08-02 13:38:18:863","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0021321268286556005,"b":0.0021315941121429205}],"Monkec":[{"a":9.702523231506348,"c":-0.24264541268348694,"b":-0.2148033082485199}]},{"Timestamp":"2014-08-02 13:38:18:866","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.665101051330566,"c":-0.23366409540176392,"b":-0.2197430431842804}]},{"Timestamp":"2014-08-02 13:38:18:868","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.693991661071777,"c":-0.2936892807483673,"b":-0.22857467830181122}]},{"Timestamp":"2014-08-02 13:38:18:872","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.684710502624512,"c":-0.2296224981546402,"b":-0.13786330819129944}]},{"Timestamp":"2014-08-02 13:38:18:873","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.67707633972168,"c":-0.31255003809928894,"b":-0.1902543604373932}]},{"Timestamp":"2014-08-02 13:38:18:875","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0}],"Monkec":[{"a":9.739496231079102,"c":-0.1899549812078476,"b":-0.18845809996128082}]},{"Timestamp":"2014-08-02 13:38:18:878","Rabbit":[{"a":-0.003197923768311739,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.721234321594238,"c":-0.19205063581466675,"b":-0.17318984866142273}]},{"Timestamp":"2014-08-02 13:38:18:881","Rabbit":[{"a":-0.003197923768311739,"c":-0.003197923768311739,"b":0.0010657970560714602}],"Monkec":[{"a":9.78545093536377,"c":-0.2501298487186432,"b":-0.1953437775373459}]},{"Timestamp":"2014-08-02 13:38:18:882","Rabbit":[{"a":0,"c":-0.0010663296561688185,"b":0.0021315941121429205}],"Monkec":[{"a":9.686058044433594,"c":-0.21630020439624786,"b":-0.18247054517269135}]},{"Timestamp":"2014-08-02 13:38:18:884","Rabbit":[{"a":-0.0010663296561688185,"c":0,"b":0.0010657970560714602}],"Monkec":[{"a":9.67198657989502,"c":-0.18546432256698608,"b":-0.23156845569610596}]},{"Timestamp":"2014-08-02 13:38:18:887","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.640103340148926,"c":-0.23276595771312714,"b":-0.25686585903167725}]},{"Timestamp":"2014-08-02 13:38:18:889","Rabbit":[{"a":-0.0010663296561688185,"c":0,"b":0}],"Monkec":[{"a":9.739346504211426,"c":-0.19130218029022217,"b":-0.22602996230125427}]},{"Timestamp":"2014-08-02 13:38:18:891","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0}],"Monkec":[{"a":9.716594696044922,"c":-0.22543121874332428,"b":-0.19728973507881165}]},{"Timestamp":"2014-08-02 13:38:18:898","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.682914733886719,"c":-0.28680360317230225,"b":-0.1740879863500595}]},{"Timestamp":"2014-08-02 13:38:18:904","Rabbit":[{"a":-0.0010663296561688185,"c":0,"b":0.0021315941121429205}],"Monkec":[{"a":9.693093299865723,"c":-0.20866607129573822,"b":-0.2586621046066284}]},{"Timestamp":"2014-08-02 13:38:18:907","Rabbit":[{"a":-0.0021321268286556005,"c":-0.0010663296561688185,"b":0}],"Monkec":[{"a":9.690997123718262,"c":-0.18681152164936066,"b":-0.23216719925403595}]},{"Timestamp":"2014-08-02 13:38:18:910","Rabbit":[{"a":-0.003197923768311739,"c":-0.0010663296561688185,"b":0.0010657970560714602}],"Monkec":[{"a":9.671688079833984,"c":-0.15388000011444092,"b":-0.2588118016719818}]},{"Timestamp":"2014-08-02 13:38:19:055","Rabbit":[{"a":-0.0010663296561688185,"c":-0.0010663296561688185,"b":0}],"Monkec":[{"a":9.689650535583496,"c":-0.23605911433696747,"b":-0.1989363133907318}]}],"Serial":"35689"}
I am inserting this in mongodb (using NodeJs MongoClient driver) using a bulk insert command:
var length = 20; // only doing 20 inserts for testing purposes
for (var i = 0; i <length;i++) {
var bulk = col.initializeUnorderedBulkOp();
bulk.insert(data["Data"][i]); // data is my JSON data of interest
bulk.execute(function(err) {
if (err) {
return cb(err);
}
if (++inserted == length) {
cb(); // callback (not seen in this code snippet)
}
}); // end function
} // end of for loop
However, when I examine the entries in the database, they are not inserted in the order in which the data resides in the originating JSON array. My source data is in ascending Timestamp order, but a few entries in the mongodb capped collection are out of order. For instance, I see this:
{ "Timestamp" : "2014-08-02 13:38:18:910", "Rabbit" : [ { "a" : -0.003197923768311739, "c" : -0.0010663296561688185, "b" : 0.0010657970560714602 } ], "Monkec" : [ { "a" : 9.671688079833984, "c" : -0.15388000011444092, "b" : -0.2588118016719818 } ], "_id" : ObjectId("548e67a683946a5d25bc6d1a") }
{ "Timestamp" : "2014-08-02 13:38:18:884", "Rabbit" : [ { "a" : -0.0010663296561688185, "c" : 0, "b" : 0.0010657970560714602 } ], "Monkec" : [ { "a" : 9.67198657989502, "c" : -0.18546432256698608, "b" : -0.23156845569610596 } ], "_id" : ObjectId("548e67a683946a5d25bc6d13") }
{ "Timestamp" : "2014-08-02 13:38:18:904", "Rabbit" : [ { "a" : -0.0010663296561688185, "c" : 0, "b" : 0.0021315941121429205 } ], "Monkec" : [ { "a" : 9.693093299865723, "c" : -0.20866607129573822, "b" : -0.2586621046066284 } ], "_id" : ObjectId("548e67a683946a5d25bc6d18") }
so Timestamp" : "2014-08-02 13:38:18:910 is stored before "Timestamp" : "2014-08-02 13:38:18:884" even though it is the other way around in the source JSON.
How to ensure mongodb inserts data in the correct order? I also tried non bulk inserts (db.col.insert or db.col.insertOne) but still get this inconsistency. Thank You
If your queries aren't asking for any specific sorting/ordering, MongoDB makes no guarantees as to in which order they'll be returned.
How you insert your data is irrelevant. What you need to do is write your find query like this:
// Sort by ascending timestamp
db.my_collection.find({ ... }).sort({"TimeStamp": 1})
See http://docs.mongodb.org/manual/reference/method/cursor.sort/#cursor.sort for more information on how sorting works.
Of course, if you want to do that, you'll greatly benefit from adding an index on Timestamp to your collection (see http://docs.mongodb.org/manual/core/indexes/).

Nodejs mongo return data with pagination information

I am using node and mongo with the native client.
I would like to add pagination to my application.
To get pagination, I need my responses to always return count alongside data
I would like to get something like:
{
count : 111,
data : [ { 'a' : 'only first item was requested' } ]
}
I can do this in mongo
> var guy = db.users.find({}).limit(1)
> guy.count()
11
> guy.toArray()
[
{
"_id" : ObjectId("5381a7c004fb02b10b557ee3"),
"email" : "myEmail#guy.com",
"fullName" : "guy mograbi",
"isAdmin" : true,
"password" : "fe20a1f102f49ce45d1170503b4761ef277bb6f",
"username" : "guy",
"validated" : true
}
]
but when I do the same with nodejs mongo client I get errors.
var cursor = collection.find().limit(1);
cursor.toArray( function(){ .. my callback .. });
cursor.count();
It seems that
count is not defined on cursor
that once I applied toArray on cursor, I cannot use the cursor again
How, using nodejs, can I accomplish the same thing I can with mongo directly?
As others have said, if you want to have a total count of the items and then the data you will need to have two queries, there is no other way. Why are you concerned with creating two queries?

Resources