I have been trying to use https://github.com/wearetheledger/node-couchdb-query-engine for querying the couchdb based on the selector query which would be supplied by the UI based on the action the user makes.
the code below used in the library takes cars as a constant which in my case need to be on the actual couchdb. the secord parameter (the selector one would be supplied from the UI)
const filteredResult = queryEngine.parseQuery( cars, {selector:{year:{$gt:2016}}} )
The main issue is i want the first parameter (cars here) to be a kind of db connection on which the selector query gets applied.
Regards,
Saurabh
The code you're trying to use is just providing a SearchInterface on a Javascript object.
If you want to use dynamic queries on CouchDB, you can do it natively with the Mango.
Since you're using NodeJS, you could use the official NodeJS library client for CouchDB:
Nano: https://github.com/apache/couchdb-nano
Code example:
const nano = require('nano')('http://localhost:5984');
nano.db.create('alice');
const alice = nano.db.use('alice');
/ find documents where the name = "Brian" and age > 25.
const q = {
selector: {
name: { "$eq": "Brian"},
age : { "$gt": 25 }
},
fields: [ "name", "age", "tags", "url" ],
limit:50
};
alice.find(q).then((doc) => {
console.log(doc);
});
Related
I have several documents in a collection in MongoDB as shown in this snapshot . I'm creating a web app for which this is the back-end data.
Every document has an attribute name. The user has an option to search for the name in the front-end. Now, suppose the user searches for "Sam", then I'm returning the documents where name is either exactly "Sam" or starts with "Sam". I have implemented this using the following code:
let name = "^"+req.body.name;
const name_regex = new RegExp(name, 'gi');
try {
let members = await Member.find({ name: req.body.name=="" ? /^$|/ : name_regex});
res.send(members);
}
i.e., if the user dosen't provide a name, return all documents, else return documents matching the regex.
So , if the user searches for "Sam", the output is all 3 documents since all of them have name starting with "Sam", but if the user searches for "Samm", then only one document is returned where the name is "Sammy".
Now, I want to implement the same logic on age attribute, i.e., if the user searches for age: 2, then I want to return all documents where age is either exactly 2 or starts with the digit 2. But I'm unable to use the above method sine it only works with strings.
Note: I'm using mongoose package and Express framework.
You can do this:
// regular expression matching anything that starts with two
const age_regex = /^2/;
Member.find({
$expr: {
$regexMatch: {
input: {
$toString: "$age" // Casting the age value to a string
},
regex: age_regex
}
}
})
Some usefule explanatory links: $expr, $regexMatch, $toString
I am building a platform for students to give mock test on. Once the test is complete, a results are to be generated for them relative to other students who attempted the said test.
Report contains multiple parameters i.e. rank, rank within their batch, and stuff like average marks people got on the given test are updated.
To get each of this data, I need to perform a separate query on the database and then I got to update 1. the result of current user who attempted the test 2. the result of everyone else (i.e. everyone's rank changes on new attempts)
So I need to perform multiple queries to get the data and run 2-3 update queries to set the new data.
Given mongodb calls are asynchronous, I can't find a way to gather all of that data at one place to be updated.
One way is to put the next query within the callback function of the previous query but I feel like there should be a better way than that.
Maybe you could use Promise.all()
Example:
const initialQueries = []
initialQueries.push("Some promise(s)")
Promise.all(initialQueries).then(results => {
// After all initialQueries are finished
updateQueries()
}).catch(err => {
// At least one failed
})
Use db.collection.bulkWrite.
It allows multiple document insertions, updates (by _id or a custom filter), and even deletes.
const ObjectID = require('mongodb').ObjectID;
db.collection('tests').bulkWrite([
{ updateOne : {
"filter" : { "_id" : ObjectID("5d400af131602bf3fa09da3a") },
"update" : { $set : { "score" : 20 } }
}
},
{ updateOne : {
"filter" : { "_id" : ObjectID("5d233e7831602bf3fa996557") },
"update" : { $set : { "score" : 15 } }
}
}
]);
New in version 3.2.
I have been working with the google cloud library, and I can successfully save data in DataStore, specifically from my particle electron device (Used their tutorial here https://docs.particle.io/tutorials/integrations/google-cloud-platform/)
The problem I am now having is retrieving the data again.
I am using this code, but it is not returning anything
function getData(){
var data = [];
const query = datastore.createQuery('ParticleEvent').order('created');
datastore.runQuery(query).then(results => {
const event = results[0];
console.log(results);
event.forEach(data => data.push(data.data));
});
console.log(data)
}
But each time it is returning empty specifically returning this :
[ [], { moreResults: 'NO_MORE_RESULTS', endCursor: 'CgA=' } ]
, and I can't figure out why because I have multiple entities saved in this Datastore.
Thanks
In the tutorial.js from the repo mentioned in the tutorial I see the ParticleEvent entities are created using this data:
var obj = {
gc_pub_sub_id: message.id,
device_id: message.attributes.device_id,
event: message.attributes.event,
data: message.data,
published_at: message.attributes.published_at
}
This means the entities don't have a created property. I suspect that ordering the query by such property name is the reason for which the query doesn't return results. From Datastore Queries (emphasis mine):
The results include all entities that have at least one value for
every property named in the filters and sort orders, and whose
property values meet all the specified filter criteria.
I'd try ordering the query by published_at instead, that appears to be the property with a meaning closest to created.
I am running an iOS app where I display a list of users that are currently online.
I have an API endpoint where I return 10 (or N) users randomly, so that you can keep scrolling and always see new users. Therefore I want to make sure I dont return a user that I already returned before.
I cannot use a cursor or a normal pagination as the users have to be returned randomly.
I tried 2 things, but I am sure there is a better way:
At first what I did was sending in the parameters of the request the IDs of the user that were already seen.
ex:
But if the user keeps scrolling and has gone through 200 profiles then the list is long and it doesnt look clean.
Then, in the database, I tried adding a field to each users "online_profiles_already_sent" where i would store an array of the IDs that were already sent to the user (I am using MongoDB)
I can't figure out how to do it in a better/cleaner way
EDIT:
I found a way to do it with MySQL, using RAND(seed)
but I can't figure out if there is a way to do the same thing with Mongo
PHP MySQL pagination with random ordering
Thank you :)
I think the only way that you will be able to guarentee that users see unique users every time is to store the list of users that have already been seen. Even in the RAND example that you linked to, there is a possibility of intersection with a previous user list because RAND won't necessarily exclude previously returned users.
Random Sampling
If you do want to go with random sampling, consider Random record from MongoDB which suggests using an an Aggregation and the $sample operator. The implementation would look something like this:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
const randomDocs = await collection
.aggregate([{
$sample: {
size: 5
}
}])
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
});
randomDocs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
client.close();
}());
Cache of Previous Users
If you go with maintaining a list of previously viewed users, you could write an implementation using the $nin filter and store the _id of previously viewed users.
Here is an example using a weather database that I have returning entries 5 at a time until all have been printed:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
let previousEntries = [], // Track ids of things we have seen
empty = false;
while (!empty) {
const findFilter = {};
if (previousEntries.length) {
findFilter._id = {
$nin: previousEntries
}
}
// Get items 5 at a time
const docs = await collection
.find(findFilter, {
limit: 5,
projection: {
main: 1
}
})
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
})
.toArray();
// Keep track of already seen items
previousEntries = previousEntries.concat(docs.map(doc => doc.id));
// Are we still getting items?
console.log(docs.length);
empty = !docs.length;
// Print out the docs
docs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
}
client.close();
}());
I have encountered the same issue and can suggest an alternate solution.
TL;DR: Grab all Object ID of the collections on first landing, randomized using NodeJS and used it later on.
Disadvantage: slow first landing if have million of records
Advantage: subsequent execution is probably quicker than the other solution
Let's get to the detail explain :)
For better explain, I will make the following assumption
Assumption:
Assume programming language used NodeJS
Solution works for other programming language as well
Assume you have 4 total objects in yor collections
Assume pagination limit is 2
Steps:
On first execution:
Grab all Object Ids
Note: I do have considered performance, this execution takes spit seconds for 10,000 size collections. If you are solving a million record issue then maybe used some form of partition logic first / used the other solution listed
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id; });
OR
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id.valueOf(); });
Result:
ObjectId("FirstObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ThirdObjectID"),
ObjectId("ForthObjectID"),
Randomized the array retrive using NodeJS
Result:
ObjectId("ThirdObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ForthObjectID"),
ObjectId("FirstObjectID"),
Stored this randomized array:
If this is a Server side script that randomized pagination for each user, consider storing in Cookie / Session
I suggest Cookie (with timeout expired linked to browser close) for scaling purpose
On each retrieval:
Retrieve the stored array
Grab the pagination item, (e.g. first 2 items)
Find the objects for those item using find $in
.
db.getCollection('my_collection')
.find({"_id" : {"$in" : [ObjectId("ThirdObjectID"), ObjectId("SecondObjectID")]}});
Using NodeJS, sort the retrieved object based on the retrived pagination item
There you go! A randomized MongoDB query for pagination :)
Is there a (proper) way to use the $oroperator in a URL query string to realize a client customizable query which does not only consist of and combined fields against my node.js powered RESTful API?
Let's say I have objects like
[{
A: 1,
B: 2,
C: 42
}, {
A: 3,
B: 1,
C: 42
}]
How do I query the MongoDB using mongoose to get both of them if searched in multiple fields? Something like $and: [{C:42}, $or: [{A:1}, {B:1}]] as URL parameters?
I tried GET /api/orders?$or=[{A:1},{B:1}]], but that results in a query like orders.find({ '$or': '[{A:1},{B:1}]]' }}), where the array is a string. The MongoDB driver then complains about $or needs an array.
All libraries like mongo-querystring have some complex operators for $gtand so on, but not for a simple OR of parameters. What am I missing here?
My goal is to build a convience search field, where the user can enter a simple string which then in turn is searched in multiple fields, returning all documents where at least one field (or a substring of that field) matches. So the frontend should decide in which fields and how the server should search, leading to a dynamic set of AND/OR combined field/value pairs.
Thanks in advance,
Ly
One option is to use the qs library which is a querystring parser with nested object support.
const qs = require('qs');
const query = {$or:[{A:1},{B:1}]};
let stringQuery = qs.stringify(query);
console.log(stringQuery); // => %24or%5B0%5D%5BA%5D=1&%24or%5B1%5D%5BB%5D=1
console.log(qs.parse(stringQuery)); // => { '$or': [ { A: '1' }, { B: '1' } ] }