I've been experimenting with Node.js's query parser and I am able to get it to parse some really complex queries into pretty deeply nested objects and arrays. Some examples:
Query Strings:
1) '?$or=foo&$or=bar'
2) '?$or[foo]=bar'
3) '?$or[0][foo]=bar&or[1][bar]=baz'
4)'?$or[0][foo]=bar&or[1][bar][]=baz&or[1][bar][]=bing&or[1][bar][]=bang'
Node parses them into:
1) {
$or: ['foo', 'bar']
}
2) {
$or: {
foo: 'bar'
}
}
3) {
$or: [
{ foo: 'bar' },
{ bar: 'baz' }
]
}
4) {
$or: [
{ foo: 'bar' },
{ bar: ['baz', 'bing', 'bang'] }
]
}
This is cool, but I can't find any docs about why this works. I've seen a couple links to this document, and it has this to say:
A host identified by an Internet Protocol literal address, version 6 [RFC3513] or later, is distinguished by enclosing the IP literal within square brackets ("[" and "]"). This is the only place where square bracket characters are allowed in the URI syntax.
Sounds like I'm not safe to use this syntax outside of Node at the very least, but why did Node decide it should work this way, and did they document this anywhere?
After some more digging I discovered in the express.js 4.x docs that express's query parser is based on the qs parser. I'm running express 3.x, whose documentation doesn't seem to mention qs or any advanced parsing capabilities, yet all of the qs documentation still seems to apply.
Related
I have documents in a MongoDB as below -
[
{
"_id": "17tegruebfjt73efdci342132",
"name": "Test User1",
"obj": "health=8,type=warrior",
},
{
"_id": "wefewfefh32j3h42kvci342132",
"name": "Test User2",
"obj": "health=6,type=magician",
}
.
.
]
I want to run a query say health>6 and it should return the "Test User1" entry. The obj key is indexed as a text field so I can do {$text:{$search:"health=8"}} to get an exact match but I am trying to incorporate mathematical operators into the search.
I am aware of the $gt and $lt operators, however, it cannot be used in this case as health is not a key of the document. The easiest way out is to make health a key of the document for sure, but I cannot change the document structure due to certain constraints.
Is there anyway this can be achieved? I am aware that mongo supports running javascript code, not sure if that can help in this case.
I don't think it's possible in $text search index, but you can transform your object conditions to an array of objects using an aggregation query,
$split to split obj by "," and it will return an array
$map to iterate loop of the above split result array
$split to split current condition by "=" and it will return an array
$let to declare the variable cond to store the result of the above split result
$first to return the first element from the above split result in k as a key of condition
$last to return the last element from the above split result in v as a value of the condition
now we have ready an array of objects of string conditions:
"objTransform": [
{ "k": "health", "v": "9" },
{ "k": "type", "v": "warrior" }
]
$match condition for key and value to match in the same object using $elemMatch
$unset to remove transform array objTransform, because it's not needed
db.collection.aggregate([
{
$addFields: {
objTransform: {
$map: {
input: { $split: ["$obj", ","] },
in: {
$let: {
vars: {
cond: { $split: ["$$this", "="] }
},
in: {
k: { $first: "$$cond" },
v: { $last: "$$cond" }
}
}
}
}
}
}
},
{
$match: {
objTransform: {
$elemMatch: {
k: "health",
v: { $gt: "8" }
}
}
}
},
{ $unset: "objTransform" }
])
Playground
The second upgraded version of the above aggregation query to do less operation in condition transformation if it's possible to manage in your client-side,
$split to split obj by "," and it will return an array
$map to iterate loop of the above split result array
$split to split current condition by "=" and it will return an array
now we have ready a nested array of string conditions:
"objTransform": [
["type", "warrior"],
["health", "9"]
]
$match condition for key and value to match in the array element using $elemMatch, "0" to match the first position of the array and "1" to match the second position of the array
$unset to remove transform array objTransform, because it's not needed
db.collection.aggregate([
{
$addFields: {
objTransform: {
$map: {
input: { $split: ["$obj", ","] },
in: { $split: ["$$this", "="] }
}
}
}
},
{
$match: {
objTransform: {
$elemMatch: {
"0": "health",
"1": { $gt: "8" }
}
}
}
},
{ $unset: "objTransform" }
])
Playground
Using JavaScript is one way of doing what you want. Below is a find that uses the index on obj by finding documents that have health= text followed by an integer (if you want, you can anchor that with ^ in the regex).
It then uses a JavaScript function to parse out the actual integer after substringing your way past the health= part, doing a parseInt to get the int, and then the comparison operator/value you mentioned in the question.
db.collection.find({
// use the index on obj to potentially speed up the query
"obj":/health=\d+/,
// now apply a function to narrow down and do the math
$where: function() {
var i = this.obj.indexOf("health=") + 7;
var s = this.obj.substring(i);
var m = s.match(/\d+/);
if (m)
return parseInt(m[0]) > 6;
return false;
}
})
You can of course tweak it to your heart's content to use other operators.
NOTE: I'm using the JavaScript regex capability, which may not be supported by MongoDB. I used Mongo-Shell r4.2.6 where it is supported. If that's the case, in the JavaScript, you will have to extract the integer out a different way.
I provided a Mongo Playground to try it out in if you want to tweak it, but you'll get
Invalid query:
Line 3: Javascript regex are not supported. Use "$regex" instead
until you change it to account for the regex issue noted above. Still, if you're using the latest and greatest, this shouldn't be a limitation.
Performance
Disclaimer: This analysis is not rigorous.
I ran two queries against a small collection (a bigger one could possibly have resulted in different results) with Explain Plan in MongoDB Compass. The first query is the one above; the second is the same query, but with the obj filter removed.
and
As you can see the plans are different. The number of documents examined is fewer for the first query, and the first query uses the index.
The execution times are meaningless because the collection is small. The results do seem to square with the documentation, but the documentation seems a little at odds with itself. Here are two excerpts
Use the $where operator to pass either a string containing a JavaScript expression or a full JavaScript function to the query system. The $where provides greater flexibility, but requires that the database processes the JavaScript expression or function for each document in the collection.
and
Using normal non-$where query statements provides the following performance advantages:
MongoDB will evaluate non-$where components of query before $where statements. If the non-$where statements match no documents, MongoDB will not perform any query evaluation using $where.
The non-$where query statements may use an index.
I'm not totally sure what to make of this, TBH. As a general solution it might be useful because it seems you could generate queries that can handle all of your operators.
Good morning.
I'm quite new to NodeJS / sequelize world and I'm currently facing a problem while trying to display a dashboard on screen.
This dashboard has three filters: two dates (period), client name, and employee name. The user can select none, one, two, or all the filters and my database needs to work accordingly.
That being said, my problem is with Sequelize because I don't know how to treat this problem of parameters not being "always" there.
I've seen this question:
Sequelize optional where clause parameters?
but this answer doesn't work anymore. I also tried another way of building the where clause, but I failed on it as well (mainly due to sequelize operators).
The last thing I tried was to make a single query with all parameters included but try to find some value (or flag) that would make sequelize ignore the parameter, for the case when the parameter was no there*, but it looks like Sequelize doesn't have anything like that.
* I've read a question here that has an answer saying that {} would do the trick but I tried that as well but didn't work.
In summary: I need to make a query that can "change" over time, for example:
Foo.findAll({
where: {
id : 1,
}
});
Foo.findAll({
where: {
id {
[Op.in] : [1,2,3,4,5]
},
name: "palmeiira",
}
});
Do you know a way of doing it without the need of using a lot if / switch statements?
I'm currently using Sequelize v. 5.5.1.
Update
I tried doing as suggested by #Anatoly and created a function to build the parameters. It was something like that. (I tried a "smaller" version just to test)
async function test() {
const where = {};
where[Op.and] = [];
where[Op.eq].push({
id: {
[Op.in]: [1,2,3]
}
});
return where;
}
I setted the return value to a const:
const query = await test()
And tried console.log(query)
The result was: { [Symbol(and)]: [ { id: [Object] } ] }, which made me believe that the problem was parsing the Op part so i tried using 'Op.and' and 'Op.in' to avoid that and it solved this problem, but led to another on sequelize that said Invalid value
Do you have any idea where is my error ?
P.S.: #Anatoly very nice idea you gave me on original answer. Thank you very much.
If these three conditions should work together then you can use Op.and with an array of conditions:
const where = {}
if (datesFilter || clientNameFilter || employeenameFilter) {
where[Op.and] = []
if (datesFilter) {
where[Op.and].push({
dateField: {
[Op.between]: [datesFilter.start, datesFilter.finish]
}
})
}
if (clientNameFilter) {
where[Op.and].push({
name: {
[Op.iLike]: `%${clientNameFilter.value}%`
}
})
}
if (employeenameFilter) {
where[Op.and].push({
employeeName: {
[Op.iLike]: `%${employeenameFilter.value}%`
}
})
}
}
const dashboardItems = await DashboardItem.findAll({ where }, {
// some options here
})
If the conditions should work as alternatives then just replace Op.and with Op.or
I create an index using the below function:
function (doc) {
if(doc.type === 'Property') {
if(doc.Beds_Max) {
try {
index("Beds_Max", parseInt(doc.Beds_Max));
}
catch(err) {
//ooopss
}
}
if(doc.YearBuilt) {
try {
index("YearBuilt", parseInt(doc.YearBuilt));
}
catch(err) {
//ooopss
}
}
}
}
using the cloudant Design Documents -> New Search Index and after the index is built I can issue queries like
"YearBuilt": [2010 TO Infinity]
But if I try to query the same index using Cloudant query I see weird behavior. If I go to Cloudant Dashboard -> Query and pass something like
{"limit": 5,
"selector": {
"_id": {
"$gt": null
},
"Beds_Max": {"$gte": 7}
},
fields: ["_id"]}
I see huge spike in data transmission, it keeps on receiving huge amounts of data even for the most unusual queries which are only supposed to return no more than 1 or 2 results and then hangs my computer so that most probably is not right. When I use Pouchdb-find npm module which has support for Cloudant 2.0 Query and issue the same selector as above I see inconsistent behavior, e.g. sometimes it returns 0 rows and sometimes it gives a ETIMEOUTERROR. If I change the index and exclude parseInt I can query using the same Pouchdb-find and even Cloudant Dashboard-> Query and get the results but in that case I lose the ability to use inequality operators which is a no go for me.
I'm open to work-arounds and even altogether different features to achieve the desired result.
The documentation for node module dbus-native is weak, and the none of the many examples apply to my use case, which seems like it ought to be simple.
My problem is trying to make a simple method call into the "connman" network manager. I can make a request (invoking GetProperties method) just fine, and get a complex structure back and generally figure out how to get data out of it. But when I try to send something back with SetProperty, I just can't figure out how to get my data into the form it wants.
There's not really any documentation about how to translate to and from DBus's type system and node's. I gathered from source that variant types are arrays with signature and value, e.g., [ 's', 'string ], but no matter how I try to massage the data I'm trying to send to SetProperty, I get an "Invalid struct data" error.
The SetProperty method I'm trying to call has an argument signature "sv"--name, value. The particular property I'm trying to set has a value with is itself (as far as I can determine--there's precious little documentation on that side as well) an "a{sv}". that is,property is a hash of property names and values. My initial attempt as calling this function was:
var sysbus = require('dbus-native').systemBus();
sysbus.invoke({
path: '/net/connman/service/ethernet_1cba8cfa0e57_cable',
destination: 'net.connman',
'interface': 'net.connman.Service',
member: 'SetPropertry',
signature: 'sv',
body: [
'IPv4.Configuration', [
'a{sv}',
{ 'Method': [ 's', 'dhcp' ] }
]
],
type: dbus.messageType.methodCall
}, function (err, res) {
// etc...
});
This gives me the "Invalid struct data" error. I've tried man other ways to wrap the data in arrays, hashes, etc., and just can't seem to find the right answer. I'm trying to emulate this working Python code:
import dbus
bus = dbus.SystemBus()
service = dbus.Interface(bus.get_object("net.connman",
"/net/connman/service/ethernet_1cba8cfa0e57_cable",
"net.connman.Service");
conf = { "Method": make_string_variant("dhcp") }
service.SetProperty("IPv4.Configuration", conf);
Any ideas?
I recently started using dbus-native for connman and also had problems with all the nested arrays. I'll be improving the marshaling/unmarshaling in the fork of the project at https://github.com/Digisoft-tv/node-dbus - you might want to have a look. Hopefully my changes will be accepted up-stream.
Anyway, to get it working now, the format of the parameters you need to pass in is as below:
mgr.CreateSession([ [ 'key', [ 's', 'value' ] ], [ 'key2', [ 'i', 2 ] ] ], '/object/path', function(error, response) {
if (error) {
return console.error('SetProperty error:', error);
}
console.info('SetProperty response', response);
});
The outermost array is the "object" (ARRAY of dict entries).
Each nested array holds a key-value pair (DICT ENTRY). key at index 0 and value at index 1.
The values is a variant, which is always encoded as an array, with "signature" at index 0 and the actual value at index 1.
Hope it helps.
The correct way to encode 'a{sv}' is [ ['string', ['signature', 'value']], ... ]
Arrays ( a ) are encoded as normal JS arrays as well as structs / hashes ( () / {} ). I'll probably add shortcut to allow JS object to be used in place of a{sv} but at the moment input has to be quite verbose.
var sysbus = require('dbus-native').systemBus();
sysbus.invoke({
path: '/net/connman/service/ethernet_1cba8cfa0e57_cable',
destination: 'net.connman',
'interface': 'net.connman.Service',
member: 'SetPropertry',
signature: 'sv',
body: [
'IPv4.Configuration', [
'a{sv}',
[ // a
[ // {
'Method', [ 's', 'dhcp' ] // sv
]
]
]
],
type: dbus.messageType.methodCall
}, function (err, res) {
// etc...
});
I've got the following code, but it doesn't seem to work:
var post_req = {
array: [
[ {
param1: 'something',
param2: 123
} ],
[ ],
[ ],
[ {
param2: 'something',
param4: 1234,
param1: 'hello'
} ]
]
};
var data_send = querystring.stringify(post_req);
var request = client.request('POST', '/', headers);
request.end(data_send);
and
if( req.method == 'POST' ) {
req.addListener('data', function(chunk)
{
POST = querystring.parse(chunk);
console.log(POST);
}
}
I end up with 5 sub-arrays, corresponding to the 5 parameters in the objects but with extra '][' characters in their names:
{ array:
[ { '][param1': 'something' }
, { '][param2': '123' }
, { '][param2': 'something' }
, { '][param4': '1234' }
, { '][param1': 'hello' }
]
}
There is a new node package that fixes this: "npm install qs".
https://github.com/ljharb/qs
"query string parser for node supporting nesting, as it was removed from 0.3.x, so this library provides the previous and commonly desired behaviour (and twice as fast)"
If anyone can tell me why it was removed from 0.3.x, I will give you an upvote for your comment. (I want my confidence in Node.js restored.)
To confirm my comment above, node's querystring.stringify function won't handle nested arrays (at the time of writing).
You can see the source of stringify at https://github.com/ry/node/blob/master/lib/querystring.js
Note that it handles one level of arrays but it doesn't recurse. When it finds an array it uses stringifyPrimitive to encode the array's values. You can see that stringifyPrimitive doesn't handle arrays, only number, boolean and string.
As I suggested in my comment, given that you're using a POST request a better idea would be to use JSON encoding for your complex data structure.
Or use https://github.com/visionmedia/node-querystring as suggested by #FriendlyDev
Don't use querystring.parse for recreating a JSON object sent as string. Use instead JSON.parse. And use JSON.stringify instead of querystring.stringify
querystring is useful when you send parameter encoded in the url or when you post a form. But there is no point in using it if you send just one JSON object with all your data.
In your scenario I would dismiss the querystring library and use JSON library, which is already included. It would be cleaner and faster.
http://www.json.org/js.html