JSON.parse fails on ObjectId - node.js

I am trying to convert a string for use in mongodb but fails.
let pipeline = JSON.parse('[{"$match": {"_id": ObjectId("5b5637acbd3e9c2068ef80c3")}]');
// results in "SyntaxError: Unexpected token O in JSON at position 20"s
let pipeline = JSON.parse('[{"$match": {"_id": "5b5637acbd3e9c2068ef80c3"}]');
let response = await db.collect('<collection_name>').aggregate(pipeline).toArray();
// returns [] parse works but mongodb doesn't return any rows!
// This works but its not the solution I am looking for.
let pipeline = [{"$match": {"_id": ObjectId("5b5637acbd3e9c2068ef80c3")}];
let response = await db.collect('<collection_name>').aggregate(pipeline).toArray();
I tried using the BSON type but had no luck.
My current work around is to remove the ObjectId() from the string and use a Reviver function with JSON.parse
const ObjectId = require('mongodb').ObjectID;
let convertObjectId = function (key,value){
if (typeof value === 'string' && value.match(/^[0-9a-fA-F]{24}$/)){
return ObjectId(value);
} else {
return value;
};
}
let pipeline = JSON.parse('[{"$match": {"_id": "5b5637acbd3e9c2068ef80c3"}]',convertObjectId);
let response = await db.collect('<collection_name>').aggregate(pipeline).toArray();
// returns one record.

Unfortunately, [{"$match": {"_id": ObjectId("5b5637acbd3e9c2068ef80c3")}] is not valid JSON.
The value of a property in JSON can only be an object (ex.: {}), an array (ex.: []), a string (ex.: "abc"), a number (ex.: 1), a boolean (ex.: true), or null. See an example of these values here: https://en.wikipedia.org/wiki/JSON#Example.
What you could do is add ObjectId() manually after parsing the JSON. This would mean that the value of _id would be a string first, which is valid JSON.
Then, you can loop through your parsed JSON to add ObjectId (see reference here: https://mongodb.github.io/node-mongodb-native/api-bson-generated/objectid.html):
const ObjectId = require('mongodb').ObjectID;
const pipeline = JSON.parse('[{"$match": {"_id": "5b5637acbd3e9c2068ef80c3"}]');
const pipelineWithObjectId = pipeline.map(query => ({
$match: {
...query.$match,
_id: ObjectId(query.$match._id)
}
});
const response = await db.collect('<collection_name>').aggregate(pipelineWithObjectId).toArray();
This should work with the example you provided but there are multiple caveats:
Parsing a query like that could be a vulnerability if the string contains user input that has not been sanitized: https://blog.websecurify.com/2014/08/hacking-nodejs-and-mongodb.html.
This particular code snippet would only work for queries with $match, which means that this code is not very scalable.
This code is not elegant.
All these reasons, for what they are worth, make me think that you would be better off using an object rather than a string for your queries.

Related

NodeJS Express create Mongoose query from string

I have this string:
var filter = '{stamps:{$gte: 2020-11-06 06:42:25.000+01:00, $lte: 2020-11-06 09:52:25.000+01:00}'
Somehow I have to query mongoDB collection with this string.
Message.find(filter, function(err, messages){
if(err){
console.log(err);
}else{
//do something here
}
})
Of course I am getting this error:
ObjectParameterError: Parameter "filter" to find() must be an object, got {stamps:{$gte: 2020-11-06 06:42:25.000+01:00, $lte: 2020-11-06 09:52:25.000+01:00}
is there some way to cast this string to object that will be acceptable to mongoDB?
You could either change the way you construct your query, to pass it as JSON an then use JSON.parse() / JSON.stringify()
// Note the necessary double quotes around the keys
const jsonFilter = '{ "stamps": { "$gte": "2020-11-06 06:42:25.000+01:00", "$lte": "2020-11-06 09:52:25.000+01:00" } }'
const query = JSON.parse(jsonFilter)
Message.find(query, callback)
Or you could introduce another dependency like mongodb-query-parser that can parse the string for you.
const parser = require('mongodb-query-parser')
// Note it still needs double quotes around the dates to run properly
const filter = '{stamps:{$gte: "2020-11-06 06:42:25.000+01:00", $lte: "2020-11-06 09:52:25.000+01:00" } }'
const query = parser(filter)
Message.find(query, callback)
https://runkit.com/5fbd473b95d0a9001a2359b3/5fbd473b98492c001a8bba06
If you can't include the quotes around the values in the string, you can use a regular expression to add them. But keep in mind that this regex is tailored to match this exact date format. If the format can change or you want to also match other data types, you should try to include them in your filter string from the start.
const parser = require("mongodb-query-parser")
const filter = '{stamps:{$gte: 2020-11-06 06:42:25.000+01:00, $lte: 2020-11-06 09:52:25.000+01:00} }'
const filterWithQuotedDates = filter.replace(/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}(\+\d{2}:\d{2})?)/g, '"$1"')
parser(filterWithQuotedDates)
Message.find(query, callback)
https://runkit.com/5fbd494bcd812c0019b491fb/5fbd4951ebe43f001a5b590a
It should be noted that a common use-case is to pass a MongoDB query via URL params and that there are special packages for that:
https://github.com/Turistforeningen/node-mongo-querystring
https://github.com/fox1t/qs-to-mongo

How to search data in mongodb with dynamic fields using mongoose?

I've a node.js api in which user sends the required fields as an array to be fetched from the mongodb database. I need to find the data of that fields using Find query. I've written forEach statement to loop through that array and got the array elements. But when I try to get the results by inserting the array elements in the query, it doesn't giving the required results. Could any one please help me in resolving the issue by seeing the code below?
templateLevelGraphData: async function(tid,payload){
let err, templateData, respData = [], test, currentValue;
[err,templateData] = await to(Template.findById(tid));
var templateId = templateData.templateId;
payload.variables.forEach(async data=>{
console.log(data); //data has the array elements like variables=["humidity"]
[err, currentValue] = await to(mongoose.connection.db.collection(templateId).find({},{data:1}).sort({"entryDayTime":-1}).limit(1).toArray());
console.log(currentValue);
});
return "success";
}
The expected output is,
[ { humidity: 36 } ]
But I'm getting only _id like,
[ { _id: 5dce3a2df89ab63ee4d95495 } ]
I think data is not applying in the query. But I'm printing the data in the console where it's giving the correct results by displaying the array elements like, humidity. What I need to do to make it work?
When you are passing {data: 1} you are passing an array where is expecting name of column.
You have to create an object where the keys are going to be the elements of the array and set them to 1.
const projection = data.reduce((a,b) => (a[b]=1, a), {});
[...] .find({}, projection) [...]
Actually I got the solution.
for(let i=0;i<payload.variables.length;i++){
var test = '{"'+ payload.variables[i] +'":1,"_id":0}';
var query = JSON.parse(test);
[err, currentValue] = await to(mongoose.connection.db.collection(templateId).find({"deviceId":deviceId},query).sort({"entryDayTime":-1}).limit(1).toArray());
console.log(currentValue); //It's giving the solution
}

How to filter results by multiple query parameters if I don't know beforehand how many query strings I may receive from client side?

I want to send in response some data according to searching by query parameters (using .find function of mongoose) from the client side. What do I need to do is a search according to the parameters received?
What I mean is :
I may receive
localhost:5000/admin/customers?customer_id=1&customer_email=abc#gmail.com
I could have used this code to send results according to this query :
Customer.find({
customer_id = req.query.customer_id,
customer_email = req.query.customer_email,
}, (err,docs)=> {
res.json(docs);
})
or
just
localhost:5000/admin/customers?customer_id=1
I could have used this code to send results according to this query :
Customer.find({
customer_id = req.query.customer_id
}, (err,docs)=> {
res.json(docs);
})
or
may be
localhost:5000/admin/customers?no_of_items_purchased=15
I could have used this code to send results according to this query :
Customer.find({
no_of_items_purchased = req.query.no_of_items_purchased
}, (err,docs)=> {
res.json(docs);
})
But what I want is to use .find function on anything received from query params. Like a general code to achieve this.
PS: Also please help with : "How to filter req.query so it only contains fields that are defined in your schema ?"
You can create a query variable to keep the field that you want to filter.
Suppose that your Customer model structure is:
{
customer_id: ...,
customer_name: ...,
customer_email: ...,
no_of_items_purchased: ...
}
Then your code will be:
let {customer_id, customer_name, customer_email, no_of_items_purchased} = req.query;
let query = {};
if (customer_id != null) query.customer_id = customer_id;
if (customer_name != null) query.customer_name = customer_name;
if (customer_email != null) query.customer_email = customer_email;
if (no_of_items_purchased != null) query.no_of_items_purchased = no_of_items_purchased;
let result = await Customer.find(query);
Just pass request.query as a parameter directly on find method:
Customer.find(request.query)

Stringifying JSON data loses object I'm trying to stringify

I'm new to NodeJS and I'm trying to learn it a bit better by writing a Discord bot. But I'm having an issue with stringifying an object to JSON. I think it's having an issue with the array I'm setting, but I'm not sure how else I would do this. If I'm not supposed to set an array and using my guildMembers example below, how else should I insert this data into my JSON file?
I've looked through a few examples here on StackOverflow and found this particular article: JSON Stringify Removing Data From Object. However, it's not clear to me given what I'm trying to achieve.
var o = {};
var guildKey = guild.id;
o[guildKey] = [];
o[guildKey]['guildMembers'] = {};
var guildMembers = []
guild.members.forEach(function(guildMember, guildMemberId) {
if (!guildMember.user.bot){
var memberData = {
id: guildMember.id,
data: {
userName: guildMember.user.username,
nickName: guildMember.nickname,
displayName: guildMember.displayName,
joinedAt: guildMember.joinedAt
}
}
guildMembers.push(memberData);
};
});
o[guildKey]['guildMembers'] = guildMembers;
json = JSON.stringify(o);
I am expecting the data to show the guildMembers object with the array under the guildKey object. However, the JSON shows only the guildKey object with an empty array:
{"guildKey":[]}
You make guildKey an array and then try to use it as an object ...
Solution is:
o[guildKey] = {};
Just like in the mentioned post.
It is because you are assigning a key-value pair to an array with
o[guildKey]['guildMembers'] = { }; <= o[guildKey]
When iterating over an array to display the contents, JavaScript does not access non-integer keys. You should store o[guildKey] as an object instead.

Multiple types in query string in nodejs

I am creating a get api in nodejs.I am requesting the following url
http://localhost:8080/api?id=20&condition1=true&arr=[{prop1:1}]&obj={a:1,b:2}
And I am getting the request query object as follows-
req.query = {
arr:"[{prop1:1}]",
condition1:"true",
id:"20",
obj:"{a:1,b:2}"
}
I want to convert the query object keys to appropriate types.My query object should be converted to
req.query = {
arr:[{prop1:1}], // Array
condition1:true, // Boolean
id:20, // Number
obj: {a:1,b:2} //Object
}
req.query object is dynamic, it can contain any number of objects, array, boolean , number or strings. Is there any way to do it?
This functionality doesn't come out of the box with express and query parameters.
The problem is that in order for the query string parser to know if "true" is actual boolean true or the string "true" it needs some sort of Schema for the query object to help parsing the string.
Option A
What I can recommend is using Joi.
In your case it will look like :
const Joi = require( "joi" );
const querySchema = {
arr: Joi.array(),
condition1: Joi.boolean(),
id: Joi.number(),
obj: {
a: Joi.number(),
b: Joi.number()
}
}
Having this schema you can attach it to your express method and use Joi.validate To validate it.
function getFoo( req, res, next ) {
const query = req.query; // query is { condition1: "true" // string, ... }
Joi.validate( query, querySchema, ( err, values ) => {
values.condition1 === true // converted to boolean
} );
}
Option B
Another way of having properly typed GET requests would be to trick the query parameters and just provide a stringified JSON.
GET localhost/foo?data='{"foo":true,"bar":1}'
This will give you the possibility to just parse the request query
function getFoo( req, res, next ) {
const data = JSON.parse( req.query.data )
data.foo // boolean
data.bar // number
}

Resources