I'm developing a node application which has an endpoint where it filters stories.
I accept multiples query params up to five. I don't know if this is too much query params but I need these params to filter in the data base.
Example:
http://localhost:3000/api/search?location=wherever&duration=123
I was wondering if there is any way better to do it like:
http://localhost:3000/api/search?filter=location:wherever,duration:123
or even I could send a stringify object from the url but it seems an ugly solution for me.
The second way I put here looks cleaner for me but I'm not sure how to handle the parameters.
In the first option I can pass req.query to my service and handle the object however in the other way handle the filter value would be harder.
What do you think? How do you manage this case to send a filter to your back?
Thanks in advance and my apologies if it was already answered, I didn't find a similar post.
What is the maximum query params url I should use?
There is no limit to the number of query parameters and it doesn't cause any problem to have lots of them.
I accept multiples query params up to five. I don't know if this is too much query params but I need these params to filter in the data base.
Five is low, not high - you're absolutely fine.
The practical limit to the total URL length is around 2000 characters so you just want to make sure your URLs stay comfortably below that and not worry about the count of separate query parameters.
I would only consider an alternative way to send the query if the total URL length was getting too long or could be too long. Aside from that, you can have as many query parameters as you want. It's not really something a human consumes directly, it's just something your code generates and parses so the quantity really doesn't matter.
That said, if you had 100 separate query parameters that would all be used at once, I'd wonder what you were doing and if there was a better way to accomplish it or express the query, but it wouldn't be a technical problem or limitation.
The structural alternatives are using POST instead of a GET and sending data as JSON or something like that. You can send as much data as you want that way, but a POST changes the usability of the URL since it's not something that can just be directly linked to or bookmarked or copied into a URL bar like a GET can.
Related
I am using mongodb and what I am trying to do is query a collection, and receive well filtered and ordered and projected result. Sadly the logic I want to implement is complex and even if we assume that it is possible to use db.collection.aggregate it will result in long, complex, hard to read aggregate descriptor, which I believe in most cases is unwanted.
So I was thinking - Mongodb understands javascript, therefor most likely I can pass a javascript function during the query itself, expecting that my mongo server will make the query, run the provided function passing the query result to it, then return the final result to me. Something like:
db.collection.find(myQuery, serverCallback).toArray(function(err, db) { ... });
Sadly it seems this is impossible. Investigating further I reached stored javascript and understood that I can define that serverCallback on the server instead of passing it. Which is good, but seems messy and wrong to me. So basically this is the reason, why i decided to ask here if someone with better mongodb experience can argument this approach.
[My understandings]
I beleive that not every case of filtering, aggregating, etc. can be achieved with db.collection.aggregate, which is pretty normal. For all the cases that need special way of filtering the query result, we have two options - to define stored javascript that we execute on the query result on the mongo server, or to fetch the information from the server and do the processing/filtering/etc. in the client.
If we choose to define stored javascript, it is very likely that we will define some project specific logic into the mongo server. I think that the project specifics should always belong to the project code instead of the database. That way we can version them with git and easily access them if we want to change them.
If we choose to apply the aggregation logic after the query we loose the capability to choose who will make the calculations - the server or the client. Which may be an opinion that we want to have.
[My question]
What is the reasoning behind not allowing serverCallback to be provided during the query? I believe that there must be reasons that I do not understand here.
[Edits]
First I want to say that I have resolve my problem and as it was way too complex to explain it easily. I will prefer to stick to something easier to explain and understand. I believe this example of MongoDB stored javascript provides great example so lets use it. Basically what i tried to ask above was is there a way to pass this sum function during db.collection.find (and why there isn't any). Something like this:
function sum(queryResultAsArray) {
//Do whatever we want with queryResultAsArray
//For the example we filter result rows with x + y == 6;
return queryResultAsArray.filter(function(row) {
return row.x + row.y == 6
});
}
db.test.find({}, queryResultAsArray);
And this to be equal to the examples:
db.test.find({$where: "sum(this.x, this.y) == 6"});
For reasoning on why would one can prefer passing function rather than stored javascript see the original post.
I have Express running in a Node.js server, which serves as a backen for my React frontend application.
The frontend application fetches data from the backend (which is stored in Mongo) through a REST call, and display this data in a table.
The amount of data is growing by the day, so I though I should look into reducing the abount of data transferred to the frontend application, so avoid unnecessary strain on the backend.
I'm not sure if this is the right way to approach this, but I've been thinking I would look into having the backen fetch a limited amount of entries, so that only these data will be displayed in the frontend table.
The problem arises with searching - when the user wants to search the data in the table, I'll need to be able to search through all entries, not just the data loaded into the table.
I guess one option would be to have the search function actually query the REST API, instead of searching the table itself.
If I'm on the right track, I guess I could implement REST API pagination, somewhere along the example found in https://refactoringfactory.wordpress.com/2012/09/08/pagination-in-node-js-and-express/. Other suggestions on how to implement pagination are welcome.
I'd very much like some input on the approach I described, and suggestions for smarter ways implement this.
EDIT: I changed the title somewhat to include "Infinite scroll pagination". This is what I'm looking to implement. At the moment I have a click on pages pagination setup, but would like to replace this for the infinite scroll pagination.
I've been thinking I would look into having the backen fetch a limited amount of entries, so that only these data will be displayed in the frontend table.
This is common practice in my experience. The term for it is "pagination." Have a look at this SO question regarding best practices for pagination in REST API's: API pagination best practices.
The problem arises with searching - when the user wants to search the data in the table, I'll need to be able to search through all entries, not just the data loaded into the table.
I guess one option would be to have the search function actually query the REST API, instead of searching the table itself.
Again, you got it. Doing small filters/searches on the client is fine for a limited number of entries, but if you need to only retrieve items matching search criteria in the first place, then adding that functionality to your REST API is the right choice.
Right, you should do
pagination: you might implement it by exposing 2 arguments in the rest endpoint for the listing
?p=<number>: page number, defaults to 1
?l=<number>: number of items per page / page length, defaults to a number maybe from 10 to 100
search: implement it by exposing 1 argument in the rest endpoint for the listing
/?q=<string>: you can define to be what you want, maybe a string that matches with one or multiple fields of the data
If you want to minimize the network traffic, you might also add one more parameter to explicitly select the fields you want to be returned, like this
/?f=<string>: string could be something like id,name,age, and so the api should return only those three fields per record.
All this parameters should be accepted by a list endpoint in your RESTful API
Example:
http://example.com/api/cars/?p=2&l=15&q=toyota&f=id,brand,model,color
I did a google search, but I could not find what I really need.
I need to query an API, which have the same route, but with different parameters.
Example:
router.get('/items/:query, function(){})
In this case, I would search for all items
router.get('/items/:id, function(){})
Here, I would look for a specific item
At the core of your issue is that you are trying to specify two different resources at the same location. If you design your API to adhere to restful principles you'll see why that's not a wise choice. Here are two good starting points:
https://en.wikipedia.org/wiki/Representational_state_transfer
http://www.restapitutorial.com/lessons/whatisrest.html
In restful api's the root resource represents the item collection:
/api/items
...and when you specify an id that indicates you want only one item:
/api/items/abc123
If you really still want to accomplish what you asked in your question you'll need to add a url parameter like /items/query?query=true or /items/abc123?detail=true but this will be confusing to 99% of web developers who ever look at your code.
Also I'm not sure if you really meant this, but when you pass a variable named "query" to the server that seems to indicate that you're going to send a SQL query (or similar data definition language) from the client into the server. This is a dangerous practice for several reasons - it's best to leave all of this type of code on your server.
Edit: if you really absolutely positively have to do it this way then maybe have a query parameter that says ?collection=true. This would at least be understood by other developers that might have to maintain the code in future. Also make sure you add comments to explain why you weren't able to implement rest so you're not leaving behind a bad reputation :)
The issue is that without additional pattern matching there isn't a way Express will be able to distinguish between /items/:query and /items/:id, they are the same route pattern, just with different aliases for the parameter.
Depending on how you intend to structure your query you may want to consider having the route /items and then use query string parameters or have a separate /items/search/:query endpoint.
I am experimenting with Dojo's dgrid (which is great!). I am using Nodejs/Mongoose on the server side.
I want to write a "log browser": I have a big mongodb table containing lots of log entries; using dgrid, I want to be able to 1) Filter by certain parameters 2) Paginate using dgrid's native pagination.
Hence the problem: dojo's JsonRest stores will send a request like this:
Accept:application/javascript, application/json
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
...
Host:localhost:3000
Range:items=0-24
Hence the problem: it will give a range (that's all it can do, really) and will display things on the client side according to what it receives from the server.
It's unrealistic to expect a cliend side JsonRest object to make requests other than "ranges". However, I am aware that skip/limit doesn't go very well with Mongoose:
What is the best way to do ajax pagination with MongoDb and Nodejs?
My idea was to render the dgrid, allowing the users to pick filters, and let them happily paginate through their logs. However, the fact that skip/limit are out of question, I am in a bit of a pickle...
Any pearls of wisdom, other than ditch dgrid altogether and implementing pagination on my own without using Dojo stores?
Merc.
Front-end
The filtering isn't as feature-full in dgrid as it is in the dojo EnhancedGrid filter plugin so you will probably need to implement that part yourself.
The good news is you get the paging simply by mixing-in "dgrid/OnDemandGrid" when you create your grid.
Back-end
The docs seem to indicate that your best bet for performance is to do some tricks with indices and query based on those to get your ranges.
You are probably already referencing these, but here they are;
http://mongoosejs.com/docs/api.html#query_Query-skip
http://docs.mongodb.org/manual/reference/method/cursor.skip/
Since log data is usually sequential and rarely modified, you could probably just use a monotonically increasing index for each row of log data and query using those to get the right offset into and count of the rows.
Is it possible to transform the returned data from a Find query in MongoDB?
As an example, I have a first and last field to store a user's first and last name. In certain queries, I wish to return the first name and last initial only (e.g. 'Joe Smith' returned as 'Joe S'). In MySQL a SUBSTRING() function could be used on the field in the SELECT statement.
Are there data transformations or string functions in Mongo like there are in SQL? If so can you please provide an example of usage. If not, is there a proposed method of transforming the data aside from looping through the returned object?
It is possible to do just about anything server-side with mongodb. The reason you will usually hear "no" is you sacrifice too much speed for it to make sense under ordinary circumstances. One of the main forces behind PyMongo, Mike Dirolf with 10gen, has a good blog post on using server-side javascript with pymongo here: http://dirolf.com/2010/04/05/stored-javascript-in-mongodb-and-pymongo.html. His example is for storing a javascript function to return the sum of two fields. But you could easily modify to return the first letter of your user name field. The gist would be something like:
db.system_js.first_letter = "function (x) { return x.charAt(0); }"
Understand first, though, that mongodb is made to be really good at retrieving your data, not really good at processing it. The recommendation (see for example 50 tips and tricks for mongodb developers from Kristina Chodorow by Oreilly) is to do what Andrew tersely alluded to doing above: make a first letter column and return that instead. Any processing can be more efficiently done in the application.
But if you feel that even querying for the fullname before returning fullname[0] from your 'view' is too much of a security risk, you don't need to do everything the fastest possible way. I'd avoided map-reduce in mongodb for awhile because of all the public concerns about speed. Then I ran my first map reduce and twiddled my thumbs for .1 seconds as it processed 80,000 10k documents. I realize in the scheme of things, that's tiny. But it illustrates that just because it's bad for a massive website to take a performance hit on some server side processing, doesn't mean it would matter to you. In my case, I imagine it would take me slightly longer to migrate to Hadoop than to just eat that .1 seconds every now and then. Good luck with your site
The question you should ask yourself is why you need that data. If you need it for display purposes, do that in your view code. If you need it for query purposes, then do as Andrew suggested, and store it as an extra field on the object. Mongo doesn't provide server-side transformations (usually, and where it does, you usually don't want to use them); the answer is usually to not treat your data as you would in a relational DB, but to use the more flexible nature of the data store to pre-bake your data into the formats that you're going to be using.
If you can provide more information on how this data should be used, then we might be able to answer a little more usefully.