I've been reading up on ways to protect against 'mongo injection' in my express backend. I typically use the package express-validator to sanitize and validate my inputs.
For example:
{"username": {"$ne": null}} //possible injection
.check('username').isString() //simple validation
Do you just verify the value is a string?
From what I understand the solution is to make sure "$" are not allowed or removed in the keys of JSON payloads passed to my endpoints. How do others protect against this? Is there a way to go through the whole POSTed payload and remove any key value pairs that contain a "$" in the key? Just curious if I'm missing something obvious.
Related
I'm currently writing an app that accesses google bigquery via their "#google-cloud/bigquery": "^2.0.6" library. In one of my queries I have a where clause where i need to pass a list of ids. If I use UNNEST like in their example and pass an array of strings, it works fine.
https://cloud.google.com/bigquery/docs/parameterized-queries
However, I have found that UNNEST can be really slow and just want to use IN on its own and pass in a string list of ids. No matter what format of string list I send, the query returns null results. I think this is because of the way they convert parameters in order to avoid sql injection. I have to use a parameter because I, myself want to avoid SQL injection attacks on my app. If i pass just one id it works fine, but if i pass a list it blows up so I figure it has something to do with formatting, but I know my format is correct in terms of what IN would normally expect i.e. IN ('', '')
Has anyone been able to just pass a param to IN and have it work? i.e. IN (#idParam)?
We declare params like this at the beginning of the script:
DECLARE var_country_ids ARRAY<INT64> DEFAULT [1,2,3];
and use like this:
WHERE if(var_country_ids is not null,p.country_id IN UNNEST(var_country_ids),true) AND ...
as you see we let NULL and array notation as well. We don't see issues with speed.
How can I make standard json keys schema for multiple API service.
What if on original response I want to map to specific keys?
Maybe adding or removing keys too?
I'd like to have same json keys output for all api services
for example: api service output:
{ "hello" : "word" }
but i want to response:
{"foo" : originalResponse.hello }
Thanks
I do not understand why the downvotes, the question seems legit to me.
In any case, you can achieve the use case by using the body modifier plugin we offer. Although it's not an official one, it should provide all you need to both add are move keys on a JSON payload based on any Javascript evaluable expression.
Cheers!
Using Hapi.JS Joi to validate inputs for an Express application. It's a boilerplate setup:
const Joi = require('joi');
const schema = Joi.object().keys({
username: Joi.string().alphanum().min(3).max(30).required(),
birthyear: Joi.number().integer().min(1900).max(2013),
}).with('username', 'birthyear');
app.use('/user/:id', function (req, res, next) {
Joi.validate({ username: 'abc', birthyear: 1994 }, schema, function
(err, value) {
if (err){
...
}
...
next()
}
});
})
QUESTION #1: What's the difference between validation and sanitization?
And should I sanitize inputs for an Express API?
It's for a mobile app, and not for a website, so I'm trying to understand if I should validate as well as sanitize.
QUESTION #2
How can I sanitize inputs with Joi or some other Express compatible library?
What's the difference between validation and sanitization?
Sanitization
Sanitizing Inputs means checking input before storing it in a database or using it for any other purpose to prevent malicious code injection.
A basic example would be SQL Injection which is to be taken in account if you want to store/verify data. Suppose you are testing login credentials submitted by user in your database. Your query might be something like
SELECT * FROM `users` WHERE `username`='$user' AND `pass`='$pass'
where $user and $pass are the username and password which user enters.
If you are not sanitizing user input and user enters something like this:
username -> admin' AND 1=1 OR 1='1
password -> pass
Your query would become:
SELECT * FROM `users` WHERE `username`='admin' AND 1=1 OR 1='1' AND `pass`='pass'
which on execution selects admin field and logs in user as admin.
But if you are sanitizing user input, your query would be:
SELECT * FROM `users` WHERE `username`='admin\' AND 1=1 OR 1=\'1' AND `pass`='pass'
which will not give the user access to any account until or unless username and password matches to a database entry.
Validation
Validation is the checking or verification of any data that comes, which helps verify the data has not been compromised or corrupted during transmission.
Like if you are taking mobile platform as an argument then you only want to allow Android or IOS as a value and other values are not valid.If some critical input is needed from the user that cannot be empty then checking it comes into validation.
But if the user gives ANDROID & IOS n input then sanitization will make that ANDROID & IOS. that will not allow the user to break the code and logic
should I sanitize inputs for an Express API?
Yes, you should always sanitize data as if you are exposing it as a rest API then the user can insert malicious data into the input of mobile app. It will be better to be ready for all the edge cases and user can do anything. (:wink:)
How can I sanitize inputs with Joi or some other Express compatible library?
With the Joi you can sanitize variable with addition options
validate(value, schema, {escapeHtml: true}, [callback])
Sanitising is for preventing malicious code
example for XSS sanitising <script>alert(1)</script>
is changed to <script>alert(1)</script>
so that it will be displayed on the browser and not executed
And Validation is for general checks like if an input is a valid email, phone number etc
example email validation,
length > 5, # should be present,. should be present after # etc
update to question 2
It is a really good practice to sanitise all the input from the user.
A great rule to remember
never to trust data from user.
What's the difference between validation and sanitization?
Validation is verifying that the data being submitted meets a rule or set of rules defined by the developer for a particular input field.
// checks that 22 is a number and must be >=99
Joi.validate(22, Joi.number().min(99));
Validation prevents unexpected or bad data entry.
Sanitization only cares about making sure the data being submitted doesn't contains any code. Like change all single quotation marks in a string to double quotation marks or change < to <
Sanitization prevents malicious code injection or execution.
Should I sanitize inputs for an Express API?
Yes you should.
I'm trying to understand if I should validate as well as sanitize.
Yes should validate as well as sanitize your data as combining these two techniques provides in-depth defense to your application. One more thing validation should always happens before sanitization.
How can I sanitize inputs with Joi or some other Express compatible library?
Joi is a validation library. It is perfect of validating data. But for sanitization if rather go with something like string.js for methods like escapeHTML() and module xss-filters for xss sanitization.
Here is below my code of route:-
app.get('/server/lead/get/:id?', leadCtrl.get);
app.get('/server/lead/filter/:filterQuery', leadCtrl.get);
As you see above i am using different route to access same controller method leadCtrl.get.
Now, i want something like route app.get('/server/lead/get/:id?:filter?', leadCtrl.get);. So, i can get params either req.params.id or req.params.filter but only one at a time.
What you asked in the question is not possible in the form that you describe it.
Now, i want something like route
app.get('/server/lead/get/:id?:filter?', leadCtrl.get);. So, i can get
params either req.params.id or req.params.filter but only one at a
time.
Your router would have no way to differentiate those two parameters. If it got a request to /server/lead/get/X then what is X? A filter or an ID?
Your options
You have few solutions here:
You can either keep using two routes like you did before.
You can use a common parameter for both cases as Robert explained in the comments.
Or you can use what seems to me the perfect solution for your use case - named query parameters - just use a route /server/lead/get and use query parameters to pass id and the filter.
Example URLs:
/server/lead/get?id=xxx
/server/lead/get?filterQuery=xxx
You will only have to make sure in your handler that only one of those two are set at a time with something like:
if (req.query.id && req.query.filterQuery) {
// respond with error
}
You can even mix the two if you have app.get('/server/lead/get/:id?') route you can have the id in the route and filterQuery as a query parameter. Now the URLs would be:
/server/lead/get/xxx (for id)
/server/lead/get?filterQuery=xxx (for filter)
For more info see: http://expressjs.com/en/api.html#req.query
Better way
If you follow some REST conventions then you can use:
app.get('/server/lead/:id') for one object with id (not optional)
app.get('/server/lead') for a list of objects (with optional filterQuery passed as a query parameter)
That way you would always know that when you access:
/server/lead/xxx - then it's one object with ID = xxx
/server/lead - then it's a list of any objects
/server/lead?filterQuery=xxx - then it's a list of objects that match the query
If you follow the REST conventions for things like this instead of inventing your own, it would be much easier for you to design the routes and handlers, and it would be much easier for other people to use your system.
You may also want to use plural /server/leads instead of /server/lead which is common with REST. That way it will be more obvious that leads is a list and leads/id is one of its elements.
For more info see:
https://en.wikipedia.org/wiki/Representational_state_transfer
http://www.restapitutorial.com/lessons/whatisrest.html
https://spring.io/understanding/REST
You have to realize that the following two routes match exactly the same:
app.get('/server/lead/get/:id?', leadCtrl.get);
app.get('/server/lead/get/:filter?', leadCtrl.get);
Express doesn't care about how you name the placeholders, so any requests for /server/lead/get/SOMEVALUE will always match the first (the one with :id).
You can add a distinction yourself, by only allowing a parameter to match a particular regular expression. From your code, it looks like :id should match MongoDB ObjectId's, so you can create a specific match for those:
app.get('/server/lead/get/:id([a-fA-F0-9]{24})?', leadCtrl.get);
If SOMEVALUE matches an ObjectId, it will call leadCtrl.get and populate req.params.id. If you also add another router for "the rest", you can also cover the req.params.filter case:
app.get('/server/lead/get/:filter?', leadCtrl.get);
As an aside: you're saying that you're passing JSON to the "filter" routes, in the URL. I would strongly suggest using a POST route for that, and post the JSON as request body content.
I use nodejs as my backend and store my data in MongoDB. I'm interested how should I check incomming data before saving into database.
I need to check as pure strings like:
"some xss test"
and object of strings:
{
"name": "xss name",
"age": 25
}
What library should I use for my task?
It is a general practice to verify the data when you are outputting it, not storing. Doing so, you do not need to worry, what if the XSS data got into database using other routes?
But your question still stands, how would a programmer check if something contains XSS or not. There is a validator module exactly for doing this job:
var validator = require('validator');
var escaped_string = validator.escape(someString);
To verify the object of strings, you might have to iterate manually through the list.
If you are actually intersted in outputting html code, but worry for XSS, then you need to use a more sophisticated XSS validator which is kept up-to-date. Example would be Google Caja