Suppose I have two tables USER_GROUP and USER_GROUP_DATASOURCE. I have a classic relation where one userGroup can have multiple dataSources and one DataSource simply is a String.
Due to some reasons, I have a custom RecordMapper creating a Java UserGroup POJO. (Mainly compatibility with the other code in the codebase, always being explicit on whats happening). This mapper sometimes creates simply POJOs containing data only from the USER_GROUP table, sometimes also the left joined dataSources.
Currently, I am trying to write the Multiset query along with the custom record mapper. My query thus far looks like this:
List<UserGroup> = ctx
.select(
asterisk(),
multiset(select(USER_GROUP_DATASOURCE.DATASOURCE_ID)
.from(USER_GROUP_DATASOURCE)
.where(USER_GROUP.ID.eq(USER_GROUP_DATASOURCE.USER_GROUP_ID))
).as("datasources").convertFrom(r -> r.map(Record1::value1))
)
.from(USER_GROUP)
.where(condition)
.fetch(new UserGroupMapper()))
Now my question is: How to create the UserGroupMapper? I am stuck right here:
public class UserGroupMapper implements RecordMapper<Record, UserGroup> {
#Override
public UserGroup map(Record rec) {
UserGroup grp = new UserGroup(rec.getValue(USER_GROUP.ID),
rec.getValue(USER_GROUP.NAME),
rec.getValue(USER_GROUP.DESCRIPTION)
javaParseTags(USER_GROUP.TAGS)
);
// Convention: if we have an additional field "datasources", we assume it to be a list of dataSources to be filled in
if (rec.indexOf("datasources") >= 0) {
// How to make `rec.getValue` return my List<String>????
List<String> dataSources = ?????
grp.dataSources.addAll(dataSources);
}
}
My guess is to have something like List<String> dataSources = rec.getValue(..) where I pass in a Field<List<String>> but I have no clue how I could create such Field<List<String>> with something like DSL.field().
How to get a type safe reference to your field from your RecordMapper
There are mostly two ways to do this:
Keep a reference to your multiset() field definition somewhere, and reuse that. Keep in mind that every jOOQ query is a dynamic SQL query, so you can use this feature of jOOQ to assign arbitrary query fragments to local variables (or return them from methods), in order to improve code reuse
You can just raw type cast the value, and not care about type safety. It's always an option, evne if not the cleanest one.
How to improve your query
Unless you're re-using that RecordMapper several times for different types of queries, why not do use Java's type inference instead? The main reason why you're not getting type information in your output is because of your asterisk() usage. But what if you did this instead:
List<UserGroup> = ctx
.select(
USER_GROUP, // Instead of asterisk()
multiset(
select(USER_GROUP_DATASOURCE.DATASOURCE_ID)
.from(USER_GROUP_DATASOURCE)
.where(USER_GROUP.ID.eq(USER_GROUP_DATASOURCE.USER_GROUP_ID))
).as("datasources").convertFrom(r -> r.map(Record1::value1))
)
.from(USER_GROUP)
.where(condition)
.fetch(r -> {
UserGroupRecord ug = r.value1();
List<String> list = r.value2(); // Type information available now
// ...
})
There are other ways than the above, which is using jOOQ 3.17+'s support for Table as SelectField. E.g. in jOOQ 3.16+, you can use row(USER_GROUP.fields()).
The important part is that you avoid the asterisk() expression, which removes type safety. You could even convert the USER_GROUP to your UserGroup type using USER_GROUP.convertFrom(r -> ...) when you project it:
List<UserGroup> = ctx
.select(
USER_GROUP.convertFrom(r -> ...),
// ...
Is there a catchall function somewhere that works well for sanitizing user input for SQL injection and XSS attacks, while still allowing certain types of HTML tags?
It's a common misconception that user input can be filtered. PHP even has a (now deprecated) "feature", called magic-quotes, that builds on this idea. It's nonsense. Forget about filtering (or cleaning, or whatever people call it).
What you should do, to avoid problems, is quite simple: whenever you embed a a piece of data within a foreign code, you must treat it according to the formatting rules of that code. But you must understand that such rules could be too complicated to try to follow them all manually. For example, in SQL, rules for strings, numbers and identifiers are all different. For your convenience, in most cases there is a dedicated tool for such an embedding. For example, when you need to use a PHP variable in the SQL query, you have to use a prepared statement, that will take care of all the proper formatting/treatment.
Another example is HTML: If you embed strings within HTML markup, you must escape it with htmlspecialchars. This means that every single echo or print statement should use htmlspecialchars.
A third example could be shell commands: If you are going to embed strings (such as arguments) to external commands, and call them with exec, then you must use escapeshellcmd and escapeshellarg.
Also, a very compelling example is JSON. The rules are so numerous and complicated that you would never be able to follow them all manually. That's why you should never ever create a JSON string manually, but always use a dedicated function, json_encode() that will correctly format every bit of data.
And so on and so forth ...
The only case where you need to actively filter data, is if you're accepting preformatted input. For example, if you let your users post HTML markup, that you plan to display on the site. However, you should be wise to avoid this at all cost, since no matter how well you filter it, it will always be a potential security hole.
Do not try to prevent SQL injection by sanitizing input data.
Instead, do not allow data to be used in creating your SQL code. Use Prepared Statements (i.e. using parameters in a template query) that uses bound variables. It is the only way to be guaranteed against SQL injection.
Please see my website http://bobby-tables.com/ for more about preventing SQL injection.
No. You can't generically filter data without any context of what it's for. Sometimes you'd want to take a SQL query as input and sometimes you'd want to take HTML as input.
You need to filter input on a whitelist -- ensure that the data matches some specification of what you expect. Then you need to escape it before you use it, depending on the context in which you are using it.
The process of escaping data for SQL - to prevent SQL injection - is very different from the process of escaping data for (X)HTML, to prevent XSS.
PHP has the new nice filter_input functions now, that for instance liberate you from finding 'the ultimate e-mail regex' now that there is a built-in FILTER_VALIDATE_EMAIL type
My own filter class (uses JavaScript to highlight faulty fields) can be initiated by either an ajax request or normal form post. (see the example below)
<?
/**
* Pork Formvalidator. validates fields by regexes and can sanitize them. Uses PHP filter_var built-in functions and extra regexes
* #package pork
*/
/**
* Pork.FormValidator
* Validates arrays or properties by setting up simple arrays.
* Note that some of the regexes are for dutch input!
* Example:
*
* $validations = array('name' => 'anything','email' => 'email','alias' => 'anything','pwd'=>'anything','gsm' => 'phone','birthdate' => 'date');
* $required = array('name', 'email', 'alias', 'pwd');
* $sanitize = array('alias');
*
* $validator = new FormValidator($validations, $required, $sanitize);
*
* if($validator->validate($_POST))
* {
* $_POST = $validator->sanitize($_POST);
* // now do your saving, $_POST has been sanitized.
* die($validator->getScript()."<script type='text/javascript'>alert('saved changes');</script>");
* }
* else
* {
* die($validator->getScript());
* }
*
* To validate just one element:
* $validated = new FormValidator()->validate('blah#bla.', 'email');
*
* To sanitize just one element:
* $sanitized = new FormValidator()->sanitize('<b>blah</b>', 'string');
*
* #package pork
* #author SchizoDuckie
* #copyright SchizoDuckie 2008
* #version 1.0
* #access public
*/
class FormValidator
{
public static $regexes = Array(
'date' => "^[0-9]{1,2}[-/][0-9]{1,2}[-/][0-9]{4}\$",
'amount' => "^[-]?[0-9]+\$",
'number' => "^[-]?[0-9,]+\$",
'alfanum' => "^[0-9a-zA-Z ,.-_\\s\?\!]+\$",
'not_empty' => "[a-z0-9A-Z]+",
'words' => "^[A-Za-z]+[A-Za-z \\s]*\$",
'phone' => "^[0-9]{10,11}\$",
'zipcode' => "^[1-9][0-9]{3}[a-zA-Z]{2}\$",
'plate' => "^([0-9a-zA-Z]{2}[-]){2}[0-9a-zA-Z]{2}\$",
'price' => "^[0-9.,]*(([.,][-])|([.,][0-9]{2}))?\$",
'2digitopt' => "^\d+(\,\d{2})?\$",
'2digitforce' => "^\d+\,\d\d\$",
'anything' => "^[\d\D]{1,}\$"
);
private $validations, $sanatations, $mandatories, $errors, $corrects, $fields;
public function __construct($validations=array(), $mandatories = array(), $sanatations = array())
{
$this->validations = $validations;
$this->sanitations = $sanitations;
$this->mandatories = $mandatories;
$this->errors = array();
$this->corrects = array();
}
/**
* Validates an array of items (if needed) and returns true or false
*
*/
public function validate($items)
{
$this->fields = $items;
$havefailures = false;
foreach($items as $key=>$val)
{
if((strlen($val) == 0 || array_search($key, $this->validations) === false) && array_search($key, $this->mandatories) === false)
{
$this->corrects[] = $key;
continue;
}
$result = self::validateItem($val, $this->validations[$key]);
if($result === false) {
$havefailures = true;
$this->addError($key, $this->validations[$key]);
}
else
{
$this->corrects[] = $key;
}
}
return(!$havefailures);
}
/**
*
* Adds unvalidated class to thos elements that are not validated. Removes them from classes that are.
*/
public function getScript() {
if(!empty($this->errors))
{
$errors = array();
foreach($this->errors as $key=>$val) { $errors[] = "'INPUT[name={$key}]'"; }
$output = '$$('.implode(',', $errors).').addClass("unvalidated");';
$output .= "new FormValidator().showMessage();";
}
if(!empty($this->corrects))
{
$corrects = array();
foreach($this->corrects as $key) { $corrects[] = "'INPUT[name={$key}]'"; }
$output .= '$$('.implode(',', $corrects).').removeClass("unvalidated");';
}
$output = "<script type='text/javascript'>{$output} </script>";
return($output);
}
/**
*
* Sanitizes an array of items according to the $this->sanitations
* sanitations will be standard of type string, but can also be specified.
* For ease of use, this syntax is accepted:
* $sanitations = array('fieldname', 'otherfieldname'=>'float');
*/
public function sanitize($items)
{
foreach($items as $key=>$val)
{
if(array_search($key, $this->sanitations) === false && !array_key_exists($key, $this->sanitations)) continue;
$items[$key] = self::sanitizeItem($val, $this->validations[$key]);
}
return($items);
}
/**
*
* Adds an error to the errors array.
*/
private function addError($field, $type='string')
{
$this->errors[$field] = $type;
}
/**
*
* Sanitize a single var according to $type.
* Allows for static calling to allow simple sanitization
*/
public static function sanitizeItem($var, $type)
{
$flags = NULL;
switch($type)
{
case 'url':
$filter = FILTER_SANITIZE_URL;
break;
case 'int':
$filter = FILTER_SANITIZE_NUMBER_INT;
break;
case 'float':
$filter = FILTER_SANITIZE_NUMBER_FLOAT;
$flags = FILTER_FLAG_ALLOW_FRACTION | FILTER_FLAG_ALLOW_THOUSAND;
break;
case 'email':
$var = substr($var, 0, 254);
$filter = FILTER_SANITIZE_EMAIL;
break;
case 'string':
default:
$filter = FILTER_SANITIZE_STRING;
$flags = FILTER_FLAG_NO_ENCODE_QUOTES;
break;
}
$output = filter_var($var, $filter, $flags);
return($output);
}
/**
*
* Validates a single var according to $type.
* Allows for static calling to allow simple validation.
*
*/
public static function validateItem($var, $type)
{
if(array_key_exists($type, self::$regexes))
{
$returnval = filter_var($var, FILTER_VALIDATE_REGEXP, array("options"=> array("regexp"=>'!'.self::$regexes[$type].'!i'))) !== false;
return($returnval);
}
$filter = false;
switch($type)
{
case 'email':
$var = substr($var, 0, 254);
$filter = FILTER_VALIDATE_EMAIL;
break;
case 'int':
$filter = FILTER_VALIDATE_INT;
break;
case 'boolean':
$filter = FILTER_VALIDATE_BOOLEAN;
break;
case 'ip':
$filter = FILTER_VALIDATE_IP;
break;
case 'url':
$filter = FILTER_VALIDATE_URL;
break;
}
return ($filter === false) ? false : filter_var($var, $filter) !== false ? true : false;
}
}
Of course, keep in mind that you need to do your sql query escaping too depending on what type of db your are using (mysql_real_escape_string() is useless for an sql server for instance). You probably want to handle this automatically at your appropriate application layer like an ORM. Also, as mentioned above: for outputting to html use the other php dedicated functions like htmlspecialchars ;)
For really allowing HTML input with like stripped classes and/or tags depend on one of the dedicated xss validation packages. DO NOT WRITE YOUR OWN REGEXES TO PARSE HTML!
No, there is not.
First of all, SQL injection is an input filtering problem, and XSS is an output escaping one - so you wouldn't even execute these two operations at the same time in the code lifecycle.
Basic rules of thumb
For SQL query, bind parameters
Use strip_tags() to filter out unwanted HTML
Escape all other output with htmlspecialchars() and be mindful of the 2nd and 3rd parameters here.
To address the XSS issue, take a look at HTML Purifier. It is fairly configurable and has a decent track record.
As for the SQL injection attacks, the solution is to use prepared statements. The PDO library and mysqli extension support these.
PHP 5.2 introduced the filter_var function.
It supports a great deal of SANITIZE, VALIDATE filters.
Methods for sanitizing user input with PHP:
Use Modern Versions of MySQL and PHP.
Set charset explicitly:
$mysqli->set_charset("utf8");manual
$pdo = new PDO('mysql:host=localhost;dbname=testdb;charset=UTF8', $user, $password);manual
$pdo->exec("set names utf8");manual
$pdo = new PDO(
"mysql:host=$host;dbname=$db", $user, $pass,
array(
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"
)
);manual
mysql_set_charset('utf8') [deprecated in PHP 5.5.0, removed in PHP 7.0.0].
Use secure charsets:
Select utf8, latin1, ascii.., dont use vulnerable charsets big5, cp932, gb2312, gbk, sjis.
Use spatialized function:
MySQLi prepared statements:
$stmt = $mysqli->prepare('SELECT * FROM test WHERE name = ? LIMIT 1'); $param = "' OR 1=1 /*";$stmt->bind_param('s', $param);$stmt->execute();
PDO::quote() - places quotes around the input string (if required) and escapes special characters within the input string, using a quoting style appropriate to the underlying driver:$pdo = new PDO('mysql:host=localhost;dbname=testdb;charset=UTF8', $user, $password);explicit set the character set$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);disable emulating prepared statements to prevent fallback to emulating statements that MySQL can't prepare natively (to prevent injection)$var = $pdo->quote("' OR 1=1 /*");not only escapes the literal, but also quotes it (in single-quote ' characters)
$stmt = $pdo->query("SELECT * FROM test WHERE name = $var LIMIT 1");
PDO Prepared Statements: vs MySQLi prepared statements supports more database drivers and named parameters: $pdo = new PDO('mysql:host=localhost;dbname=testdb;charset=UTF8', $user, $password);explicit set the character set$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);disable emulating prepared statements to prevent fallback to emulating statements that MySQL can't prepare natively (to prevent injection)
$stmt = $pdo->prepare('SELECT * FROM test WHERE name = ? LIMIT 1');
$stmt->execute(["' OR 1=1 /*"]);
mysql_real_escape_string [deprecated in PHP 5.5.0, removed in PHP 7.0.0].
mysqli_real_escape_string Escapes special characters in a string for use in an SQL statement, taking into account the current charset of the connection. But recommended to use Prepared Statements because they are not simply escaped strings, a statement comes up with a complete query execution plan, including which tables and indexes it would use, it is a optimized way.
Use single quotes (' ') around your variables inside your query.
Check the variable contains what you are expecting for:
If you are expecting an integer, use:
ctype_digit — Check for numeric character(s);$value = (int) $value;$value = intval($value);$var = filter_var('0755', FILTER_VALIDATE_INT, $options);
For Strings use:
is_string() — Find whether the type of a variable is stringUse Filter Function filter_var() — filters a variable with a specified filter:$email = filter_var($email, FILTER_SANITIZE_EMAIL);$newstr = filter_var($str, FILTER_SANITIZE_STRING);more predefined filters
filter_input() — Gets a specific external variable by name and optionally filters it:$search_html = filter_input(INPUT_GET, 'search', FILTER_SANITIZE_SPECIAL_CHARS);
preg_match() — Perform a regular expression match;
Write Your own validation function.
One trick that can help in the specific circumstance where you have a page like /mypage?id=53 and you use the id in a WHERE clause is to ensure that id definitely is an integer, like so:
if (isset($_GET['id'])) {
$id = $_GET['id'];
settype($id, 'integer');
$result = mysql_query("SELECT * FROM mytable WHERE id = '$id'");
# now use the result
}
But of course that only cuts out one specific attack, so read all the other answers. (And yes I know that the code above isn't great, but it shows the specific defence.)
There's no catchall function, because there are multiple concerns to be addressed.
SQL Injection - Today, generally, every PHP project should be using prepared statements via PHP Data Objects (PDO) as a best practice, preventing an error from a stray quote as well as a full-featured solution against injection. It's also the most flexible & secure way to access your database.
Check out (The only proper) PDO tutorial for pretty much everything you need to know about PDO. (Sincere thanks to top SO contributor, #YourCommonSense, for this great resource on the subject.)
XSS - Sanitize data on the way in...
HTML Purifier has been around a long time and is still actively updated. You can use it to sanitize malicious input, while still allowing a generous & configurable whitelist of tags. Works great with many WYSIWYG editors, but it might be heavy for some use cases.
In other instances, where we don't want to accept HTML/Javascript at all, I've found this simple function useful (and has passed multiple audits against XSS):
/* Prevent XSS input */
function sanitizeXSS () {
$_GET = filter_input_array(INPUT_GET, FILTER_SANITIZE_STRING);
$_POST = filter_input_array(INPUT_POST, FILTER_SANITIZE_STRING);
$_REQUEST = (array)$_POST + (array)$_GET + (array)$_REQUEST;
}
XSS - Sanitize data on the way out... unless you guarantee the data was properly sanitized before you add it to your database, you'll need to sanitize it before displaying it to your user, we can leverage these useful PHP functions:
When you call echo or print to display user-supplied values, use htmlspecialchars unless the data was properly sanitized safe and is allowed to display HTML.
json_encode is a safe way to provide user-supplied values from PHP to Javascript
Do you call external shell commands using exec() or system() functions, or to the backtick operator? If so, in addition to SQL Injection & XSS you might have an additional concern to address, users running malicious commands on your server. You need to use escapeshellcmd if you'd like to escape the entire command OR escapeshellarg to escape individual arguments.
What you are describing here is two separate issues:
Sanitizing / filtering of user input data.
Escaping output.
1) User input should always be assumed to be bad.
Using prepared statements, or/and filtering with mysql_real_escape_string is definitely a must.
PHP also has filter_input built in which is a good place to start.
2) This is a large topic, and it depends on the context of the data being output. For HTML there are solutions such as htmlpurifier out there.
as a rule of thumb, always escape anything you output.
Both issues are far too big to go into in a single post, but there are lots of posts which go into more detail:
Methods PHP output
Safer PHP output
If you're using PostgreSQL, the input from PHP can be escaped with pg_escape_literal()
$username = pg_escape_literal($_POST['username']);
From the documentation:
pg_escape_literal() escapes a literal for querying the PostgreSQL database. It returns an escaped literal in the PostgreSQL format.
You never sanitize input.
You always sanitize output.
The transforms you apply to data to make it safe for inclusion in an SQL statement are completely different from those you apply for inclusion in HTML are completely different from those you apply for inclusion in Javascript are completely different from those you apply for inclusion in LDIF are completely different from those you apply to inclusion in CSS are completely different from those you apply to inclusion in an Email....
By all means validate input - decide whether you should accept it for further processing or tell the user it is unacceptable. But don't apply any change to representation of the data until it is about to leave PHP land.
A long time ago someone tried to invent a one-size fits all mechanism for escaping data and we ended up with "magic_quotes" which didn't properly escape data for all output targets and resulted in different installation requiring different code to work.
Easiest way to avoid mistakes in sanitizing input and escaping data is using PHP framework like Symfony, Nette etc. or part of that framework (templating engine, database layer, ORM).
Templating engine like Twig or Latte has output escaping on by default - you don't have to solve manually if you have properly escaped your output depending on context (HTML or Javascript part of web page).
Framework is automatically sanitizing input and you should't use $_POST, $_GET or $_SESSION variables directly, but through mechanism like routing, session handling etc.
And for database (model) layer there are ORM frameworks like Doctrine or wrappers around PDO like Nette Database.
You can read more about it here - What is a software framework?
Just wanted to add that on the subject of output escaping, if you use php DOMDocument to make your html output it will automatically escape in the right context. An attribute (value="") and the inner text of a <span> are not equal.
To be safe against XSS read this:
OWASP XSS Prevention Cheat Sheet
PHP filter extension has many of the functions needed for checking the externaluser input & it is designed for making data sanitization easier and quicker.
PHP filters can comfortably sanitize & validate the external input.
I'm working with Kohana 3.2 and have the following code in my controller.
$Blog_Post = new Model_Blogpost();
$Blog_Post->where('id', '=', 1);
$Blog_Post->find();
$content = $Blog_Post->content;
I Currently have 3 records in my db with id's 1, 2, and 3.
$Blog_Post->content, or any other field return null. and I'm not sure why.
Use ORM::factory('blogpost', $id) or new Model_Blogpost($id) if you need an object with PK == $id.
Check your model after loading.
if $Blog_Post->loaded()
{
// it works!
}
else
{
// record not found
}
If record not found, you can see last DB query with $Blog_Post->last_query()
UPD. From comments. Your model will not work with this modifications. Note that ORM data stored in $_object property, and $Blog_Post->content is just a shortcut for $Blog_Post->_object['content'] via __get() method. Of course, if you define public $content property, $Blog_Post->content will return NULL value instead of using DB data.
There is no reason for defining model fields as properties. If you need IDE hints, just use PHPDOC.
At the firm I work for we were looking into upgrading to 3.2 very recently. However, in our evaluation I don't recall seeing a difference in ORM handling methods.
Yours above looks like it should be something like this:
$Blog_Post = ORM::factory('blogpost')->where('id', '=', 1)->find();
$content = $Blog_Post->content;
Assuming your table is called blogposts, of course. I may be wrong about that and if I am, can you link to the documentation that shows this kind of model interaction?