I was searching about something on Git when I noticed that in the query string, there's a parameter utf8 and its value is ✓! Not utf8=yes or utf8=true.
The full URL is following in Chrome and Firefox:
https://github.com/Modernizr/Modernizr/search?utf8=✓&q=browser&type=
But following in IE:
https://github.com/Modernizr/Modernizr/search?utf8=%E2%9C%93&q=browser&type=
So it seems to be a way of detecting the encoding scheme for the URL, but does anyone know for sure? Also, aren't there any simpler ways of doing this?
From a StackExchange question:
By default, older versions of IE (<=8) will submit form data in Latin-1 encoding if possible. By including a character that can't be expressed in Latin-1, IE is forced to use UTF-8 encoding for its form submissions, which simplifies various backend processes, for example database persistence.
If the parameter was instead utf8=true then this wouldn't trigger the UTF-8 encoding in these browsers.
This is a hack/feature of Rails (which Github is built with) to force IE to submit UTF-8 text.
It's a feature of Rails (which GitHub is built with), although it's not specific to Rails.
Related
I am returning text field from DB column:
"algún nombre de juego"
when I try to use this text in an html file, I get the following string:
algún nombre de juego
This happens consistently with any string that is not pure english. I have even tried grabbing simple text from online in different languages and as soon as I plug them into an html text box, the print out with improper encoding like the example above.
From reading Unicode documentation for python, UTF-8 should be able to handle pretty much any character in any language.
I have tried many ways of encoding/decoding, and either there is an encoding error, or I get back weird character where the letter u with acute should be.
Per comments, I am not using Django or Flask. Just taking strings from a DB that might be in several different languages and generating an HTML file for internal use.
A few ideas:
1. Remember to declare the encoding. Either:
Insert a <meta charset="utf-8"> just after your <head> tag.
OR
do this in the HTTP header (making it look something like Content-Type:text/html; charset=UTF-8). This can be done by changing the webserver's settings or using server-side code, should you choose to host the page.
2. Ensure that your IDE saves files as UTF-8. This will be somewhere in your settings and should be done automatically, but worth checking if option 1 doesn't work.
Hope one of these works.
We are currently using OWASP Antisamy project to protect our application against XSS attacks. Every input field is sanitized when any given form is submitted to server. It works fine, but we have issues with the fields like company name, organization name, etc.
Ex: Ampersand is escaped for AT&T and the company name is displayed wrong (displayed with escaped characters).
We manually update the fields on database to fix this issue. However, this is a pain in the neck as you can imagine.
Is there a way to address this using OWASP antisamy or should we use a different library?
You should only encode on output, not on input. If a user enters AT&T in your application, this should be stored at AT&T in the database. There is no need to encode it, but of course make sure that you are using parameterised queries which will prevent characters such as ' from breaking out of the SQL command context and causing SQL injection.
When you output, this is the only time you need to encode characters. e.g. AT&T should be encoded as AT&T when output to HTML, so it is displayed in the browser as AT&T.
It seems like your application is encoding the input and also encoding the output, so strings like above will be double encoded at then output as AT&T in your HTML, causing the problem. Remove your input encoding to solve this.
The reason you should only encode when output is that if you decide you want to output data to a different format such as JSON or JavaScript, then the encoding is different. O'leary would become O\x27leary if encoded properly for JavaScript, which would not display properly in HTML where the encoding is O'leary.
I am working on a site that have an international aim; I.o.w., logged in users can add text in their own language. I am hoping for international page names and content.
An URL example, like the Japanese Wikipedia: http://ja.wikipedia.org/wiki/メインページ (Both pagename and content text).
I know by using UTF-8, I can do this, but how should I control it?
UTF-8 contains way to many languages/letters to control in a script, I guess, so how safe/unsafe is it to allow people to add UTF-8 text?
I can see that someone could add harmful code this way, but how to prevent it?
All information regarding safety/control when using UTF-8 is appreciated!
EDIT: PS! I use PHP and MySQL.
Warning: perhaps a slightly rusty response:
Note: not discussing host name (IDNS) issues.
The only completely safe thing here is to use %-escaped UTF-8. Some browsers will display this as what you want, and some will display the %-escapes. (e.g. http://foo.bar/%ee%cc%cf.html)
If you put 'real UTF-8' in the URLs, many things will work, but there may be unpleasant surprises lurking for some people in some browsers. I'm reading your question as dealing with 100% static content. If you are trying to do this with code behind the site, you have additional issues to work on.
The 'unpleasant surprises' would be (a) people finding the %xx's in the URL unreadable, (b) a browser that melts, (c) some data scraping or aggregating application that melts.
I wish I were more up to date on this, but I'm not, so my recommendation is to deploy a test site and then try to access it with everything you can put your hands on, including mobile phones. Persuade Google to index it, and see what happens there.
For domain names, this is called IDN. For page names, you may want to think of the possibility of IDN spoofs.
It's safe as long as you don't interpret it literally as SQL (SQL injection) or HTML (XSS) or any other language. Just escape any user-controlled input (request URL, request headers, request parameters, request body, etc..etc..) at the point it's going to be used in SQL or HTML.
It's unclear what server side programming language you're using, so I can't go further in detail.
How to safe gaurd a form against script injection attacks. This is one of the most used form of attacks in which attacker attempts to inject a JS script through form field. The validation for this case must check for special characters in the form fields. Look for
suggestions, recommedations at internet/jquery etc for permissible characters &
character masking validation JS codes.
You can use the HTML Purifier (in case you are under PHP or you might have other options for the language you are under) to avoid XSS (cross-site-scripting) attacks to great level but remember no solution is perfect or 100% reliable. This should help you and always remember server-side validation is always best rather than relying on javascript which bad guys can bypass easily disabling javascript.
For SQL Injection, you need to escape invalid characters from queries that can be used to manipulate or inject your queries and use type-casting for all your values that you want to insert into the database.
See the Security Guide for more security risks and how to avoid them. Note that even if you are not using PHP, the basic ideas for the security are same and this should get you in a better position about security considerations.
If you output user controlled input in html context then you could follow what others and sanitize when processing input (html purify, custom input validation) and/or html encode the values before output.
Cases when htmlencodng/strip tags (no tags needed) is not sufficient:
user input appears in attributes then it depends on whether you always (double) quote attributes or not (bad)
used in on* handlers (such as onload="..), then html encoding is not sufficient since the javascript parser is called after html decode.
appears in javascript section - depends on whether this is in quoted (htmlentity encode not sufficient) or unquoted region (very bad).
is returned as json which may be eval'ed. javascript escape required.
appears in CSS - css escape is different and css allows javascript (expression)
Also, these do not account for browser flaws such as incomplete UTF-8 sequence exploit, content-type sniffing exploits (UTF-7 flaw), etc.
Of course you also have to treat data to protect against other attacks (SQL or command injection).
Probably the best reference for this is at the OWASP XSS Prevention Cheat Sheet
ASP.NET has a feature called Request Validation that will prevent unencoded HTML from being processed by the server. For extra protection, one can use the AntiXSS library.
you can prevent script injection by encoding html content like
Server.HtmlEncode(input)
There is the OWASP EASPI too.
I am not concerned about other kinds of attacks. Just want to know whether HTML Encode can prevent all kinds of XSS attacks.
Is there some way to do an XSS attack even if HTML Encode is used?
No.
Putting aside the subject of allowing some tags (not really the point of the question), HtmlEncode simply does NOT cover all XSS attacks.
For instance, consider server-generated client-side javascript - the server dynamically outputs htmlencoded values directly into the client-side javascript, htmlencode will not stop injected script from executing.
Next, consider the following pseudocode:
<input value=<%= HtmlEncode(somevar) %> id=textbox>
Now, in case its not immediately obvious, if somevar (sent by the user, of course) is set for example to
a onclick=alert(document.cookie)
the resulting output is
<input value=a onclick=alert(document.cookie) id=textbox>
which would clearly work. Obviously, this can be (almost) any other script... and HtmlEncode would not help much.
There are a few additional vectors to be considered... including the third flavor of XSS, called DOM-based XSS (wherein the malicious script is generated dynamically on the client, e.g. based on # values).
Also don't forget about UTF-7 type attacks - where the attack looks like
+ADw-script+AD4-alert(document.cookie)+ADw-/script+AD4-
Nothing much to encode there...
The solution, of course (in addition to proper and restrictive white-list input validation), is to perform context-sensitive encoding: HtmlEncoding is great IF you're output context IS HTML, or maybe you need JavaScriptEncoding, or VBScriptEncoding, or AttributeValueEncoding, or... etc.
If you're using MS ASP.NET, you can use their Anti-XSS Library, which provides all of the necessary context-encoding methods.
Note that all encoding should not be restricted to user input, but also stored values from the database, text files, etc.
Oh, and don't forget to explicitly set the charset, both in the HTTP header AND the META tag, otherwise you'll still have UTF-7 vulnerabilities...
Some more information, and a pretty definitive list (constantly updated), check out RSnake's Cheat Sheet: http://ha.ckers.org/xss.html
If you systematically encode all user input before displaying then yes, you are safe you are still not 100 % safe.
(See #Avid's post for more details)
In addition problems arise when you need to let some tags go unencoded so that you allow users to post images or bold text or any feature that requires user's input be processed as (or converted to) un-encoded markup.
You will have to set up a decision making system to decide which tags are allowed and which are not, and it is always possible that someone will figure out a way to let a non allowed tag to pass through.
It helps if you follow Joel's advice of Making Wrong Code Look Wrong or if your language helps you by warning/not compiling when you are outputting unprocessed user data (static-typing).
If you encode everything it will. (depending on your platform and the implementation of htmlencode) But any usefull web application is so complex that it's easy to forget to check every part of it. Or maybe a 3rd party component isn't safe. Or maybe some code path that you though did encoding didn't do it so you forgot it somewhere else.
So you might want to check things on the input side too. And you might want to check stuff you read from the database.
As mentioned by everyone else, you're safe as long as you encode all user input before displaying it. This includes all request parameters and data retrieved from the database that can be changed by user input.
As mentioned by Pat you'll sometimes want to display some tags, just not all tags. One common way to do this is to use a markup language like Textile, Markdown, or BBCode. However, even markup languages can be vulnerable to XSS, just be aware.
# Markup example
[foo](javascript:alert\('bar'\);)
If you do decide to let "safe" tags through I would recommend finding some existing library to parse & sanitize your code before output. There are a lot of XSS vectors out there that you would have to detect before your sanitizer is fairly safe.
I second metavida's advice to find a third-party library to handle output filtering. Neutralizing HTML characters is a good approach to stopping XSS attacks. However, the code you use to transform metacharacters can be vulnerable to evasion attacks; for instance, if it doesn't properly handle Unicode and internationalization.
A classic simple mistake homebrew output filters make is to catch only < and >, but miss things like ", which can break user-controlled output out into the attribute space of an HTML tag, where Javascript can be attached to the DOM.
No, just encoding common HTML tokens DOES NOT completely protect your site from XSS attacks. See, for example, this XSS vulnerability found in google.com:
http://www.securiteam.com/securitynews/6Z00L0AEUE.html
The important thing about this type of vulnerability is that the attacker is able to encode his XSS payload using UTF-7, and if you haven't specified a different character encoding on your page, a user's browser could interpret the UTF-7 payload and execute the attack script.
One other thing you need to check is where your input comes from. You can use the referrer string (most of the time) to check that it's from your own page, but putting in a hidden random number or something in your form and then checking it (with a session set variable maybe) also helps knowing that the input is coming from your own site and not some phishing site.
I'd like to suggest HTML Purifier (http://htmlpurifier.org/) It doesn't just filter the html, it basically tokenizes and re-compiles it. It is truly industrial-strength.
It has the additional benefit of allowing you to ensure valid html/xhtml output.
Also n'thing textile, its a great tool and I use it all the time, but I'd run it though html purifier too.
I don't think you understood what I meant re tokens. HTML Purifier doesn't just 'filter', it actually reconstructs the html. http://htmlpurifier.org/comparison.html
I don't believe so. Html Encode converts all functional characters (characters which could be interpreted by the browser as code) in to entity references which cannot be parsed by the browser and thus, cannot be executed.
<script/>
There is no way that the above can be executed by the browser.
**Unless their is a bug in the browser ofcourse.*
myString.replace(/<[^>]*>?/gm, '');
I use it, then successfully.
Strip HTML from Text JavaScript